Index of every page on this site: llms.txt
All skills
kubernetes-sigs avatar

/fix-flakes

@413f058
Run it yourself

Diagnose and fix flaky tests tracked as open kind/flake issues in kubernetes-sigs/agent-sandbox — reproduce the flake, apply a minimal fix, and open a PR linking the issue. Use when asked to fix a flake, work through kind/flake issues, or when run on a schedule after dev/tools/flake-report has filed flake issues.

  • 1 file
  • 4.6 KB
  • Updated 2 weeks ago
  • GitHub
Use this Skill: https://skilld.dev/gh/kubernetes-sigs/agent-sandbox/fix-flakes

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈82 tokens for metadata: the name and description. ≈1.1k when used: this file.

Description uses 4.1% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈327 characters, or 4.1%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Fix flaky tests from kind/flake issues

Work through open kind/flake issues (filed nightly by dev/tools/flake-report), reproduce each flake, land a minimal fix, and open a PR that closes the issue. Evidence before edits: never change a test you could not observe failing or whose failure output you have not read.

Inputs

  • A specific issue number, if the caller gives one; otherwise discover work: gh issue list --repo kubernetes-sigs/agent-sandbox --label kind/flake --state open --json number,title,body
  • Skip any issue that already has an open fix PR: search gh pr list --repo kubernetes-sigs/agent-sandbox --state open --search "<issue number> in:body" and check the results actually reference Fixes #<n> / Closes #<n>.
  • Skip infra-failure issues (title contains "infrastructure failures") and job-level issues (title contains "job fails intermittently", or the body carries a <!-- flake-report:job-flake= or <!-- flake-report:job= marker) unless explicitly asked — those name a Prow job, not a test, so there is nothing to reproduce or edit here. Infra failures are CI tooling work (dev/ci/, prow job config in kubernetes/test-infra); job-level issues need the job's build log and usually a build or publish fix. If you do take an infra one, the fix lives in dev/ci/shared/runner.py retries, image prepulls, or job resources.

Reproduce first

  1. Read the issue: test name, tab/job, failure timestamps, job-history link. Fetch a failing run's junit + build log from the GCS links to read the real error before touching anything.
  2. Locate the test: git grep -n "func <TestName>(" (Go) or git grep -n "def <test_name>(" (Python SDK e2e). For a Go subtest (TestFoo/subtest), grep for the parent (func TestFoo() only — the part before the first / — then find the t.Run("subtest", ...) inside it.
  3. Stress it:
    • Unit tests: go test ./<package>/... -run '^<TestName>$' -race -count=50 (for a subtest, -run '^TestFoo$/^subtest$')
    • e2e tests need a cluster: make deploy-kind first (see AGENTS.md), then go test ./test/e2e/... -run '^<TestName>$' -count=10 with KUBECONFIG=bin/KUBECONFIG.
  4. If it will not reproduce locally after a reasonable stress run, do NOT guess-fix. Analyze the CI failure output instead; if the cause is still unclear, post your findings as a comment on the issue and stop.

Diagnose — the usual suspects

  • Missing eventual-consistency handling: asserting on controller-driven state without polling. Fix by waiting with a deadline (see existing helpers in test/e2e/), not by sleeping.
  • Timeouts tuned for fast machines: CI under Docker-in-Docker is slow. If a timeout must grow, justify it in a comment; prefer replacing fixed sleeps with polling.
  • Shared state between parallel tests: cluster-scoped resources, fixed names, fixed ports. Namespace or randomize per test.
  • Ordering assumptions on lists/maps, time-of-day assumptions, leaked goroutines from a previous test (-race output helps).

Fix and verify

  • Minimal diff, repo conventions (AGENTS.md). Never disable or skip a test to make it green; quarantine decisions belong to maintainers on the issue.
  • Re-run the stress loop from "Reproduce" — it must pass repeatedly (unit: -count=50 clean; e2e: -count=10 clean).
  • Run make test-unit (and the affected e2e suite if applicable).

Deliver

  • Branch flake-fix/issue-<n>, one issue per branch/PR.
  • Commit style matches recent history (e.g. fix(e2e): wait for sandbox ready in TestFoo (#<n>)).
  • PR body: Fixes #<n>, the diagnosis in two or three sentences, and the before/after stress-run evidence (command + result).
  • If the fix is a revert-worthy product bug rather than a test bug, say so on the issue and file/label accordingly instead of papering over it in tests.

Scheduled / automated mode

When invoked non-interactively (e.g. via dev/tools/flake-fix on cron):

  • Process at most the number of issues the caller specifies (default 1) so a run stays bounded.
  • Only act on issues where reproduction or CI failure output confirms the diagnosis; otherwise leave an analysis comment on the issue and move on.
  • Push the branch and open the PR autonomously, then report: issues examined, PRs opened, issues commented, issues skipped and why.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 413f058. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 weeks ago

README badge

README badge for kubernetes-sigs/agent-sandbox/fix-flakes