Index of every page on this site: llms.txt
All skills
datadog-labs avatar

/triage-flaky-test

@618fae3 official
Run it yourself

Load when investigating a specific flaky test. Gets history, failure pattern, and category, then recommends fix, quarantine, or escalate.

Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/triage-flaky-test

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesfix-patterns.md

≈1.9k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Forbidden Fixes

These patterns are always invalid, regardless of category. A fix that only reduces flake probability instead of eliminating the root cause is not a fix.

  • Blind timing hacks: adding time.Sleep, or widening timeouts/retries when the root cause is not the timeout itself (e.g. to paper over a race condition) — these mask the problem and slow the test suite. Exception: increasing a timeout IS valid when the timeout category confirms the deadline is genuinely too tight for the operation under test.
  • Relaxed assertions: changing exact matches to at_least(1), contains, or range checks when the exact value is deterministic and should be asserted precisely
  • Wrong-layer patches: fixing a test when the bug is in production code, or touching production code for a test-only issue
  • Partial fixes: fixing one callsite of a shared helper when multiple callsites share the root cause — find and fix all callsites
  • Infrastructure workarounds: any code change that attempts to work around Docker, network, or CI runner unavailability — these cannot be fixed in test code

Flaky Test Fix Patterns by Category

For each flaky_category value, apply the corresponding fix pattern. Use stack trace evidence from STEP 2 to confirm the pattern before proposing a change.


timeout

Signal: Test exceeds time limit; stack trace shows timeout or deadline exceeded.

Confirm first: Verify the deadline itself is the root cause — not an underlying race, stuck goroutine, or async ordering issue that happens to surface as a timeout. If concurrency or asynchronous_wait signals are present, fix the synchronization instead.

Preferred fix: Replace real waits with mocked/injectable time so the test runs deterministically regardless of environment speed.

type Clock interface { Now() time.Time }
// inject a fake clock in tests instead of relying on wall time

Acceptable fix: Increase the timeout when the deadline is provably too tight for the operation (e.g. a real I/O call that consistently needs more time than allowed). Increase proportionally — 2–5× the current value — not arbitrarily.

// Before — too tight for the operation
ctx, cancel := context.WithTimeout(ctx, 1*time.Second)

// After — proportional increase with justification
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)

Invalid: Bumping a timeout to make a race condition or async ordering issue fail less often — this masks the real problem.

For polling loops, add a maximum retry count with exponential backoff rather than a fixed sleep.


concurrency

Signal: Data race, deadlock, or non-deterministic ordering. Stack trace may show goroutine contention.

Fix: Add synchronization. Common patterns:

// Add mutex around shared state
var mu sync.Mutex
mu.Lock()
sharedState = value
mu.Unlock()

// Use sync.WaitGroup for goroutine coordination
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); doWork() }()
wg.Wait()

time

Signal: Assertions compare timestamps or durations that vary by wall-clock time.

Fix: Inject a clock interface; use a fake clock in tests.

// Before — uses time.Now() directly
func IsExpired(t time.Time) bool { return time.Now().After(t) }

// After — injectable clock
type Clock interface { Now() time.Time }
func IsExpired(t time.Time, clk Clock) bool { return clk.Now().After(t) }

In tests, pass a fakeClock that returns a deterministic value.


network

Signal: Connection refused, DNS failure, or timeout on external calls.

Fix: Mock the HTTP client or network dependency in tests.

// Inject an http.Client so tests can swap it
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte(`{"ok":true}`))
}))
defer server.Close()
client := &Client{httpClient: server.Client()}

randomness

Signal: Test output varies between runs with no environmental change.

Fix: Seed the random number generator with a fixed value in tests.

// Before
r := rand.Int()

// After — deterministic in tests
rng := rand.New(rand.NewSource(42))
r := rng.Int()

order_dependency

Signal: Test passes in isolation but fails when run with other tests; order of test execution matters.

Fix: Add proper setup and teardown to isolate test state.

func TestMyFeature(t *testing.T) {
    // Reset shared state before test
    db.Reset()
    cache.Clear()
    defer func() {
        db.Reset()
    }()
    // ... test body
}

environment_dependency

Signal: Test fails only in CI or only on certain machines; depends on env vars or installed tools.

Fix: Inject configuration rather than reading env vars directly in the function under test.

// Before — reads env var in tested function
func NewClient() *Client { return &Client{endpoint: os.Getenv("API_URL")} }

// After — injectable config
type Config struct { APIEndpoint string }
func NewClient(cfg Config) *Client { return &Client{endpoint: cfg.APIEndpoint} }

io

Signal: File not found, permission denied, or leftover files from previous runs.

Fix: Use t.TempDir() for test files; it is cleaned up automatically after the test.

func TestFileProcessing(t *testing.T) {
    dir := t.TempDir()
    path := filepath.Join(dir, "test.txt")
    os.WriteFile(path, []byte("content"), 0644)
    // ... test using path
}

resource_leak

Signal: Resource exhaustion (open files, goroutines, DB connections) accumulates across tests.

Fix: Use defer to release resources; use t.Cleanup for test-scoped cleanup.

conn, err := db.Connect()
require.NoError(t, err)
t.Cleanup(func() { conn.Close() })

too_restrictive_range

Signal: Assertion checks for an exact value that legitimately varies (latency, size, timestamp).

Fix: Widen the assertion to a range or relative comparison.

// Before — exact match on timing
assert.Equal(t, 100*time.Millisecond, elapsed)

// After — allow variance
assert.InDelta(t, 100.0, elapsed.Milliseconds(), 50.0) // ±50ms

unordered_collection

Signal: Assertion fails because slice or map iteration order varies between runs.

Fix: Sort before comparing, or use a set-based assertion.

// Before
assert.Equal(t, []string{"a", "b", "c"}, result)

// After — sort both sides, or use ElementsMatch
sort.Strings(result)
assert.Equal(t, []string{"a", "b", "c"}, result)

// Or with testify
assert.ElementsMatch(t, []string{"a", "b", "c"}, result)

floating_point

Signal: Floating-point equality assertion fails due to precision differences.

Fix: Use tolerance-based comparison.

// Before
assert.Equal(t, 0.3, 0.1+0.2)

// After
assert.InDelta(t, 0.3, 0.1+0.2, 1e-9)

asynchronous_wait

Signal: Test races against an async operation; result is checked before it completes.

Fix: Use channels, sync.WaitGroup, or a polling helper with a deadline instead of fixed sleeps.

// Before — fixed sleep, races against async completion
time.Sleep(100 * time.Millisecond)
assert.Equal(t, "done", getStatus())

// After — poll with a deadline
require.Eventually(t, func() bool {
    return getStatus() == "done"
}, 5*time.Second, 50*time.Millisecond)

unknown

Category could not be determined automatically. Do not attempt a code fix. Follow STEP 4/5 in the main skill: self-introduced flakes escalate to the author; pre-existing or undetermined flakes follow the impact thresholds and infrastructure guard. Include the full triage brief for manual investigation.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning16d3 checks · Risk SAFE
  • Gen Agent Trust Hub17d

    This skill provides an automated workflow for triaging flaky tests using Datadog tools. It is functionally safe and uses trusted resources, but it possesses an indirect prompt injection surface by processing external test logs and stack traces.

  • Socket16d

    No alerts

  • Snyk17d

    Risk: MEDIUM · 1 issue

Signed by skilld at 618fae3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 3 weeks ago
Other metadata
metadata
{
  "version": "1.1.0",
  "author": "datadog-labs",
  "repository": "https://github.com/datadog-labs/agent-skills",
  "tags": "datadog,ci,cicd,flaky,flaky-tests,test-optimization",
  "alwaysApply": "false"
}

README badge

README badge for datadog-labs/agent-skills/triage-flaky-test

Analyzes a flaky test using Datadog CI Visibility data — retrieves failure history, patterns, and root cause category, then recommends whether to fix the test, quarantine it, or escalate to the owning team. Requires Datadog MCP server or pup CLI with authentication.

Generated from the current SKILL.md.

What does this skill require to run?
The skill requires either the Datadog MCP server or the pup CLI (installed and authenticated). It will detect which backend is available at runtime.
Can this skill fix a flaky test automatically?
No. The skill analyzes the test, identifies the root cause category, and recommends either a specific code fix, quarantine, or escalation to codeowners. Any code changes are the user's responsibility.
What does quarantine do to a test?
Quarantine suppresses test failures so CI is not blocked, but the test still runs and is tracked. It is reversible — you can restore the test to active status later.
Does the skill require me to provide the repository ID?
No. If not provided, the skill derives it from `git remote get-url origin`. You only need the fully qualified test name.
What happens if the skill cannot confirm the root cause?
If the root cause is uncertain or the test category is unknown, the skill recommends escalation to codeowners with the triage brief rather than proposing a code fix.

Generated from the current SKILL.md. These answers refresh after source changes.