Index of every page on this site: llms.txt
All skills
launchdarkly avatar

/alert-investigation

@f0ee110 official
Run it yourself

Investigates a firing or recently fired observability alert: finds the firing and replays the value it evaluated, decides whether the alert is chronic noise or a new problem, reuses any automatic investigation, then finds the cause (what changed, which host, version or route is over-represented) and returns a structured diagnosis with cause, scope and next steps.

Use this Skill: https://skilld.dev/gh/launchdarkly/ai-tooling/alert-investigation

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

traces.md

≈684 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Alert investigation: traces

Load this file for a trace alert: latency (a percentile or average of duration), span error rate, or span count.

Before you look for a cause

  • Replay the firing with the replay arguments from get-alert-firing-history, then again with compareTo: "1w". Latency alerts are the most common source of noise; a value that matches last week's is a standing condition.
  • Count the rows. Check the replay for a low_sample block, and add a Count expression. A P99 over about 600 spans per window is the sixth-slowest span, and over sampled data the rows read are far fewer than the spans. Set exact: true for the alert window. If the alert's percentile rests on a few dozen rows, the alert is fragile; recommend P95, a longer window or a minimum count.
  • Compare the firing group with its siblings. For an alert grouped by task.type or http.route, replay with the alert's groupBy and bucketCount: 1: is only the firing group slow, and was it already slow last week?

Investigation shape

  1. Slowest or failing spans in the window: search-traces with the alert's query, environments and services, sortColumn: "duration", sortDirection: "DESC", limit 5; or has_errors=true for error-rate alerts.
  2. Read the slowest trace with get-trace (pass timestamp). top_self_time and critical_path name the span that owns the time (self_ms, share_of_root); without them, subtract the children's durations from the parent's. For errors, find the lowest error span.
  3. Is that span the pattern? Aggregate Sum and P95 of duration for the parent and its child span names with exact: true and compareTo: "1w". "Child X is 65 % of the operation's time, as it was last week" is a cause and a verdict at once.
  4. What is over-represented: break the alert's signal down by host.name, the version key and http.route with compareTo (step 4 of the first five minutes in investigate).
  5. What changed: get-changes for the service from 1 to 2 hours before the onset.

What goes in the diagnosis

  • What triggered: alert name and ID, group, function and threshold, the replayed value in ms or s, and the row count behind it.
  • Verdict: new, chronic or noise, with firings in 7 days and last week's value.
  • Likely cause: the span that owns the time or the failure, with a trace ID and its share of total time.
  • Next steps: the code or dependency to fix, or a concrete alert retune.

Common mistakes

  • Reporting raw nanoseconds.
  • Treating one slow trace as the pattern without an aggregate.
  • Blaming the longest child span when the parent's own time is larger.
  • Quoting a sampled P99 over a few dozen rows.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warningtoday3 checks · Risk SAFE
  • Gen Agent Trust Hubtoday

    The skill is designed to investigate observability alerts by querying a LaunchDarkly MCP server. It retrieves logs, traces, metrics, and session data to diagnose system failures. The primary security consideration is the ingestion of untrusted data from logs and error messages, which presents a surface for indirect prompt injection.

  • Sockettoday

    No alerts

  • Snyktoday

    Risk: MEDIUM · 2 issues

Signed by skilld at f0ee110. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
compatibility
Requires the remotely hosted LaunchDarkly MCP server
metadata
{
  "author": "launchdarkly",
  "version": "0.3.0"
}

README badge

README badge for launchdarkly/ai-tooling/alert-investigation