Alert investigation — logs

Load this file when investigating a log alert (log-pattern threshold, log-volume spike, log-level anomaly).

Log alerts arrive with at minimum:

alertID, alertName, alertValue — identity and the numeric value that crossed threshold
group, groupValue — the dimension the alert groups on (e.g. service_name=auth, level=error)
query — the log filter the alert uses
thresholdWindow — how long the condition held before firing
logsLink — deep-link to the logs UI showing the triggering data
timeRange — the window during which the threshold was breached

Reconstruct the matching logs. Use the query-logs tool with the alert's query and timeRange. Start with a small limit (10–20) to sample.
Identify the pattern. Are these repeated occurrences of the same error, or heterogeneous failures that share an attribute (service, endpoint)?
Correlate with traces. If the logs carry trace IDs, pull those traces — the span hierarchy often reveals which downstream dependency failed.
Correlate with errors. If the log messages match a specific exception, check query-error-groups for onset timestamps.
Check for an external trigger. Did a deploy happen in the thresholdWindow? A flag flip? A traffic surge?

What triggered — cite the alertName, threshold, and value.
Likely cause — the log pattern with a specific example log line quoted verbatim, plus the suspected trigger (deploy, flag, dependency).
Scope — number of affected services, users, or sessions within the window.
Next steps — specific: "roll back the N+1 deploy of <service>", "disable flag <key> in <env>", "restart <service> if the errors continue past <time>".

Querying a time window wider than the alert's. Stick to the alert's window unless the investigation explicitly needs pre/post comparison.
Paraphrasing log lines. Quote verbatim — that's the evidence.
Skipping get-keys before filtering on an attribute you haven't used before. Wrong name returns empty silently.

/alert-investigation