Alert investigation: errors
Load this file for an error alert: error count or rate, a new error group, crash rate.
Before you look for a cause
- Replay the firing from
get-alert-firing-history, then withcompareTo: "1w". An error count that was the same last week is a standing bug, not this firing's cause. - Name the groups.
search-error-groupswith the alert's query, the firing window andsort: "count"; for new-group alerts,sort: "first_seen".
Investigation shape
- Read the top group with
get-error-group(secureId): stack, one occurrence, itstrace_idand session. The originating frame is the deepest application frame, not framework code. - Onset:
search-aggregationswithproductType: "errors", the group'sexception.messagefilter, over a window wider than the alert,bucketCount24 to 48. - What is over-represented: group by
service_name, the version key andhost.namewithcompareTo: "1d". - What changed:
get-changesfor the service from 1 to 2 hours before the first occurrence. A version first seen just before it is the first suspect. - Scope:
sort: "sessions", orCountDistinctofsecure_session_idwithexact: true.
What goes in the diagnosis
- What triggered: alert, threshold, value, and the error group secure IDs behind it.
- Likely cause: the root frame, the trigger (version, flag, host), and one quoted error message.
- Scope: occurrences, users or sessions, first-seen time.
- Next steps: specific, such as "roll back version X" or "guard nil at handler.go:142".
Common mistakes
- Citing a framework frame as the cause.
- Filtering on
eventinstead ofexception.message. - Counting without a comparison window.