Investigating with error groups
Load this file when the investigation concerns exceptions, crashes, stack traces, or a new error.
When to reach for error groups
- The user reports a crash, an exception or a regression
- You need an error's first-seen time, frequency, stack, or affected users
- You have an error group ID and want its occurrences, traces and sessions
An error group collapses many occurrences by stack similarity. A group can still hold unrelated messages, so read its occurrences before concluding.
Find groups (search-error-groups)
startDateis required. OnlyOPENgroups are returned unlessstatesays otherwise (ANYfor every state).- Filter on
exception.message,exception.type,service_nameandenvironment. Do not filter onevent: it matches only the group's original title.query: exception.message="Cannot use 'in' operator*" AND service_name=gonfalon-web sort:spikefor groups growing fastest (count in the window divided by the count in the previous window of the same length;in_windowaddsbaseline_count,spike_ratioandnew_in_window, and the response givesspike_baseline_window),countfor the biggest in the window,first_seenfor new groups,sessionsfor user impact (backend-only groups have no sessions and are left out).- Each group's
in_windowblock counts only the matching occurrences in your window.
Read one group (get-error-group)
secureIdfromsearch-error-groups. It returns the state, the stack and one occurrence (the latest, or the latest instartDate..endDatethat matchesquery) with itstrace_idand session.include: ["distribution", "trend"]adds context overstartDate..endDate(default the last 7 days, echoed ascontext_window):distributiongives the top values of browser, os_name, environment, service_name and service_version with count and percent, so "92 % on version X" or "only Safari" shows up in one call;trendgives occurrence counts in up to 60 buckets with thepeakbucket; the first non-zero bucket is the group's onset in that window. Not withoccurrences.- Frames carry
in_app,link(to the source on GitHub when connected) andsource_map_error;top_frameaddscode_beforeandcode_after. - Step through occurrences with
next_idandprevious_id, or setoccurrences(withstartDate) to list them: each row has timestamp, service, version and session. - Pivot:
_pivotsholds readyget-traceandsearch-logs(trace_id=<id>, ±1 h) arguments for the occurrence; pass them as they are. Without_pivots, build them from the occurrence'strace_id. search-logsrows at error level and above carryerror_group_secure_idanderror_object_idwhen the log matches an error object, which links a log line straight to its group.
Typical patterns
- New regression:
search-error-groupswithsort: "spike"(or"first_seen") over the onset window, thenget-changesfor the service: did a version appear just before the first occurrence? - What is over-represented:
get-error-groupinclude: ["distribution"]for one group, orsearch-aggregationswithproductType: "errors", grouped byservice_name, the version key orhost.name, withcompareTo: "1d". For a flag, filterfeature_flag.key=<flag>and group byfeature_flag.result.value. - Blast radius:
sort: "sessions", orsearch-aggregationsCountDistinctofsecure_session_idfor the group's filter withexact: true. - Stack analysis: the originating frame is the deepest frame in application code, not framework code. If it is in a third-party dependency, say so.
Interpreting results
- Scope first, cause second. One user in a rare path is different from 30 % of traffic.
- Onset vs standing. Compare the group's count with
compareTo; a group that was just as busy last week is not this incident's cause. - Cite the error group secure ID, the occurrence count, first-seen time, and one trace ID.
Common mistakes
- Filtering on
event. Useexception.message. - Reporting "many errors" without a number and a comparison window.
- Treating backend errors that are only logged (not recorded as error groups or error spans) as absent. Check error logs too.