Alert investigation: logs
Load this file for a log alert: a log count or rate threshold, an error-log spike, a message pattern.
Before you look for a cause
- Replay the firing from
get-alert-firing-history, then withcompareTo: "1w". - Group the matching logs with
search-log-patternsusing the alert's query and the firing window. The top two or three templates usually are the alert. If one template is a known, harmless message, the alert's query needs an exclusion.
Investigation shape
- Templates with counts:
search-log-patternswithenvironmentsand the whole service family inservices(get-keys,key: "service_name",search: "<svc>"), widening aservice_nameclause in the alert's query to the family; addpatternKeys: ["service_name", "message"]to see whether one service or all of them log it. Do not filter to only the alert's service here: a single-serviceservicesmakes every template look like one service and hides a failure shared across them. - Where:
search-aggregationswithproductType: "logs",environmentsand the service family inservices, widening aservice_nameclause in the alert's query to the family (or dropping it), grouped byhost.name,service_nameor the version key, withcompareTo: "1d". A failure logged by every service on one host points at the host. - Verbatim evidence:
search-logsfor two or three examples, withenvironmentsandservices, withfieldsto keep rows small. Follow a line'strace_idintoget-trace. - What changed:
get-changesfor the service.
What goes in the diagnosis
- What triggered: alert, threshold, replayed value.
- Likely cause: the template with its count, one verbatim line with timestamp and host, and the trigger.
- Scope: services, hosts and requests affected.
- Next steps: the fix, or an alert query change.
Common mistakes
- Paging
search-logsto count. Usesearch-log-patterns. - Paraphrasing log lines. Quote them.
- Filtering on an unchecked key; a bad clause is dropped silently.