Investigating with traces
Load this file when the investigation concerns request flow, latency, failed spans, or which service in a chain is at fault.
When to reach for traces
- A route or operation got slow or started failing
- You have a trace ID (from a log line, an error occurrence or a session) and want the full request
- You need to know which span owns the time, or which downstream call failed first
- You want to know whether a flag variation, version or host is tied to the failures
Size the problem with search-aggregations (productType: "traces") or search-service-map first; read individual traces once you know which slice is bad.
Find the slow or failing slice
- Service and route health:
search-service-mapwithserviceandinclude: ["health"]returns requests, errors, error rate and p50/p95/p99 for the service and perhttp.route. It covers services with server spans; for queue consumers and workers, aggregate the consumer span byspan_nameinstead. - Endpoints: HTTP middleware often names server spans only by method (
GET). Filterspan_kind=Serverand group byhttp.route, notspan_name. - Breakdown against a baseline: group by
host.name, the version key, orhttp.routewithcompareTo(see step 4 inSKILL.md). For latency, reportP95ofdurationand rank groups byCount. - Time per child span: to see which child operation owns a parent's time, filter
span_name=(childA OR childB OR childC)and aggregateSumofdurationgrouped byspan_namewithexact: true; compare each child's sum with the parent's. Read one trace withget-tracefirst to learn the child span names. - Flag variation: spans carry flag evaluations as
feature_flag.keyandfeature_flag.result.value(alsofeature_flag.result.variationIndex; check withget-keys search=feature_flag). Filterfeature_flag.key=<flag>and group byfeature_flag.result.value, with and withouthas_errors=true, to see whether one variation holds the failures.search-flag-evaluationswithflagKeyand nosessionSecureIdcounts the flag's evaluations per value across the project.
Pull traces (search-traces)
startDateis required; a raw search covers at most 24 hours.- Pass
environmentsandservices(exact values fromget-keys, see step 1 inSKILL.md), onsearch-aggregationstoo.view: "traces"rejects them: putenvironment=<env> service_name=<svc>inqueryinstead. - Slowest:
sortColumn: "duration",sortDirection: "DESC",limit5 to 10. It reads a sample; for the true slowest, getMaxofdurationwithexact: truefromsearch-aggregations, then searchduration>=<that value>. - Failing:
has_errors=trueplus the slice filter. - One row per trace:
view: "traces"(takes onlyquery, the window,limitandcursor, so scope it inquery). fieldspicks attributes and keeps rows small;fullAttributesreads long values whole.- Pages are at most 50 rows; follow
page.next_cursor(not available withsortColumn). - Within one trace:
span(A) THEN span(B)andspan(A) THEN no_span(B)match order and absence;parent_span(...),child_span(...),root_span(...)andany_span(...)match other spans in the same trace.parent_span_id NOT EXISTSselects root spans.
Read one trace (get-trace)
Pass traceId and, whenever you know a span time, timestamp (the span's timestamp from search-traces or a log line). The result is a span tree with each span's service, duration and status, plus the trace's error groups.
- For large traces,
onlyServiceEntrySpans: truekeeps one span per service hop and every error span. expandSpanIdreturns one span with its full attributes (route, exception details) and its subtree uncollapsed.- Find where the time goes. A span's own time is its duration minus the time covered by its children. A 2.3 s span whose two children take 0.25 s and 0.49 s spent about 1.5 s in its own code; blaming the children would be wrong.
get-tracecomputes this for you on the full tree, beforeonlyServiceEntrySpansormaxTokenstrimming:self_mson each span: its duration minus the union of its children's intervals.critical_path: up to 15 spans, walking back from the root to the child that finished last, each withduration_ms,self_msandcritical_ms(the time it held the request up).top_self_time: the 5 spans with the mostself_ms, each withshare_of_root. Name the cause fromtop_self_timeandcritical_path, not from the longest child. On a server without these fields, subtract the children's durations from the parent's by hand.
- Find where the failure started. Error spans propagate upward; the lowest error span in the tree, usually in a downstream service, is the origin.
- If the trace is incomplete (a Kafka consumer whose parent is the producer span, or sampled-out spans), say so; do not infer timing from gaps.
Logs and errors for a trace
search-logswithquery: "trace_id=<id>",environmentsonly (noservices: the request crosses services, and the downstream services' lines must appear) and a window of a few minutes around the span returns every log line of the request.- The trace's error groups come back from
get-trace; open one withget-error-group.
Interpreting results
- Duration is in nanoseconds.
2500000000is 2.5 s. - One slow trace is an example, not a pattern. Confirm with an aggregation that the slow span is slow across many traces before recommending a fix.
- Compare like with like. For a regression, compare the same
span_nameand route withcompareTorather than eyeballing one old trace. - Cite the trace ID, service and span name for each claim.
Common mistakes
- Searching raw traces over more than 24 hours. Aggregate instead.
- Passing
serviceswithoutenvironments, or listing every environment. Name the one the question is about; the query then reads the service's rows in that environment and is less often sampled. - Quoting a sampled P99 over a few dozen rows. See "Sampling and percentiles" in
SKILL.md. - Blaming the child span with the longest duration when the parent's own time is larger.
- Guessing the version or status-code key. Take the version key from
get-changesand check status keys withget-keys.