Service health and dependency analysis in CloudWatch Omni
How to answer "is <service> healthy", "give me a health summary of <service>",
"what does <service> depend on and what role does each dependency play", and "which
services call <service>" — grounded in the context graph and confirmed with telemetry,
never from a resource's control plane and never as a bare verdict. (Pure blast radius —
"what breaks if <datastore> goes down" — belongs to context-graph.md,
which owns upstream/blast-radius traversal; this file uses that traversal only to place a
dependency's role and to confirm a caller's health.)
This file is the investigation method that ties three surfaces together. It does not restate them:
- Topology — the dependency map, roles, and direction come from
GetContextGraph. The traversal mechanics, edge semantics, pagination, truncation, and blast-radius/upstream walk live in context-graph.md; this file uses them and cites the roles each edge type assigns. - Metrics (RED / USE) — the exact PromQL, the gauge-vs-counter read, the per-service signal catalog (§5), and the empty-series rules are in ../cloudwatch/promql.md. Do not duplicate a selector here; for a specific resource type (EC2, RDS, Lambda, …) take the resource-specific signals and their aggregation from that §5 catalog rather than the generic USE shape below.
- Failed / slow spans and logs — the SQL over
traces.default/logs.default,durationNano, andstatus['code']are in query/sql-logs-traces.md. - Alarm state — an Omni alert's state (
OK/WARNING/CRITICAL/NODATA) and how to read it withListAlerts/GetAlertare in alerts.md.
Contents
- Service health & dependency question — facts you MUST surface
- Dependency map with roles
- Blast radius — who is affected
- The multi-signal health verdict
- Building the checks from the graph
- Gotchas
Service health & dependency question — facts you MUST surface
"Is <service> healthy," "health summary of <service>," "what does <service> depend
on and what role does each play," and "which services call <service>" are
observability-data questions answered from the telemetry surface and the context graph,
not from the resource's control plane. (A pure "what breaks if <datastore> goes down"
blast-radius question is context-graph.md's.) Answering with the
methodology — the graph call, the signals to check with the PromQL that reads each one
(gauge or counter, per the §5 catalog), the verdict rubric — is an output
contract: surface every item below that applies, whether or not you also pull live
numbers and even if no matching service exists in the account yet. Presenting a verdict
without a telemetry check, or a dependency list without roles, is the single most common
wrong answer here.
- Answer for the type the user named. If the probe finds no such resource, or finds a
different type under that name (the user said RDS and the graph shows DynamoDB), say so
in one line, then still give the full check for the type the user named: its signals,
how each is read, and the verdict rule. Call
GetContextGraphon the named service (nodeFilters.name,depth: 1,includeMetadata: true, a 24-hour window ending at the incident) before writing any query. Do not open with a metric scan of the fleet or adepth: 3sweep. - A dependency list MUST carry each dependency's ROLE, not just its name. Derive the
role from the edge type and direction (next section): a callee service (
CALLSfrom<service>), a datastore (ACCESSESfrom<service>— database, queue, cache), the host / compute it runs on (RUNS_ONfrom<service>), and its callers (edges whosetois<service>). "3+ dependencies with correct roles" is the bar; a bare list of names is not. - A health VERDICT MUST rest on more than one signal, and MUST END with an explicit
verdict word. The minimum shape for a service is: its RED signals — request rate,
error/fault rate (errors over total, not a raw count), and latency including a
p99 (not just an average) — plus alarm state (list the service's Omni alerts with
ListAlerts/GetAlert) plus any live failing edge on the graph; for a resource, the USE signals (utilization, saturation, a headroom signal, errors) plus alarm state. Then state the verdict in one of exactly three words — HEALTHY, DEGRADED, or DOWN — with the evidence behind it. A one-signal answer ("CPU is fine, it's healthy"), or a description of signals with no explicit verdict word, does not meet the bar. With no live numbers yet, end on the verdict rule instead: which readings make it HEALTHY, which DEGRADED, which DOWN. - Never give a verdict from no telemetry. A generic health answer with no metric,
span, or alarm check is the lowest-scoring answer there is. This is not a reason to stop
short: when the resource cannot be found or queried, still give each signal's query and
read (gauge with
avg/max/min, neverrate(); a headroom signal near its limit is DEGRADED even when utilization looks calm) and end on the verdict rule. An empty or zero result is a scoping signal (wrong Region/Space, wrong window, wrong label) — never proof of health. Never conclude "healthy" from a missingerrorCode, an absent metric series, or an empty node list. - RED for services, USE for resources — and take the specific signals from the §5
catalog. A request-serving
SERVICEnode gets Rate / Errors / Duration; aRESOURCEnode (database, host, queue, cache) gets Utilization / Saturation / Errors / headroom. Pick the shape from the node'snodeType/nodeProperties.category, then — for a specific resource type (EC2, RDS, ELB, …) — take the resource-specific signals and their aggregation from the ../cloudwatch/promql.md §5 per-service catalog, not the generic USE list: e.g. an EC2 fleet needsStatusCheckFailed*read withmax(a 0/1 flag; averaging hides a failure),CPUCreditBalancewithmin(burstable only), grouped byInstanceId— those specifics live in §5, not here. - A
REMOTE_SERVICEnode cannot be queried directly. It is an external dependency with no telemetry of its own in this Space, so it gets neither RED nor USE — judge it from the caller-side edge evidence (anerrorCode/ non-2xxhttpStatusCodeon the edge into it, plus the calling service's own spans), per the "a node you cannot query" guidance in context-graph.md. - Build every query from the node's
metadataselectors, not from the graph's display name. WithincludeMetadata: true,metadata.metrics[]gives the metric names and the rawattributesto match,metadata.traces[]/metadata.logs[]give the exact column→value selectors, andalternateNamesgives the rawservice.namevalues a merged node is stored under. Querying by the resolved display name misses a merged node's telemetry. - Say which direction answers the question. Cause of
<service>'s own errors/latency → walk downstream (its dependencies). Who is affected if<service>fails → its callers (upstream). State this before traversing. For the full transitive blast-radius closure, defer to context-graph.md, which owns that walk; here you only need the immediate callers to place their role and confirm their health. - Metrics are PromQL, spans/logs are SQL, alarm state is
GetAlert. Never reach a metric with SQL over ametrics.defaulttable (it does not exist), and never read a verdict off the resource control plane (ecs describe-clusters,dynamodb describe-table,ec2 describe-instance-status).
The dependency-map mechanics, the verdict rubric, and how to assemble the checks are below.
Dependency map with roles
A dependency map answers "what does <service> rely on, and how" — and the how is the
role. What is health-specific here is only the role each edge type assigns; the
traversal itself (inbound vs outbound edges, pagination, truncation) is
context-graph.md's — see its "Reading the graph" section. Roles come
straight from the edge type and direction (edges are directed from → to, and from
depends on to):
| Role | Edge, relative to <service> |
Where it comes from |
|---|---|---|
| Caller (upstream) | edge whose to is <service> |
the reverse query for inbound edges (edgeFilters.to) — a node's own edges never include its callers |
| Callee service (downstream) | CALLS from <service> |
<service>'s own outbound edges, edgeType: CALLS |
| Datastore (downstream) | ACCESSES from <service> |
<service>'s own outbound edges, edgeType: ACCESSES — database, queue, cache |
| Host / compute (runs-on) | RUNS_ON from <service> |
<service>'s own outbound edges, edgeType: RUNS_ON |
Procedure:
GetContextGraphwithnodeFilters.name = <service>,depth: 1,includeMetadata: true, and a 24-hour window ending at the question's time (or now).- Bucket
<service>'s outbound edges byedgeTypeinto the callee / datastore / runs-on roles above, and fetch its callers with the reverse (inbound-edge) query — see context-graph.md for the traversal mechanics. - Report each dependency with its role and, where present, its edge evidence
(
operations[],errorCode,httpStatusCode,lastObservedAt). An edge carrying anerrorCodeor a non-2xxhttpStatusCodeis a proven failing dependency within the window — follow that one first. An edge with no error fields is unreported, not proven healthy. - Raise
depthone hop at a time if the question asks for the transitive set, and confirm the health of a suspect dependency with telemetry before expanding further.
Two hops (depth: 2) is usually enough to answer "what does it depend on and what role
does each play" for a service and its immediate datastores. Go deeper only along a branch
that shows signal.
Blast radius — who is affected
A pure blast-radius question ("what breaks if <datastore> goes down," "who is affected
if <service> fails") is owned by context-graph.md — it is the
reverse (upstream) traversal, and the mechanics (inbound-edge query, node-by-node
pagination, silent hub truncation, collect every page) all live there. Use it for that.
What is health-specific — and all you need here — is the read on an affected caller:
a caller whose edge to the target carries an errorCode or a non-2xx httpStatusCode
is proven-affected; a caller with an edge but no error fields is merely dependent. For a
who-is-affected list, label each caller this way without querying telemetry; offer that as
the next step.
The multi-signal health verdict
A verdict rests on several signals, chosen by whether the node is a service or a resource.
The RED/USE shapes below are only the categories to cover — they are not the
per-resource signal list. For a specific resource type (EC2, RDS, ELB, DynamoDB, …), the
exact metrics, their aggregation, and the identity label to group by come from the
../cloudwatch/promql.md §5 per-service catalog — go there and
use its signals rather than the generic USE list here (which would under-cover a resource
type: an EC2 verdict needs StatusCheckFailed* with max, CPUCreditBalance with min
on burstable, memory/disk from the CloudWatch agent, grouped by InstanceId — §5 names
them, this section does not).
For a SERVICE node — RED:
- Rate — request throughput. From span RED (
traces.span.metrics.calls: CloudWatch'scloudwatch.aws/xray/span-metricsseries are delta,sum_over_time(...[5m]); a collector'sspanmetricsconnectorseries are cumulative,rate(); scope"@instrumentation.@name"to one) or Application Signals request count. - Errors — error/fault rate, derived as
errors / totalover the same window, never a numerator alone. Application SignalsFault(5xx + span-status errors) andError(4xx), or the span error split onstatus['code']. - Duration — latency including a p99, not just an average. From the native
histogram
traces.span.metrics.durationviahistogram_quantile(0.99, sum_over_time(...[5m]))(sum_over_timefor a delta series,rate()for a cumulative one: read__temporality__, since it differs by producer: CloudWatch's span metrics are delta, a collector's spanmetrics connector is cumulative), scoped to the service and to one producer, Application SignalsLatency, or spandurationNanoin SQL (sql-logs-traces.md).
For a RESOURCE node — USE:
- Utilization — CPU / memory / connections (e.g.
CPUUtilization,DatabaseConnections), read as a gauge (avg/max; on the enriched vended surface a delta exponential histogram, sohistogram_avg/histogram_quantile). - Saturation — queue depth,
DiskQueueDepth, message backlog / age, task counts. - Errors — throttles,
5xx, status-check failures (read a 0/1 check withmax, never averaged away). - Headroom — a signal that trends toward a limit and is alerted below a threshold:
FreeStorageSpace,FreeableMemory,CPUCreditBalance(burstable only). A balance near zero while utilization looks flat is throttling, not health.
Across both — always add:
- Alarm state — check Omni alerts on the service with
ListAlerts/GetAlert(alerts.md). AWARNINGorCRITICALstate, or acontributorSummarycount above zero, is direct evidence of a breaching signal;NODATAis itself a finding (the rule ran and saw nothing), not "healthy".OKis one signal, not the verdict. - A live failing edge on the graph — any
errorCode/ non-2xxhttpStatusCodeon an edge into or out of the node, with alastObservedAtat or near the incident (the graph window is 24 hours, so an older error is history).
State the verdict explicitly:
| Verdict | Shape of the evidence |
|---|---|
| HEALTHY | Error rate at baseline, p99 latency within norms, no saturation/headroom signal near a limit, alarms OK, no failing edges in the window — and at least one of these actually queried, not assumed. |
| DEGRADED | One or more signals elevated but the service is still serving: raised error rate or latency, a headroom signal trending toward a limit, an alarm at WARNING, or a proven failing edge to one dependency while others are clean. Name which signal and the number. |
| DOWN | Serving is broken: request rate collapsed to ~0 with errors, alarm CRITICAL, or the node/its critical datastore unreachable. Distinguish "the service is the cause" from "a dependency is down and this service is a victim" using the downstream walk. |
Every verdict cites the evidence: the metric and its value, the span/log query, the alarm state, and the edge. A verdict with no cited signal is the failure this file exists to prevent.
Building the checks from the graph
The metadata block (only with includeMetadata: true) is what makes the queries match
what is actually stored, rather than the graph's normalized name:
- Metrics —
metadata.metrics[]gives each metricname, itspreferredStat, and the rawattributes(service.name,service.namespace,cloud.region, ...) to use as PromQL label matchers. Build the selector per ../cloudwatch/promql.md — do not guess the metric name from the node's display name. - Traces / logs —
metadata.traces[]/metadata.logs[]are per-signal query selectors (OR'ed blocks of AND'ed exact column→value maps). Use them andalternateNamesto write the SQL overtraces.default/logs.defaultper sql-logs-traces.md, always with a`@timestamp`range. - Alarms — the service name (and its
alternateNames) is what you match when listing its alerts (alerts.md).
Read the account and Region for every follow-up query off the node
(nodeProperties.region, nodeProperties.sourceAccountId) — do not assume they match the
credentials you signed with, because a Space can centralize telemetry forwarded from other
accounts.
Gotchas
- A verdict without a telemetry check is the top failure. Health and dependency answers that reason from prose alone score lowest. Always query at least one grounded signal, and state the verdict from it.
- A dependency list without roles is incomplete. The role (callee / datastore / runs-on / caller) is the substance of the answer, and it comes from the edge type and direction — not from the name.
- An empty result is a scoping signal, never a verdict. Empty nodes, zero rows, or an absent metric series most often mean the wrong Region/Space, a window that does not overlap, or a name/label mismatch (see context-graph.md and ../cloudwatch/promql.md empty-series tables). Never report "healthy" from it.
- Hub fan-out is silently truncated and pagination is by node — so a dependency map or affected-caller set built from one page can be incomplete. This mechanic (and its fix) is in context-graph.md; collect every page and raise the per-node edge cap (or query the relationship directly) before treating a map as complete.
NODATAon an alert is a finding, not health. The rule evaluated and saw nothing — investigate why (no telemetry reaching the Space, wrong scope) rather than reading it as a pass.- Absent is not negative. An edge with no
errorCode, a metric with no series, and an alert atOKeach mean "this one signal did not report a problem", not "the service is healthy". The verdict comes from the set of signals. - Do not read health off the control plane.
ecs describe-clusters,dynamodb describe-table,ec2 describe-instance-status, andlist-metricsdescribe configuration, not health. Health is on the telemetry surface. - Use
GetContextGraph, never X-RayGetServiceGraph. If a too-old AWS CLI rejects thecloudwatchomnicommand (invalid choice), or boto3/botocore raisesUnknownServiceErrorfor thecloudwatch-omniservice, that is a client-version issue (upgrade the CLI/SDK), not a reason to switch products.