/aws-observability

Querying metrics in CloudWatch Omni with PromQL

How metric querying works in CloudWatch Omni, and which metric answers which symptom for each AWS service. Metrics in Omni are queried with PromQL, not SQL — the SQL dialect in sql-logs-traces.md reaches logs and traces only. This file covers the decision between the two surfaces, what is and is not queryable, how to discover metric names and labels, the PromQL conventions Omni adds (the @-labels and the __name__ matcher), a per-service catalog of the metrics that matter (unit, gauge vs counter, identity label, derived formulas, dimension traps), and how to read an empty result.

Treat every metric, label, and scope name in this file as vocabulary to recognise, not schema to trust. Confirm the exact name against the account's own data (section 3) before putting it in a query, a dashboard panel, or an alert rule.

Contents

1. Metrics are PromQL, not SQL
2. What is queryable — and what is not
   • Service-health question — facts you MUST surface
3. Discovering metric names and labels
4. Query mechanics
5. Per-service metric catalog
6. Reading results
7. Where PromQL is used

1. Metrics are PromQL, not SQL

Omni has two query surfaces, and the shape of the ask decides which one you use:

Rules that follow from this:

• There is no metrics.default SQL table. A metric request is answered with a raw PromQL expression, not SELECT ... FROM metrics.default — that table does not exist. The one place PromQL text legally appears inside SQL is the promql('<data set>', '<expression>', <step-seconds>) table function (section 4, "Time range binding"; section 7) — never as a bare expression. That embedding is moot for metrics anyway: metric reads through the ad hoc StartTelemetryQuery path are disabled per account, so a metric read there fails either way: a bare PromQL expression is rejected by the SQL parser with ValidationException: Invalid SQL, while SELECT ... FROM metrics.default or a promql() wrapper parses but is rejected with ValidationException: Metrics queries are not supported. For an ad hoc metric read use a dashboard panel or an alert rule instead (section 4, section 7), where the language is set explicitly and the service runs the query. The raw PromQL construction pattern is still the answer to a "how do I query X" metric question (see the rule below) — you just deliver it as panel/alert PromQL, not as a string for the ad hoc query API.
• Nothing SQL-specific applies to metrics. `@record`, `@message`, TABLESAMPLE, EXPLAIN (ANALYZE_FIELDS), and fine-grained access control (FGAC) are logs/traces features. Metrics do not apply FGAC.
• No cross-surface JOIN. There is no metric↔metric or metric↔logs/traces JOIN. To correlate a metric with logs or traces, run the PromQL query and the SQL query as separate steps and line them up on a shared time window and a shared identity (service name, function name, instance id).
• Derive from spans only as a fallback. A per-request latency percentile or error count can also be computed from traces.default (span durationNano, status['code']) in SQL. Prefer the metric when one exists — it is cheaper and pre-aggregated — and fall back to the span derivation only when no metric grounds the signal.

Answer the construction question directly — do not stall or give up. A metric-query how-to ("how do I query / what PromQL shows X for <service>", "which metric tells me if <service> is throttled/slow/erroring") is answered with the PromQL construction pattern, even when a live probe returns no matching metric, or you have no Space in hand. When the metric is not discoverable live, still give the query template — the __name__ matcher, the identity-label / "@instrumentation.@name" (or "@resource.*") scope, and the correct gauge-vs-counter read — and note to confirm the exact names against the account's own data (section 3). Never substitute "no matching metrics found" or a clarifying question for the query; the construction pattern is the answer.

How this differs from CloudWatch metrics

CloudWatch (aws cloudwatch get-metric-data, metric math, Metrics Insights) and Omni PromQL are different query models over different surfaces. Do not translate one into the other mechanically:

For publishing custom metrics, EMF, metric filters, and CloudWatch metric math, see ../../cloudwatch/metrics.md. If the account does not have Omni enabled, or the metric is a CloudWatch-only vended metric, the CloudWatch route is the right answer — not a forced PromQL query.

2. What is queryable — and what is not

Omni serves metrics from one OpenTelemetry-native, PromQL-queryable surface. It carries resource and scope attributes/labels rather than a CloudWatch namespace.

Not queryable: CloudWatch vended metrics that are not enriched

Vended CloudWatch metrics that have not been OTel-enriched, and usage/billing-style metrics such as AWS/Logs IncomingBytes / IncomingLogEvents, are not yet supported in CloudWatch Omni. There is no CloudWatch execution path inside Omni (no GetMetricData or Metrics Insights route behind the PromQL surface).

When the ask resolves to a CloudWatch-only metric:

• Say so plainly: "CloudWatch metric queries are not yet supported in CloudWatch Omni (some vended metrics are not available)." Do not force a PromQL query that can only return empty.
• Do not substitute a log-content scan for a metric. "How much are my log groups ingesting?" is a metric question (IncomingBytes), not a logs.default query.
• If the user only needs the number, offer the CloudWatch route as a separate, non-Omni step (aws cloudwatch get-metric-data; see ../../cloudwatch/metrics.md). If they want it in Omni, the fix is enrichment (below) — for resource types enrichment supports.

OTel enrichment for vended AWS metrics

Vended AWS metrics are CloudWatch metrics by default. OTel enrichment ("vended metrics in PromQL" in the AWS documentation) is the opt-in feature that also projects them onto the PromQL surface:

• Opt-in per account, per region. Enable it in each region you need. The CloudWatch metric is unchanged and stays available through the CloudWatch APIs.
• Ordered prerequisite. First enable resource tags on telemetry for the account (aws observabilityadmin start-telemetry-enrichment, IAM action observabilityadmin:StartTelemetryEnrichment, which provisions the Resource Explorer indexing that enrichment relies on; check with get-telemetry-enrichment-status), then enable OTel enrichment (aws cloudwatch start-otel-enrichment, IAM action cloudwatch:StartOTelEnrichment; check with get-otel-enrichment, disable with stop-otel-enrichment). Both are public operations of their own services — CloudWatch Observability Admin and CloudWatch — not of cloudwatch-omni, and the CloudWatch API itself documents this order. An AWS CLI released before start-otel-enrichment will not have that subcommand; upgrade the CLI rather than look for it under another service.
• Mapping is by metric name + dimensions → resource ARN, not by tags. A supported resource does not need to be tagged to become PromQL-queryable; tags that are present are simply surfaced as @aws.tag.* labels.
• Gated on a supported-resource list. Only supported AWS resource types are enriched.
• Names are preserved. The metric keeps its CloudWatch name (CPUUtilization, 4xxErrors), its dimensions become PromQL labels, and it gains an instrumentation scope label "@instrumentation.@name"="cloudwatch.aws/<service>" (service segment lowercase).

So "why isn't my EC2 / Lambda metric in PromQL?" almost always means one of: enrichment is not enabled for that account/region, resource-tags-on-telemetry was never enabled first, or the resource type is not on the supported list.

Service-health question — facts you MUST surface

A large share of real questions arrive as an operator symptom, not as a query: "is my <service> throttled / slow / erroring / unhealthy," "which of my <service>s are <symptom>," "what's the health of my <service>," "what signals / what should be on a dashboard or view for <service>." These are observability-data questions — answer them from the telemetry surface, not from the resource's control plane. Answering with the methodology — the correct signals, aggregation, scoping, and caveats — is an output contract: you owe the facts below whether or not you also pull live numbers, and even if no matching resource exists in the account. Presenting real metric values, or reporting "none found" and asking for a resource name, without these facts is the single most common wrong answer here. Find the service's entry in the per-service catalog (section 5) and surface every item that applies:

1. Answer in PromQL, scoped. Select each metric by name with the __name__ matcher and scope it with "@instrumentation.@name"="cloudwatch.aws/<service>" — or, for OTLP workloads (e.g. Kubernetes), the double-quoted "@resource.*" labels, which have no vended @instrumentation scope. Metrics are PromQL only — never SQL over metrics.default. The scoped selector is part of the answer even when the request names no product, the account has no Space in reach, or a cloudwatch-omni probe fails: write the selectors, then name the AWS/<Service> namespace as the CloudWatch-side equivalent for an account without enrichment. Do not replace the selectors with console statistic names.
2. Gauge vs counter decides the read — universal, independent of the service. Counters → rate() / increase(); gauges → avg / max / min. Never rate() a gauge; never average or read a raw counter value. On the enriched vended surface the spelling follows the instrument type (section 3): statistic-set metrics are delta exponential histograms, so max is histogram_quantile(1, …), min histogram_quantile(0, …), avg histogram_avg(…), and a percentile histogram_quantile(0.99, …); vended counts are delta Sums read with sum_over_time(…[5m]). On a histogram, max() and min() come back empty while avg() folds the buckets into one histogram instead of a number, and rate() / increase() over a delta Sum return a wrong (not an empty) number — confirm the instrument with the __type__ / __temporality__ labels.
3. Group by the service's identity label (<identity-label>, e.g. the resource-id/name dimension) so each series is attributed to a specific resource.
4. Derived signals must be derived. A ratio such as an error rate is rate(<numerator>) / rate(<denominator>) over the same window — there is no single vended "rate" metric. Never read a numerator alone without its denominator.
5. Vended latency/duration metrics carry no pNN label, but a true percentile is still available — they arrive as delta exponential histograms, so read the percentile with histogram_quantile(0.99, {__name__="<metric>", …}) (section 3/4). Deriving percentiles from span histograms via SQL over traces.default remains an alternative.
6. An empty series is often CORRECT, not a fault — universal. A signal that exists only under a specific configuration (an opt-in feature, an installed agent, a burstable instance type, or a dimensionality the metric is not published at) returns nothing when that configuration is absent. Never claim an empty result proves the symptom is absent ("no datapoints ⇒ not throttled/slow/erroring").
7. Confirm exact metric, dimension, and label names against the account's own data (section 3) before relying on the query.
8. Surface the service-specific traps from the catalog entry (section 5) — units, opt-in-only metrics, dimension tiers, instance-type-only metrics, cold-start exclusions, binary 0/1 metrics that must be read with max. These are what make the answer correct; the universal rules above are necessary but not sufficient.

Asking which signals or panels a <service> needs on a dashboard, view, or panel set is this same question. The dashboard, view, or panel wording does not turn it into a widget-mechanics or layout request, and it is not a live-data request: do not probe for a Space first, and a probe that fails or finds no Space does not change the content. Decide the panel set from the catalog entry (section 5) and write the panels as the scoped PromQL selectors above, in the answer itself — the "@instrumentation.@name"="cloudwatch.aws/<service>" scope and the identity label appear in the reply, not in a follow-up. Add the AWS/<Service> namespace form in one line as the equivalent for an account without enrichment, and do not close by asking which product the customer is on: the customer asked which signals belong on the view, which is the same set either way, and the answer gives both spellings. Neither dashboard file carries these facts: open ../dashboards.md (Omni body and grid) or the CloudWatch dashboards reference only afterwards, and only if the customer wants it built. Do not enumerate the fleet through the service's control plane (ec2 describe-instances, ecs describe-clusters, list-metrics) to decide the content; a resource probe is for grounding names once the panel set is agreed.

Never answer a service-health question by refusing / asking for a resource name first, by presenting live numbers without the discipline above, by calling the service control plane (ecs describe-clusters, ec2 describe-instance-status, dynamodb describe-table, …), or by falling back to the CloudWatch-console / Prometheus mental model (no pNN label on vended gauges; OTLP Kubernetes metrics are k8s.* with "@resource.*" labels, not pod_number_of_container_restarts or kube_* selectors). A dependency / blast-radius question ("what does <service> depend on / what's broken downstream") is answered from the context graph, not here — use GetContextGraph (never X-Ray GetServiceGraph) per context-graph.md, whose "Dependency / blast-radius question — facts you MUST surface" section carries that discipline. If the CLI/SDK errors with an unknown cloudwatch-omni service, that is a client-version issue (upgrade botocore/AWS CLI), not a reason to switch products.

3. Discovering metric names and labels

Metric names, label names, and scope spellings are not guaranteed — they depend on what has been ingested and enriched in this account. A misspelled name does not error; it returns an empty series. Discover before you write.

Method 1 (not available for metrics): ListTelemetryFields. The ListTelemetryFields API enumerates the known fields of a data set for LOGS and TRACES only — METRICS is not an accepted telemetryType in the current service model (the value was removed), so there is no API call that lists metric names. Its fields entries carry only name and children, so even for the telemetry types it does cover it returns no instrument type (Sum / Gauge / Histogram) and no temporality (delta / cumulative). For metrics, take names from the catalog below or from Method 2, and read the instrument type and temporality off the __type__ / __temporality__ labels a raw selector returns (Method 2) — the catalog names the statistic each metric needs (max, min, avg, a rate); the instrument type decides how that statistic is spelled.

Once you know the instrument type, read the aggregation off it:

Method 2: probe with a selector. This is a confirmatory check, not a prerequisite — per sections 1 and 2 you answer most metric questions from the catalog without any live probe, and reach for this only when you already have a Space open and want to confirm a name or its labels before committing it to a dashboard or alert. There is no ad hoc PromQL probe surface: StartTelemetryQuery always parses its queryString as SQL and metric reads through it are disabled per account (section 1; section 4, "Time range binding"), so a selector cannot be run through the CLI/API ad hoc path. Run it as a scratch dashboard panel query (queryLanguage: "promql") or an alert-rule condition (language: PROMQL) instead (section 7), then inspect the label set that comes back for a candidate name over a short window:

{__name__="Invocations", "@instrumentation.@name"="cloudwatch.aws/lambda"}

The returned series show every label the metric actually carries (FunctionName, @aws.account, @aws.region, any @aws.tag.*), which is what you group and filter by — plus the instrument metadata labels __type__ (Gauge, Sum, ExponentialHistogram), __temporality__ (delta / cumulative), __monotonicity__, and __unit__. Read those first: they decide whether the statistic you want is spelled max(), histogram_quantile(1, …), rate(), or sum_over_time() (table above). A raw selector on a histogram-typed metric returns the series with their labels but no plottable samples, which is the tell. __name__ must be an exact match — a regex matcher on __name__ (=~) is rejected with Selector must have a metric name; write one selector per metric name and combine with or (plus label_replace to tag each side) when one panel needs several.

Do not use the SQL discovery tools for metrics — EXPLAIN (ANALYZE_FIELDS) and SELECT \@record`` are logs/traces-only.

Use the names discovery returns exactly — do not shorten, alias, pluralize, or invent. Nouns in the request ("throttles", "5xx", "timeouts") describe what to aggregate, not field names; map each to a real, grounded metric.

4. Query mechanics

Label conventions

Labels on Omni's metrics surface follow an @-prefixed convention:

Quoting rule: any label name that is dotted or @-prefixed must be double-quoted inside the braces — {"@instrumentation.@name"="cloudwatch.aws/ec2"}. The same applies inside by (...): sum by ("@resource.service.name") (...). Bare dimension labels (InstanceId) are written unquoted as usual.

For OTLP / Kubernetes workloads and span RED metrics, the identity is always a "@resource.*" label. Never use a generic Prometheus job= or bare service= selector, and never the underscore k8s_* form — those are ungrounded and return nothing.

Selecting a metric: the __name__ matcher

A PromQL metric name is really the reserved __name__ label. A name that starts with a digit or contains %, ., or spaces is not a legal bare identifier — common for vended AWS metrics (4xxErrors, 4XXError, 5XXError, 4xxErrorRate, EBSIOBalance%, Memory % Committed Bytes In Use) and for every dotted OTel name (traces.span.metrics.calls). Select those with a __name__ matcher inside the braces:

{__name__="4xxErrors", FilterId="EntireBucket"}
{__name__="EBSIOBalance%", InstanceId="<instance-id>"}
{__name__="traces.span.metrics.calls", "@resource.service.name"="<service>"}

The value is an ordinary double-quoted string, so any character is legal there. A plain identifier-safe name (CPUUtilization, http_requests_total) may be written bare (CPUUtilization{InstanceId="..."}), but the __name__ form always works — prefer it for vended metrics so the scope and dimensions read uniformly:

{__name__="Errors", "@instrumentation.@name"="cloudwatch.aws/lambda", FunctionName="<fn>"}
{__name__="CPUUtilization", "@instrumentation.@name"="cloudwatch.aws/ec2", InstanceId="<id>"}

Never write the metric name as a bare token inside the braces (a leading "name", before the label matchers). That form does not parse on this surface. This applies everywhere a metric is selected, including inside rate(...), sum(...), and histogram_quantile(...).

Gauge, counter, histogram — the read decides the function

• Gauge — a level (CPU %, queue depth, free memory, connections, latency of a vended metric). Read with avg / max / min. Never rate() a gauge.
• Counter (cumulative Sum) — accumulates (requests, bytes, errors, invocations). Read with rate() (per-second) or increase() (total over the window). Never read a raw counter value or average it.
• Delta Sum — a per-interval count that does not accumulate. Read with sum_over_time(<m>[w]) (divide by the window for a per-second value); rate() / increase() assume a cumulative counter and misread delta temporality, so they return a wrong — not an empty — number. Enriched vended count metrics (NetworkIn, NetworkOut, ConsumedReadCapacityUnits, …) are delta Sums.
• Enriched vended statistic-set metric — CPUUtilization, StatusCheckFailed*, CPUCreditBalance, Lambda Duration and the other "gauge"-shaped vended metrics reach this surface as delta exponential histograms that carry CloudWatch's statistic set. The catalog's gauge reads still name the right statistic, but spell them as histogram_quantile(1, <m>) for max, histogram_quantile(0, <m>) for min, histogram_avg(<m>) for avg (equivalently histogram_sum(<m>) / histogram_count(<m>)), and histogram_quantile(0.99, <m>) for a percentile — so a true pNN is available here, unlike what a "no percentile label" note might suggest. Reading it with a plain max() or min() yields nothing, and a plain avg() folds the buckets into one histogram rather than a single value. histogram_quantile(1, …) / (0, …) approximate the Max / Min via the top / bottom bucket bound (close to CloudWatch's Maximum / Minimum); intermediate quantiles are bucket-estimated.
• Histogram — traces.span.metrics.duration is a native exponential histogram. Read percentiles with histogram_quantile(0.99, rate({__name__="traces.span.metrics.duration", ...}[5m])) on the base metric name — no _bucket suffix, no by (le).
• Already-per-second gauges (RDS ReadIOPS, ReadThroughput) must not be wrapped in rate() — that double-rates into nonsense.

Applying rate() to a gauge, or averaging a per-period counter, is the most common way to produce a confident-but-meaningless number. When unsure, check the instrument type via discovery (section 3).

Time range binding

Where the query runs decides how time is expressed:

Common functions

Worked examples:

# Lambda error rate per function (enriched vended delta Sums -> sum_over_time totals, then divide)
sum by (FunctionName) (sum_over_time({__name__="Errors", "@instrumentation.@name"="cloudwatch.aws/lambda"}[5m]))
  /
sum by (FunctionName) (sum_over_time({__name__="Invocations", "@instrumentation.@name"="cloudwatch.aws/lambda"}[5m]))

# Request rate per service from span RED metrics
sum by ("@resource.service.name") (rate({__name__="traces.span.metrics.calls"}[5m]))

# p99 latency per service from the native histogram
histogram_quantile(0.99, sum by ("@resource.service.name") (rate({__name__="traces.span.metrics.duration"}[5m])))

# Pods per deployment in a namespace (gauge -> count as-is)
count by ("@resource.k8s.deployment.name") ({__name__="k8s.pod.phase", "@resource.k8s.namespace.name"="<ns>"})

# S3 request error ratio (digit-leading names -> __name__)
sum_over_time({__name__="4xxErrors", FilterId="EntireBucket"}[5m]) / sum_over_time({__name__="AllRequests", FilterId="EntireBucket"}[5m])

5. Per-service metric catalog

Each section names the CloudWatch namespace (for recognition only — the CloudWatch surface is not queryable here) and the identity label to filter and group by. On the PromQL surface an enriched vended metric is reached via "@instrumentation.@name"="cloudwatch.aws/<service>" (lowercase) plus the identity label. Confirm the exact scope and label spelling against discovery.

Two universal rules:

• Gauge vs counter decides the read (section 4).
• On the enriched vended surface, spell the read by the OTLP instrument type, not the CloudWatch shape. No enriched vended metric is a raw gauge or a cumulative counter: every one arrives with delta temporality as either a Sum (counts — NetworkIn, Invocations, request/error counts) or an ExponentialHistogram (levels, latency, utilization — CPUUtilization, Duration, TargetResponseTime, *Latency). The Type and Read as columns below name the CloudWatch-side shape and the statistic you want; spell that statistic per section 3/4 — histogram: max histogram_quantile(1, …), min histogram_quantile(0, …), avg histogram_avg(…), percentile histogram_quantile(0.99, …); delta Sum: a windowed total with sum_over_time(…[5m]), used on both sides of a ratio. So a "gauge … avg/max" row for a vended latency metric is read with histogram_avg / histogram_quantile, and a true percentile is available even where a row says "no percentile label". A plain max() / min() empties on a histogram; rate() / increase() misread a delta Sum and can return an impossible value on a short window, so derive a rate as a sum_over_time ratio. This is spelling only where the surface is enriched — the same metric read through the CloudWatch APIs (non-enriched) keeps its classic gauge/counter statistics.
• An empty series is often correct, not a bug. A signal that exists only under a specific configuration (an agent installed, a burstable instance type, a request-metrics filter, an export enabled) returns nothing when that condition is absent. Read "no data" as "not configured / not applicable here" before treating it as an outage.

Signal taxonomies:

• Request-serving services (ALB, API Gateway, CloudFront, S3 request metrics, Application Signals) → RED: Rate, Errors, Duration.
• Resources / infrastructure (EC2, EBS, RDS, ECS, Lambda concurrency) → USE / saturation: Utilization, Saturation, Errors, plus headroom signals (credit balances, free storage, free memory, queue depth) that trend toward a limit.
• Flow / queue services (SQS, SNS) → throughput (counters) vs backlog and age (gauges).

EC2 — AWS/EC2 (+ CWAgent), identity InstanceId

• CPUCreditBalance is read with min and exists only on burstable (T-family) instances — an empty series on other types is correct, not a fault. On a burstable instance a balance near 0 while CPU sits at the baseline is throttling: the hypervisor is capping the instance, so a flat "healthy-looking" CPU chart is the symptom, not reassurance. Say so whenever a credit panel is on a health view.

• Memory and disk-space metrics exist only in CWAgent; an empty guest-OS chart usually means "no agent," not "no data" — the hypervisor cannot see guest RAM. Memory utilization is not available by default from AWS/EC2.

• StatusCheckFailed* is a per-period 0/1 flag, not a level: read it with max — spelled histogram_quantile(1, …) on the enriched surface — because an average over a window that contained one failed check rounds the failure away.

• The family is four metrics, and each points at a different owner — say which when you chart them: StatusCheckFailed (either check failed), StatusCheckFailed_System (the AWS host hardware or network underneath the instance — nothing to fix in the guest; stop/start moves it to a healthy host), StatusCheckFailed_Instance (the customer's own OS, boot, or network configuration inside the instance), and StatusCheckFailed_AttachedEBS (an attached volume is unreachable). A panel that shows the parent alone cannot tell the operator which side to look at.

• DiskRead* / DiskWrite* count instance-store I/O only. The EBS side of an instance's I/O is the AWS/EBS family keyed by VolumeId (next entry), not an AWS/EC2 metric.

• A health dashboard for a group of EC2 instances is built from this table, and the answer is the selectors themselves — written into the reply, not offered after the customer picks a product, and not replaced by CloudWatch console statistic names. Every selector carries the "@instrumentation.@name"="cloudwatch.aws/ec2" scope and groups by InstanceId; state that scope in the answer and write the selectors out:

  # S = "@instrumentation.@name"="cloudwatch.aws/ec2"; every result already carries InstanceId
  histogram_quantile(1, {__name__="CPUUtilization", "@instrumentation.@name"="cloudwatch.aws/ec2"})              # max CPU per instance
  histogram_avg({__name__="CPUUtilization", "@instrumentation.@name"="cloudwatch.aws/ec2"})                     # avg CPU per instance
  histogram_quantile(1, {__name__="StatusCheckFailed_System", "@instrumentation.@name"="cloudwatch.aws/ec2"})   # 0/1 per instance
  histogram_quantile(1, {__name__="StatusCheckFailed_Instance", "@instrumentation.@name"="cloudwatch.aws/ec2"})
  histogram_quantile(1, {__name__="StatusCheckFailed_AttachedEBS", "@instrumentation.@name"="cloudwatch.aws/ec2"})
  histogram_quantile(0, {__name__="CPUCreditBalance", "@instrumentation.@name"="cloudwatch.aws/ec2"})           # min credits; burstable only
  sum by (InstanceId) (sum_over_time({__name__="NetworkIn", "@instrumentation.@name"="cloudwatch.aws/ec2"}[5m])) # bytes per 5m; / 300 for B/s
  avg by (InstanceId) ({__name__="mem_used_percent", "@instrumentation.@name"="cloudwatch.aws/ec2"})   # CloudWatch agent only; a gauge — check __type__ first

  These are the spellings that return usable series on an enriched Space: max by (InstanceId) (...) returns nothing on the histogram metrics, and rate(...NetworkIn...) misreads the delta Sum and returns a wrong number rather than the per-5m total (section 3). __name__ takes no regex — one selector per check type, combined with or when one panel needs all three. The same names exist under the AWS/EC2 / CWAgent namespaces on the CloudWatch side for an account without enrichment; mention that as the fallback, not as the answer. The Omni dashboard body for this set is the EC2 fleet archetype in ../dashboards.md.

• Correlate with: EBS (volume side of the same I/O), ALB (instances as targets), RDS (app tier driving the DB).

EBS — AWS/EBS, identity VolumeId

• Xen-era latency is derived: per-op read latency ≈ rate(VolumeTotalReadTime[5m]) / rate(VolumeReadOps[5m]) (x1000 for ms) — an average only, never a percentile. On Nitro, VolumeTotalReadTime is documented as not relevant; use the VolumeAvg* gauges.
• BurstBalance is empty on gp3/io1/io2 (correct).
• Correlate with: EC2 (instance side), RDS (its storage behaves like an EBS volume — a write-latency spike with healthy CPU points at storage).

Lambda — AWS/Lambda, identity FunctionName

• Error rate is derived: sum_over_time(Errors[5m]) / sum_over_time(Invocations[5m]) (section 4 example) — Errors and Invocations are enriched delta Sums, so a rate()/rate() ratio can read a wrong (even impossible) value on a short window; the windowed-total ratio is stable. Do not read Errors alone without the denominator.
• Grouping UnreservedConcurrentExecutions by function returns nothing (correct).
• Cold starts are not a metric (Duration excludes cold-start time) — cold-start and error detail live in the function log group /aws/lambda/<function>, a SQL log query.
• A true pNN for Duration is available on the enriched surface — it arrives as a delta exponential histogram (section 3/4), so histogram_quantile(0.99, {__name__="Duration", "@instrumentation.@name"="cloudwatch.aws/lambda", FunctionName="<fn>"}) returns a real p99. Deriving the percentile from spans over traces.default in SQL stays a valid alternative when the function's metric is not enriched.
• Correlate with: API Gateway / ALB (invokers), SQS / DynamoDB (event sources).

DynamoDB — AWS/DynamoDB, identity TableName

• A SystemErrors alert scoped to TableName only never has data — include Operation.
• Compare consumed against provisioned capacity for saturation.
• Correlate with: Lambda (pair throttles with function errors), Application Signals.

ECS — AWS/ECS and ECS/ContainerInsights, identity ClusterName (+ ServiceName)

• Split the tiers by dimension: service-tier series carry ClusterName and a non-empty ServiceName; cluster-tier series carry ClusterName only. Filtering the wrong one blends or empties the series.
• Fargate emits no reservation metrics; GPUReservation also needs GPU instances (both correct-empty). Reservations are not under Container Insights.
• Correlate with: ALB (in front), EC2 (container instances for EC2 launch type).

RDS — AWS/RDS, identity DBInstanceIdentifier

• Best chart: CPUUtilization overlaid with DatabaseConnections. CPU tracking connections is an app problem; CPU high at flat connections is a query/plan problem.
• RDS has no *TotalTime metrics — latency is raw, unlike EBS.
• Logs require export: /aws/rds/instance/<id>/<logType> (error / slowquery / audit / general / postgresql); the error log is free-form.
• Correlate with: EBS (storage layer), EC2 (app tier).

Application Load Balancer — AWS/ApplicationELB, identity LoadBalancer

• The names carry a _Count suffix — HTTPCode_Target_5XX alone returns nothing.
• An ELB-5xx spike with flat target-5xx points at the LB / health-check path (e.g. no healthy target), not the app.
• The LoadBalancer label value is the app/<name>/<hash> segment, not the full ARN. ALB (AWS/ApplicationELB) ≠ NLB (AWS/NetworkELB) ≠ CloudWatch (AWS/ELB).
• Fractional host counts are normal from flapping.
• Correlate with: EC2 (instance targets), Lambda (Lambda targets).

SQS — AWS/SQS, identity QueueName

• A low-depth queue with high oldest-age is a poison pill or stalled consumer.
• Total enqueued ≈ Visible + NotVisible + Delayed.
• FIFO queues share the same metrics. A dead-letter queue is just another QueueName — rising Visible on a DLQ means the source queue is failing processing.
• Correlate with: SNS (fan-out source), Lambda (event-source-mapping consumer — pair age/backlog with function Errors / Throttles / Duration).

SNS — AWS/SNS, identity TopicName

• NumberOfNotificationsFailed is a delivery failure — a failed Publish call surfaces in API 4xx/5xx, not here.
• Do not compute delivered / published as a success rate: one publish fans out to N subscribers, so delivered normally exceeds published.
• Correlate with: SQS (subscriber queue depth / DLQ), Lambda (subscribed functions).

S3 — AWS/S3, identity BucketName (two families)

• Storage metrics look flat or broken at short ranges because emission is daily. Omitting StorageType returns multiple or no series.
• Request metrics exist only where a request-metrics filter is configured; omitting FilterId double-counts. An empty request overview is a config gap, not a bug.
• Error ratio: see the S3 example in section 4.
• Correlate with: CloudFront (most public reads hit the CDN and never reach S3 request metrics), ALB (same RED grammar).

API Gateway — AWS/ApiGateway, identity varies by flavour

• Using the wrong identity label (ApiName vs ApiId) returns an empty series. WebSocket APIs do not emit 4xx / 5xx / Count / Latency.
• Derived: gateway overhead = Latency - IntegrationLatency. High Latency with flat IntegrationLatency implicates the gateway/config; the reverse implicates the backend.
• CacheHitCount / CacheMissCount are empty unless caching is enabled (correct). Method/route-level metrics need "detailed metrics" enabled. Access/execution logs are opt-in with no fixed group name.
• Correlate with: Lambda (pair IntegrationLatency with Duration), Application Signals.

CloudFront — AWS/CloudFront, identity DistributionId (+ fixed Region="Global")

• 4xx (client / cache-miss 403/404) vs 5xx (origin-fetch) split the headline error rate.
• Metrics live only in us-east-1 under Region="Global" regardless of edge location — query us-east-1; never emit another region matcher for CloudFront.
• Cache-hit-rate and per-edge metrics are opt-in (paid). Access logs go to S3 or a stream, not CloudWatch Logs.
• Correlate with: S3 (pair 5xxErrorRate with S3 5xxErrors), ALB origin (target 5xx).

Application Signals — AWS/ApplicationSignals, identity Service (SLOs: SloName)

• The names are Fault / Error, not FaultCount / ErrorCount. Service dimensions are Service, Operation, Environment; dependency metrics add RemoteService / RemoteOperation.
• Availability = (1 - Fault / Total) x 100, where Total is the request count (classically SampleCount(Latency)).
• This is the entry point for "how is <service> doing?" when the entity is an Application Signals service or SLO — it aggregates across the infrastructure services above.
• Correlate with: the infrastructure behind the span (EC2/ECS/Lambda), and the traces themselves for the offending span path.

Kubernetes / EKS (OTLP) — identity "@resource.service.name" (RED), "@resource.k8s.*" (USE)

OTLP-ingested Kubernetes workloads have no CloudWatch namespace and no vended-AWS instrumentation scope. Group and filter by the double-quoted "@resource.*" labels.

• This is the same span-derived layer the service map uses.
• An empty series usually means the workload is not exporting that OTLP metric (no collector/exporter for it), not that usage is zero.
• Correlate with: Application Signals (application-scoped RED of the same spans), node / EC2 metrics for host-level saturation.

6. Reading results

Empty series: expected vs real

Before treating "no data" as an outage, check whether the metric can exist here:

An empty result is not a reason to fabricate a number or switch to a log scan; it is a reason to re-ground the name (section 3) or explain the missing configuration.

Common errors and their causes

7. Where PromQL is used

PromQL is the metric language everywhere in Omni, not only for ad hoc queries:

• Dashboards — ../dashboards.md. Each explore panel takes a queryString, an explicit queryLanguage ("promql" for metrics, "sql" for log/trace content), autoRun: true, and promqlQueryOptions.step in seconds. Panel expressions are windowless; the @-labels are double-quoted PromQL and therefore JSON-escaped once more inside the queryString. The dashboard reference includes per-archetype panel templates (Lambda, EC2, ALB, RDS, Kubernetes) whose PromQL follows the conventions in this file.
• Alerts — ../alerts.md. A condition on a metric must use language: PROMQL; the expression carries its own range selector ([5m]) and no threshold comparison. Metric alerts on a gauge, a counter rate, or a histogram_quantile percentile all go through this path; SQL alerts reach logs and traces only.
• Ad hoc queries — StartTelemetryQuery (queryString + sessionId) and GetTelemetryQueryResults carry no language or time-range parameter: the query string is always SQL, PromQL is accepted only embedded as promql('<data set>', '<expression>', <step-seconds>) with the window bound by that relation's @timestamp predicate, and metrics through this path are disabled per account at GA (section 1; section 4, "Time range binding"). Prefer a dashboard panel or an alert rule.
• Views — views.md are SQL over logs and traces; they do not wrap PromQL.

For the concepts behind Spaces, data sets, and access — which govern what an account's PromQL can see — see ../concepts.md.