/aws-observability

CloudWatch Custom Metrics

Publishing, querying, and managing custom metrics — EMF, PutMetricData, metric filters, and retention.

Contents

• EMF vs PutMetricData
• Embedded Metric Format (EMF)
• PutMetricData API
• Metric filters
• Metric retention
• Dimension design
• Metric math
• EMF constraints

EMF vs PutMetricData

Default recommendation: Use EMF for Lambda and containerized workloads. Use PutMetricData for batch jobs or when you need synchronous confirmation.

Embedded Metric Format (EMF)

JSON structure

{
  "_aws": {
    "Timestamp": 1574109732004,
    "CloudWatchMetrics": [{
      "Namespace": "MyService",
      "Dimensions": [["ServiceName", "Environment"]],
      "Metrics": [
        { "Name": "Latency", "Unit": "Milliseconds", "StorageResolution": 60 },
        { "Name": "RequestCount", "Unit": "Count" }
      ]
    }]
  },
  "ServiceName": "OrderService",
  "Environment": "Production",
  "Latency": 100,
  "RequestCount": 1,
  "RequestId": "abc-123"
}

EMF limits

• Max 100 metrics per MetricDirective
• Max 30 dimensions per DimensionSet (may be empty)
• Dimension value: max 1024 characters, must be string
• Metric value: must be numeric or array of numerics (max 100 values)
• Max log event size: 1 MB
• Namespace: 1–1024 characters, should not start with AWS/
• Timestamp in _aws is required per the EMF spec and JSON schema (milliseconds since epoch). In practice, if omitted, CloudWatch uses the log event's ingestion time — but explicitly setting it is recommended to avoid clock-skew issues.

EMF libraries

For Lambda/containers, use a library that handles EMF serialization (e.g., Lambda Powertools Metrics, aws-embedded-metrics). These libraries manage the _aws metadata block, dimension limits, and metric flushing automatically.

PutMetricData API

Limits

• 500 TPS per account per region (adjustable via Service Quotas) — NOT 150 TPS
• Up to 1,000 MetricDatum items per request
• Up to 150 values per MetricDatum (for percentile statistics support)
• Max 30 dimensions per metric
• Metric name: max 255 characters
• Namespace: max 255 characters, should not start with AWS/

StatisticSets (batch optimization)

Instead of publishing individual data points, aggregate into StatisticSets:

{
  "MetricName": "Latency",
  "StatisticValues": {
    "SampleCount": 100,
    "Sum": 5000,
    "Minimum": 10,
    "Maximum": 200
  },
  "Unit": "Milliseconds"
}

Reduces API calls and cost.

Metric filters

Extract metrics from log events automatically.

• Max 100 metric filters per log group
• Filter pattern: space-delimited terms or JSON property matching
• PutMetricFilter API: 5 TPS
• Metric filter → CloudWatch metric → alarm pipeline is the standard log-to-alert pattern

Example: count 5xx errors from access logs

{ $.statusCode >= 500 }

Publishes a metric with value 1 for each matching log event.

Metric retention

Automatic aggregation cascade

Key insight: You cannot query 1-minute data from 2 months ago. It has been automatically aggregated to 5-minute resolution. High-resolution (1-second) data is only available for 3 hours.

OTel metrics: Only 30 days retention (public preview) — significantly shorter than traditional CloudWatch metrics (15 months).

Metric expiry

• Metrics with no new data for 15 months expire
• Metrics with no data for 2 weeks are not listed by ListMetrics (but still exist)

Dimension design

Note: Each unique dimension combination = separate metric = separate cost.

Anti-patterns

• Do not use requestId, userId, sessionId as dimensions — creates millions of metrics
• Do not publish {InstanceId, InstanceType} and expect to query by InstanceId alone — must publish both combinations separately
• Do not use inconsistent units — metrics with different units are separate data streams

Best practices

• Use low-cardinality dimensions: ServiceName, Environment, Operation, StatusCode
• Use the SEARCH function for cross-dimension queries
• Always specify units consistently
• Audit custom metrics regularly — remove unused ones

Metric math

Combine metrics using expressions in alarms and dashboards.

Functions

SUM, AVG, MIN, MAX, STDDEV, PERIOD, SEARCH, IF, FILL, ANOMALY_DETECTION_BAND

Error rate pattern

errors * 100 / invocations

SEARCH expression (dynamic metrics)

SEARCH('{AWS/Lambda,FunctionName} MetricName="Errors"', 'Sum', 300)

Automatically includes new functions matching the pattern — useful in dashboards and graphs (SEARCH cannot be used in alarms).

Limits

• Max 10 metrics in a metric math alarm expression
• Use Metrics Insights queries for more (max 10,000 metrics, 500 time series returned)
• Metrics Insights alarm data window: 3 hours only
• Max 500 metrics+expressions per dashboard graph

Metric math in alarms — constraints

• FILL can permanently stick an alarm: If a metric is published with slight delay, FILL replaces the missing latest point with the fill value, keeping the alarm in a fixed state. Use M-of-N alarms instead.
• RATE on sparse metrics is unpredictable: The evaluation range varies, causing inconsistent rate calculations. Avoid RATE in alarms on metrics that don't publish every period.
• Anomaly detection restrictions (non-exhaustive): Cannot use more than one ANOMALY_DETECTION_BAND per expression, cannot combine with METRICS() or SEARCH, cannot use high-resolution metrics. See CloudWatch metric math docs for full list.

EMF constraints

• Flush interval affects alarms: Flush EMF logs to CloudWatch at ≤5 second intervals. Longer intervals cause alarms to evaluate partial or missing data. In Lambda (where flush is automatic), use M-of-N alarms to compensate.
• Monitor EMF parsing failures: AWS/Logs namespace publishes EMFValidationErrors and EMFParsingErrors metrics. Check these if metrics aren't appearing.
• Target values cannot be nested: "A.a" matches { "A.a": 1 }, NOT { "A": { "a": 1 } }. Metric and dimension values must be on the root node.
• Multiple DimensionSets multiply metrics: Dimensions: [["Service"], ["Service", "Operation"]] creates 2 metrics per data point, not 1. Libraries like Powertools do this by default.
• Dimension key max 250 chars (per EMF schema); dimension value max 1024 chars.