/aws-observability

Distributed Tracing: X-Ray and ADOT

X-Ray SDK is in maintenance mode. Use ADOT (OpenTelemetry) for all new projects.

Contents

• ADOT vs X-Ray SDK
• Trace structure
• Annotations vs metadata
• Sampling rules
• ADOT collector configuration
• Instrumentation patterns
• Migration constraints
• Common mistakes

ADOT vs X-Ray SDK

Migration path: AWS provides migration guides from X-Ray SDK to OpenTelemetry SDK. The CloudWatch agent now also supports sending traces to X-Ray — no separate daemon needed.

Trace structure

• Trace — collection of all segments from a single request, identified by trace ID
• Segment — JSON document with a 64 KB documented limit representing work done by a service. Do not exceed this; behavior above 64 KB is undocumented and may change.
• Subsegment — granular detail within a segment (downstream calls, custom code blocks)
• Inferred segment — generated by X-Ray from subsegments for uninstrumented downstream services

Trace ID format

X-Amzn-Trace-Id: Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1

Format: 1-{8 hex epoch}-{24 hex unique}. W3C trace IDs are supported (reformatted).

Retention

• Trace data: 30 days (not configurable)
• Service graph: 30 days

Annotations vs metadata

Rule of thumb: If you need to search for it → annotation. If you just need to store it → metadata.

WARNING: 50 annotations per trace is a hard limit. Plan your annotation schema carefully.

Sampling rules

Default rule

• Reservoir: 1 request per second (shared across all instances)
• Rate: 5% of additional requests
• Conservative default to control costs

Rule evaluation

• Rules evaluated in ascending priority order (1–9999, lower = higher priority)
• Default rule priority = 10000 (always last)
• First matching rule wins

Rule parameters

Parent-based sampling (critical concept)

Sampling decision is made once by the root service. Downstream services honor the upstream decision regardless of their own rules. Custom rules only apply where no sampling decision exists yet.

Adaptive sampling (newer)

• SamplingRateBoost — auto-increases rate during anomalies
• MaxRate — ceiling for boosted rate
• CooldownWindowMinutes — prevents continuous boosts (recommended when SamplingRateBoost is configured)

ADOT collector configuration

Architecture

[Receivers] → [Processors] → [Exporters]

CloudWatch + X-Ray pipeline

The 0.0.0.0 receivers below listen on every interface with no TLS and no authentication. Bind them to 127.0.0.1 when the senders are on the same host; otherwise it is recommended to restrict the OTLP 4317 / 4318 ports via security groups or host firewall and to avoid co-locating untrusted workloads. This guide does not apply those controls, so assess and configure them for your environment.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

EKS DaemonSet deployment

resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 250m
    memory: 100Mi

Cardinality prevention (three-layer defense)

1. OTel SDK level: Don't emit high-cardinality attributes (ContainerID, CustomerID, RequestID)
2. ADOT Collector level: Use Filter Processor to drop metrics by name/attribute
3. Backend level: Use backend-specific dimension filtering (CloudWatch: dimension_rollup_option + metric_declarations; Prometheus: metric_relabel_configs)

Filter as early as possible in the pipeline to reduce cost and cardinality.

Instrumentation patterns

Lambda: enable active tracing (CDK)

import { Tracing } from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,
});

API Gateway: enable tracing

const api = new apigateway.RestApi(this, 'MyApi', {
  deployOptions: {
    tracingEnabled: true,
  },
});

Or via CLI: aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true

Trace-log correlation

Inject trace ID into application logs for cross-pillar correlation:

import logging
from opentelemetry import trace

ctx = trace.get_current_span().get_span_context()
trace_id = format(ctx.trace_id, '032x')
logging.info("Processing request", extra={"trace_id": trace_id})

Migration constraints (X-Ray SDK → OTel)

Annotations require explicit opt-in

In OTel, all span attributes become X-Ray metadata by default. To make an attribute a searchable X-Ray annotation, add its key to the aws.xray.annotations list:

span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")

Without this, you lose all annotation-based filtering after migration.

Centralized sampling requires a proxy

The ADOT collector config must include the awsproxy extension (or use the CloudWatch agent as a proxy) for X-Ray centralized sampling rules to work. Without a proxy, the SDK falls back to a default local rule (1 req/sec + 5%):

extensions:
  awsproxy:
    endpoint: 127.0.0.1:2000
service:
  extensions: [awsproxy]

SDK env vars: OTEL_TRACES_SAMPLER=xray and OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000

Centralized sampling language support: Java, .NET, Python, Node.js (ADOT). Vanilla OTel SDK: Java, .NET, Go.

Mixed propagation during incremental migration

OTel defaults to W3C Trace Context; X-Ray SDK uses X-Ray trace header. During migration, configure both:

OTEL_PROPAGATORS=xray,tracecontext

Without this, traces break at service boundaries between old and new instrumentation.

Port conflict: stop X-Ray daemon before starting ADOT

Both use port 2000. Running both simultaneously causes silent data loss.

Lambda ADOT layer adds cold start latency

ADOT Lambda layers increase memory usage and cold start time. For latency-sensitive functions where you don't need OTel's multi-backend capabilities, X-Ray SDK may still be preferable.

W3C trace ID version requirement

ADOT Collector 0.34.0+ (X-Ray Exporter 0.86.0+) is required to accept W3C-format trace IDs. Older versions silently reject them.

Common mistakes

1. Using X-Ray SDK for new projects — Maintenance mode. Use ADOT/OpenTelemetry.

2. Storing searchable data as metadata — Metadata is NOT indexed. Use annotations for data you need to filter by.

3. Exceeding 50 annotations per trace — Hard limit. Plan your annotation schema.

4. Not stripping X-Amzn-Trace-Id from untrusted requests — Users can inject trace IDs or sampling decisions.

5. Default sampling for all services — 1 req/sec + 5% is too conservative for low-traffic services (may miss issues) and too aggressive for high-traffic (unnecessary cost). Tune per service.

6. StepFunctions tracing overrides Lambda — When StepFunction tracing is enabled, downstream Lambda tracing is always enabled regardless of Lambda's own config.

7. Cross-account tracing — Trace IDs propagate naturally across accounts, but unified cross-account viewing requires CloudWatch Observability Access Manager (OAM) setup with monitoring/source account links.