All skills
aws avatar

/aws-observability

@bb272a8

Builds, configures, debugs, and optimizes AWS observability — operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch: Log Insights, alarms, Dynamic Instrumentation, and Application Signals — instrumenting/onboarding a service to Application Signals with ADOT on EC2/ECS/EKS/Lambda: auto-instrumentation, monitored service, reporting telemetry, ServiceEvents, CI/CD metadata, Terraform/manifest. Also fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation) and driving Omni from a coding agent or skills, and evaluating AI agent quality from traces — on-demand and continuous online scoring of live agent traffic, readback, and custom trace evaluators. For first-time Omni setup — creating a Space, granting access, ingestion, or ADOT instrumentation — use setting-up-cloudwatch-observability. Not for app logging or threat detection.

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/aws-observability

This session only. Nothing lands on disk.

referencescloudwatchsynthetics.md

≈1.7k tokens on demand. Your agent reads this file only when SKILL.md points to it.

CloudWatch Synthetics

Runtime constraints, blueprint compatibility, and common pitfalls for CloudWatch Synthetics canaries.

Contents


Runtime and blueprint compatibility

Blueprint Puppeteer Playwright Python/Selenium Java
Heartbeat Yes Yes Yes No
API canary Yes No Yes Yes
Broken link checker Yes No Yes No
Visual monitoring Yes No No No
Canary recorder Yes No No No
GUI workflow Yes Yes Yes No
Multi checks Yes Yes Yes Yes

Playwright cannot use 4 of 7 blueprints. Java has no browser — API-only.

Family Latest Node/Python X-Ray tracing
syn-nodejs-puppeteer-* 15.0 Node 22 Yes (not with Firefox)
syn-nodejs-playwright-* 6.0 Node 22 Yes (not with Firefox)
syn-python-selenium-* 10.0 Python 3.11 Yes
syn-java-* 1.0 Java 21 Yes

Run aws synthetics describe-runtime-versions for the latest runtime versions.

Deprecated runtimes continue running but you cannot update code or config without upgrading first.


Key flags

CDK:

const canary = new synthetics.Canary(this, 'ApiCanary', {
  // ... standard props ...
  activeTracing: true,              // X-Ray — adds 2.5-7% to run time
  provisionedResourceCleanup: true, // delete Lambda on canary delete
  artifactsBucketLifecycleRules: [{ expiration: Duration.days(30) }], // prevent S3 accumulation
});

// BREACHING — canary not running IS the problem
canary.metricSuccessPercent().createAlarm(this, 'CanaryAlarm', {
  threshold: 90,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.LESS_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.BREACHING,
});

maxRetries (via Schedule.RetryConfig) and dryRunAndUpdate are not exposed in the CDK L2 construct — use CfnCanary escape hatch or CLI.

CLI — alarm on canary success rate:

aws cloudwatch put-metric-alarm \
  --alarm-name my-api-canary-success \
  --namespace CloudWatchSynthetics \
  --metric-name SuccessPercent \
  --dimensions Name=CanaryName,Value=my-api-canary \
  --statistic Average --period 300 \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --threshold 90 --comparison-operator LessThanThreshold \
  --treat-missing-data breaching

CLI — safe update via dry run:

aws synthetics start-canary-dry-run --name my-api-canary --runtime-version syn-nodejs-puppeteer-15.0
aws synthetics get-canary --name my-api-canary --dry-run-id $DRY_RUN_ID
aws synthetics update-canary --name my-api-canary --dry-run-id $DRY_RUN_ID

Key CDK/CloudFormation constraints:

  • ExecutionRoleArn is required — CloudFormation does not auto-create roles (unlike the console)
  • Changing Name triggers replacement (delete + create), causing monitoring gaps
  • Without provisionedResourceCleanup: true, deleting the stack orphans Lambda functions and layers
  • Editing any canary property resets the schedule — next run happens immediately

VPC canaries

Canaries in VPCs must run in private subnets (Lambda ENIs don't get public IPs, even in public subnets).

Internet access (required for uploading metrics to CloudWatch and artifacts to S3):

  • Option A: NAT Gateway in a public subnet + route from private subnet
  • Option B: VPC endpoints — Interface endpoint for monitoring, Gateway endpoint for s3

VPC endpoint policy constraint: The S3 gateway endpoint policy must include s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:PutObject — separate from the IAM role policy.

DNS: Both DNS Resolution and DNS Hostnames must be enabled on the VPC.

Silent failure mode: If the VPC has no internet access and no VPC endpoints, the canary runs but cannot upload metrics or artifacts — it appears as if it never ran.


Common failures

Symptom Cause Fix
"Cannot find module" Wrong ZIP structure Node.js: nodejs/node_modules/<folder>/<file>.js. Python: python/<file>.py
"Unable to fetch S3 bucket location: Access Denied" Missing s3:ListAllMyBuckets on role (must be Resource: "*") Add s3:ListAllMyBuckets, s3:GetBucketLocation, s3:PutObject to execution role
net::ERR_NAME_NOT_RESOLVED in VPC No DNS resolution or no route to AWS endpoints Enable DNS Resolution + DNS Hostnames on VPC; add NAT Gateway or VPC endpoints
"No test result returned" Canary in public subnet Move to private subnet — Lambda ENIs don't get public IPs
Timeout with no artifacts Lambda timeout < canary timeout Ensure Lambda timeout ≥ canary timeout; set canary timeout ≥ 15s for cold starts
Canary stops running DurationInSeconds set to non-zero value Set DurationInSeconds: 0 for continuous running
Can't update canary Runtime deprecated Upgrade runtime first — deprecated runtimes block all config changes
Visual monitoring fails after upgrade Chromium version changed Re-baseline screenshots after runtime upgrades
CORS failures with X-Ray Active tracing adds trace headers triggering preflight Disable active tracing or configure CORS to allow X-Ray headers
SuccessPercent alarm in INSUFFICIENT_DATA Canary timed out — no metric published for that run Use treatMissingData: BREACHING so timeouts trigger the alarm

Limits

Limit Value Consequence
Canaries per region 200 (default, adjustable via Service Quotas) At scale with retries, can exhaust Lambda concurrent execution (1000 default)
Timeout Max 840s (14 min) Cannot be longer than the canary's schedule frequency
Memory 960-3008 MiB (default 1024) Not the standard Lambda 128-10240 range
Canary name Max 255 chars, lowercase alphanumeric plus _ and - Pattern: ^[0-9a-z_\-]+$
Groups 20 per account, 10 canaries/group Cross-region grouping supported
X-Ray tracing Not supported in ap-southeast-3 Also not supported with Firefox browser
Minimum timeout 15 seconds recommended Below this, cold starts cause silent failures
Orphaned resources on delete Lambda, logs, S3, IAM role NOT auto-deleted Set provisionedResourceCleanup: true (CDK) or AUTOMATIC (CFN); manually clean the rest

Source: SKILL.md on GitHub

1 warning6d3 checks · Risk SAFE
  • Gen Agent Trust Hub6d

    This skill provides comprehensive capabilities for AWS observability and debugging, including agent evaluation and dynamic instrumentation. It includes some security considerations, such as the processing of untrusted telemetry data and the use of external scripts for service onboarding, which are handled with a focus on user confirmation and best practices.

  • Socket6d

    2 alerts: gptAnomaly

  • Snyk6d

    Risk: LOW · No issues

Signed by skilld at bb272a8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last week
metadata
{
  "version": "6"
}

README badge

README badge for aws/agent-toolkit-for-aws/aws-observability