All skills
aws avatar

/aws-observability

@bb272a8

Builds, configures, debugs, and optimizes AWS observability — operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch: Log Insights, alarms, Dynamic Instrumentation, and Application Signals — instrumenting/onboarding a service to Application Signals with ADOT on EC2/ECS/EKS/Lambda: auto-instrumentation, monitored service, reporting telemetry, ServiceEvents, CI/CD metadata, Terraform/manifest. Also fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation) and driving Omni from a coding agent or skills, and evaluating AI agent quality from traces — on-demand and continuous online scoring of live agent traffic, readback, and custom trace evaluators. For first-time Omni setup — creating a Space, granting access, ingestion, or ADOT instrumentation — use setting-up-cloudwatch-observability. Not for app logging or threat detection.

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/aws-observability

This session only. Nothing lands on disk.

referencescloudwatchappsignals-guidesecs-java.md

≈1.8k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Enable AWS Application Signals for Java on ECS

Overview

This guide provides complete steps to enable AWS Application Signals for ECS services (both EC2 and Fargate launch types), including distributed tracing, performance monitoring, and service mapping.

Prerequisites

  • Services running on ECS (EC2 or Fargate launch types)
  • Applications using Java language

Implementation Steps

Constraints: You must strictly follow the steps in the order below, do not skip or combine steps.

Step 1: Setup CloudWatch Agent Task

1.1 Add CloudWatch Agent Permissions to ECS Task Role
const taskRole = new iam.Role(this, 'EcsTaskRole', {
  assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('AWSXRayDaemonWriteAccess'),
    iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentServerPolicy'),
  ],
});
1.2 Create CloudWatch Agent Log Group
const cwAgentLogGroup = new logs.LogGroup(this, 'CwAgentLogGroup', {
  logGroupName: '/ecs/ecs-cwagent',
  removalPolicy: cdk.RemovalPolicy.DESTROY,
  retention: logs.RetentionDays.ONE_MONTH,
});
1.3 Add CloudWatch Agent Container to Each Task Definition
const cwAgentContainer = taskDefinition.addContainer('ecs-cwagent-{{SERVICE_NAME}}', {
  image: ecs.ContainerImage.fromRegistry('public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest'), // Use latest. ServiceEvents requires 1.300070.0+ (or 1.300069.0+).
  essential: false,
  memoryReservationMiB: 128,
  cpu: 64,
  environment: {
    CW_CONFIG_CONTENT: JSON.stringify({
      "traces": {
        "traces_collected": {
          "application_signals": {}
        }
      },
      "logs": {
        "metrics_collected": {
          "application_signals": {}
        }
      }
    }),
  },
  logging: ecs.LogDrivers.awsLogs({
    streamPrefix: 'ecs',
    logGroup: cwAgentLogGroup,
  }),
});

Step 2: Add AWS Distro for OpenTelemetry Zero-Code Auto-Instrumentation to Main Service

2.1 Add Bind Mount Volumes to Task Definition
const taskDefinition = new ecs.FargateTaskDefinition(this, '{{SERVICE_NAME}}TaskDefinition', {
  // Existing configuration...
  volumes: [
    {
      name: "opentelemetry-auto-instrumentation-java"
    }
  ],
});
2.2 Add ADOT Auto-instrumentation Init Container
const initContainer = taskDefinition.addContainer('init', {
  image: ecs.ContainerImage.fromRegistry('public.ecr.aws/aws-observability/adot-autoinstrumentation-java:v2.28.2'), // Minimum version for ServiceEvents. Check ../application-signals-onboarding.md for how to query the latest version.
  essential: false,
  memoryReservationMiB: 64,
  cpu: 32,
  command: ['cp', '-a', '/javaagent.jar', '/otel-auto-instrumentation-java/javaagent.jar'],
  logging: ecs.LogDrivers.awsLogs({
    streamPrefix: 'init-{{SERVICE_NAME}}',
    logGroup: serviceLogGroup,
  }),
});

initContainer.addMountPoints({
  sourceVolume: 'opentelemetry-auto-instrumentation-java',
  containerPath: '/otel-auto-instrumentation-java',
  readOnly: false,
});
2.3 Configure Main Application Container OpenTelemetry Environment Variables
const mainContainer = taskDefinition.addContainer('{{SERVICE_NAME}}-container', {
  // Existing configuration...
  environment: {
    // Existing environment variables...

    // ADOT Configuration for Application Signals
    OTEL_RESOURCE_ATTRIBUTES: 'service.name={{SERVICE_NAME}}',
    OTEL_METRICS_EXPORTER: 'none',
    OTEL_LOGS_EXPORTER: 'none',
    JAVA_TOOL_OPTIONS: ' -javaagent:/otel-auto-instrumentation-java/javaagent.jar',
    OTEL_TRACES_EXPORTER: 'otlp',
    OTEL_EXPORTER_OTLP_PROTOCOL: 'http/protobuf',
    OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: 'http://localhost:4316/v1/traces',
    OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT: 'http://localhost:4316/v1/metrics',
    OTEL_AWS_APPLICATION_SIGNALS_ENABLED: 'true',
  },
});
2.4 Add Mount Point to Main Container
mainContainer.addMountPoints({
  sourceVolume: 'opentelemetry-auto-instrumentation-java',
  containerPath: '/otel-auto-instrumentation-java',
  readOnly: false,
});
2.5 Configure Container Dependencies
mainContainer.addContainerDependencies({
  container: initContainer,
  condition: ecs.ContainerDependencyCondition.SUCCESS,
});

mainContainer.addContainerDependencies({
  container: cwAgentContainer,
  condition: ecs.ContainerDependencyCondition.START,
});

Completion

Before reciting the summary below (guidance for you, not for the user): report both managed policies — the task role now carries two, and earlier versions of this summary named only CloudWatchAgentServerPolicy. Current CloudWatchAgentServerPolicy already grants the xray:Put* and sampling actions, so AWSXRayDaemonWriteAccess is largely redundant here. If the user asks to trim to least privilege, verify against the live policy documents (aws iam get-policy-version) rather than hand-rolling an inline policy from this guide. Resource scoping is the more valuable axis than pruning actions: CloudWatchAgentServerPolicy grants everything on Resource: "*", so dropping actions still leaves logs:PutLogEvents on every log group in the account. And do not assume step 1.2 covers the agent's log needs — /ecs/ecs-cwagent is the awslogs destination for the agent container's stdout, written under the task execution role. The Application Signals data the agent itself publishes goes to /aws/application-signals/data, which this guide does not pre-create, so logs:CreateLogGroup and logs:CreateLogStream must survive any trim. A denial does not surface in the console — check the agent's own log — and the symptom is that telemetry never starts arriving.

Tell the user:

"I've completed the Application Signals enablement for your application. Here's what I modified:

Files Changed:

  • IAM role: Added CloudWatchAgentServerPolicy and AWSXRayDaemonWriteAccess (two managed policies)
  • ECS container: Installed and configured CloudWatch Agent as sidecar
  • ADOT SDK container: Mounted ADOT SDK dependencies into Application container
  • Application container: Enabled zero-code auto-instrumentation for Application

Next Steps:

  1. Ensure that Application Signals is enabled in AWS account.
  2. Review the changes I made using git diff
  3. Deploy your infrastructure
  4. After deployment, wait 5-10 minutes for telemetry data to start flowing

Verification:

  • Open AWS CloudWatch Console → Application Signals → Services
  • Look for your service (named: {{SERVICE_NAME}})

Troubleshooting Refer to the CloudWatch APM troubleshooting guide.

Let me know if you'd like me to make any adjustments before you deploy!"

Source: SKILL.md on GitHub

1 warning6d3 checks · Risk SAFE
  • Gen Agent Trust Hub6d

    This skill provides comprehensive capabilities for AWS observability and debugging, including agent evaluation and dynamic instrumentation. It includes some security considerations, such as the processing of untrusted telemetry data and the use of external scripts for service onboarding, which are handled with a focus on user confirmation and best practices.

  • Socket6d

    2 alerts: gptAnomaly

  • Snyk6d

    Risk: LOW · No issues

Signed by skilld at bb272a8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last week
metadata
{
  "version": "6"
}

README badge

README badge for aws/agent-toolkit-for-aws/aws-observability