/aws-observability

Enable AWS Application Signals for Python Applications on Amazon EKS

This guide shows how to modify existing CDK and Terraform infrastructure code to enable AWS Application Signals for Python applications running on Amazon EKS.

Prerequisites

• Application Signals enabled in your AWS account (see Enable Application Signals in your account)
• Existing EKS cluster deployed using CDK or Terraform code
• Python application containerized and pushed to ECR
• AWS CLI configured with appropriate permissions

Critical Requirements

Error Handling:

• If you cannot determine required values from the IaC, STOP and ask the user
• Preserve all existing configuration; add new resources/annotations in addition

Do NOT:

• Run deployment commands automatically (cdk deploy, terraform apply, etc.)
• Remove existing application startup logic
• Skip the user approval step before deployment

CDK Implementation

1. Install CloudWatch Observability Add-on

Create an IAM role and install the CloudWatch Observability add-on:

import * as eks from 'aws-cdk-lib/aws-eks';
import * as iam from 'aws-cdk-lib/aws-iam';

// Create IAM role for CloudWatch agent
const cloudwatchRole = new iam.Role(this, 'CloudWatchAgentAddOnRole', {
  assumedBy: new iam.OpenIdConnectPrincipal(cluster.openIdConnectProvider),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentServerPolicy')
  ],
});

// Install the CloudWatch Observability add-on
new eks.CfnAddon(this, 'CloudWatchAddon', {
  addonName: 'amazon-cloudwatch-observability',
  clusterName: cluster.clusterName,
  serviceAccountRoleArn: cloudwatchRole.roleArn
});

2. Add Python Instrumentation Annotation

Update your deployment template metadata to include the Python instrumentation annotation:

template: {
  metadata: {
    labels: { app: config.appName },
    annotations: {
      'instrumentation.opentelemetry.io/inject-python': 'true'
    }
  },
  // ... rest of your template configuration
}

Terraform Implementation

1. Add CloudWatch Agent IAM Permissions

Add the CloudWatch policy to the node role:

Neither path in this guide confines CloudWatchAgentServerPolicy to the agent. Attaching to node_role grants it to every pod scheduled on those nodes, since any pod can reach the node's instance credentials unless IMDS access is blocked. The CDK path above uses an IRSA-style role instead of the node role, but as written its trust policy has no :sub condition — new iam.OpenIdConnectPrincipal(cluster.openIdConnectProvider) with no conditions lets any pod holding a projected service-account token call AssumeRoleWithWebIdentity on it. To actually scope it, add a :sub condition naming the addon's service accounts (the addon installs two — cloudwatch-agent and cloudwatch-agent-cluster-scraper) plus :aud sts.amazonaws.com. Tell the customer which of the three states the change leaves them in.

resource "aws_iam_role_policy_attachment" "cloudwatch_agent_policy" {
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
  role       = aws_iam_role.node_role.name
}

Important: Add this policy attachment to your node group's depends_on block:

resource "aws_eks_node_group" "app_nodes" {
  # ... existing configuration ...

  depends_on = [
    aws_iam_role_policy_attachment.node_policy,
    aws_iam_role_policy_attachment.cloudwatch_agent_policy
  ]
}

2. Install CloudWatch Observability Add-on

resource "aws_eks_addon" "cloudwatch_observability" {
  cluster_name = aws_eks_cluster.app_cluster.name
  addon_name   = "amazon-cloudwatch-observability"

  depends_on = [
    aws_eks_node_group.app_nodes
  ]
}

3. Add Python Instrumentation Annotation

Update your Kubernetes deployment template:

template {
  metadata {
    labels = {
      app = var.app_name
    }
    annotations = {
      "instrumentation.opentelemetry.io/inject-python" = "true"
    }
  }
  # ... rest of your template configuration
}

Important Notes

• The Python instrumentation annotation will cause pods to restart automatically
• Ensure your Python application meets the prerequisites for Application Signals
• It may take a few minutes for data to appear in the Application Signals console after deployment

Completion

Before reciting the summary below (guidance for you, not for the user): say which role the policy actually landed on, because the two paths differ in blast radius. The CDK path puts it on an IRSA-style role; the Terraform path attaches it to the node role, which extends it to every pod scheduled on those nodes. Also state that the CDK role's trust policy has no :sub condition unless one was added, so as written neither path confines the policy to the agent. The summary bullet has a your-node-role slot — fill it in with the role this change actually used before reciting.

Tell the user:

"I've completed the Application Signals enablement for your Python application. Here's what I modified:

Files Changed:

• IAM role: Added CloudWatchAgentServerPolicy to your-node-role
• CloudWatch Observability EKS add-on: Added to the EKS Cluster
• Kubernetes Deployment: Instrumentation annotation added with inject-python set to true

Next Steps:

1. Ensure that Application Signals is enabled in AWS account.
2. Review the changes I made using git diff
3. Deploy your infrastructure:
   • For CDK: cdk deploy
   • For Terraform: terraform apply
4. After deployment, wait 5-10 minutes for telemetry data to start flowing

Verification:

• Open AWS CloudWatch Console → Application Signals → Services
• Look for your service and check that traces and metrics are being collected

Warning for Django: If your application is built with Django, you must follow additional steps to prevent startup failures.

Troubleshooting Refer to the CloudWatch APM troubleshooting guide.

Let me know if you'd like me to make any adjustments before you deploy!"