skilld
All skills
datadog-labs avatar

/dd-apm

@9bcb3ce official
by Datadog Labsdatadog-labs/agent-skills176 stars
29

APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.

Ask your Agent

Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/dd-apm

This session only. Nothing lands on disk.

linux-ssienable-ssiSKILL.md

≈2k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Configure SSI and Unified Service Tags on Linux

Before doing anything else: Fully resolve all variables in ## Context to resolve before acting. Do not begin Step 0 until every variable has a concrete value.

Triggers

Invoke this skill when:

  • The Datadog Agent is already installed with SSI (DD_APM_INSTRUMENTATION_ENABLED=host was used) and you need to configure Unified Service Tags on the application service
  • The user wants to set DD_SERVICE, DD_ENV, DD_VERSION on a running service
  • SSI is installed but /proc/<pid>/maps doesn't show the language tracer (launcher-only injection)

Do NOT invoke this skill if:

  • The Datadog Agent is not yet installed — run agent-install first
  • SSI packages are missing from /opt/datadog-packages/ — re-run agent-install
  • The target is a Kubernetes cluster — use dd-apm-k8s-enable-ssi instead

Background

When the install script runs with DD_APM_INSTRUMENTATION_ENABLED=host, it:

  1. Installs datadog-apm-inject and language library packages under /opt/datadog-packages/
  2. Writes the launcher path into /etc/ld.so.preload
  3. SSI is now armed — every new process on the host gets the launcher injected at startup

What SSI does NOT configure automatically:

  • DD_SERVICE, DD_ENV, DD_VERSION — these must be set on the application process for traces to be tagged correctly
  • Without DD_SERVICE, the tracer auto-detects a service name (often the process name or framework name), which may not match what the user expects

Prerequisites

Verify SSI is armed:

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "cat /etc/ld.so.preload && ls /opt/datadog-packages/ | grep apm"

If /etc/ld.so.preload contains a path to the launcher, and /opt/datadog-packages/datadog-apm-inject exists — SSI is armed.

ERROR: Either missing — run agent-install first.

Check for existing manual instrumentation:

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> "
grep -r 'import ddtrace\|from ddtrace\|require .dd-trace.\|opentelemetry' <SOURCE_DIR> 2>/dev/null | head -5 || echo 'No manual instrumentation found'
"

ERROR: Manual instrumentation found — SSI silently disables itself when it detects an existing tracer. Remove the manual import/package before proceeding.

Check base libc:

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "ldd --version 2>&1 | head -1"

ERROR: musl — SSI requires glibc. No workaround; must use a glibc-based OS.

Context to resolve before acting

Variable How to resolve
SERVICE_NAME Ask the user — how the service should appear in Datadog APM (e.g. payment-api)
ENV Ask the user — environment name (e.g. production, staging, dev)
VERSION Ask the user or read from the app's version file / git tag
SYSTEMD_SERVICE_NAME From systemctl list-units --type=service --state=running on the host — the unit running the app
SSH_KEY Path to SSH private key
SSH_USER SSH username
SSH_HOST Hostname or IP of the target host

Step 0 (Only if existing instrumentation detected): Remove Manual Instrumentation

  • Python: pip uninstall ddtrace, remove import ddtrace / ddtrace-run from CMD
  • Node.js: npm uninstall dd-trace, remove require('dd-trace')
  • Java: remove -javaagent:/path/to/dd-java-agent.jar JVM flag
  • Ruby: gem uninstall ddtrace, remove require 'ddtrace'
  • .NET: remove Datadog.Trace NuGet and profiler env vars

After removing, restart the service. Confirm with the user before restarting. Tell the user: "I need to restart <SYSTEMD_SERVICE_NAME> to remove the old instrumentation. This will cause a brief outage. Ready to proceed?" Wait for confirmation.

Step 1: Set Unified Service Tags on the Application Process

Without UST, traces arrive with an auto-detected service name that may not match user expectations, and won't be tagged with env or version.

For systemd-managed services (most common):

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl cat <SYSTEMD_SERVICE_NAME>"

Add a drop-in override (preserves the original unit file):

What you need to do in a terminal

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST>
sudo systemctl edit <SYSTEMD_SERVICE_NAME>

Add to the editor:

[Service]
Environment="DD_SERVICE=<SERVICE_NAME>"
Environment="DD_ENV=<ENV>"
Environment="DD_VERSION=<VERSION>"

Apply:

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl daemon-reload && sudo systemctl show <SYSTEMD_SERVICE_NAME> | grep -E 'DD_SERVICE|DD_ENV|DD_VERSION'"

If the UST vars appear in the output — configuration applied.

For supervisord:

# In [program:<name>] section of supervisord.conf
environment=DD_SERVICE="<SERVICE_NAME>",DD_ENV="<ENV>",DD_VERSION="<VERSION>"

Reload: sudo supervisorctl reload

For pm2:

// ecosystem.config.js
env: { DD_SERVICE: "<SERVICE_NAME>", DD_ENV: "<ENV>", DD_VERSION: "<VERSION>" }

Reload: pm2 reload <app>

Step 2: Restart the Service

Confirm with the user before restarting. Tell the user: "I need to restart <SYSTEMD_SERVICE_NAME> for SSI to inject into it. This will cause a brief outage. Ready to proceed?" Wait for confirmation.

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl restart <SYSTEMD_SERVICE_NAME> && sleep 3 && sudo systemctl is-active <SYSTEMD_SERVICE_NAME>"

If active is returned — service is running.

ERROR: Returns failed — check logs:

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo journalctl -u <SYSTEMD_SERVICE_NAME> --since '1 minute ago' | tail -30"

Step 3: Confirm Injection and UST in the Running Process

Claude runs

ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "pgrep -a -f '<SERVICE_NAME>' | head -3"

Use the PID:

# Authoritative injection check
ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo cat /proc/<PID>/maps | grep -E 'launcher|apm-library|datadog'"

# UST vars in process environment
ssh -o StrictHostKeyChecking=no -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo cat /proc/<PID>/environ | tr '\0' '\n' | grep -E 'DD_SERVICE|DD_ENV|DD_VERSION'"

If both the launcher and language library appear in maps, and UST vars are in environ — SSI and tagging are fully configured.

ERROR: Launcher in maps but no language library — injection attempted but failed. Run:

pup apm troubleshooting list --hostname <DD_HOSTNAME> --timeframe 15m

Go to troubleshoot-ssi if errors are present.

Done

Exit when ALL of the following are true:

  • Launcher and language library visible in /proc/<PID>/maps
  • DD_SERVICE, DD_ENV, DD_VERSION present in /proc/<PID>/environ
  • Service is running and healthy

Automatically proceed to verify-ssi now — do not ask the user for permission.

Security constraints

  • Never write a raw API key into any file or chat message
  • Always confirm with the user before restarting production services
  • Do not modify application source code — configure only via environment variables in the service unit

Source: SKILL.md on GitHub

Third-party checks

1 warning2d5 checks · Risk SAFE
  • Gen Agent Trust Hub2d

    The skill facilitates the installation, configuration, and troubleshooting of Datadog APM for Kubernetes and Linux environments. It uses official Datadog installation scripts and a CLI tool ('pup') from the 'datadog-labs' GitHub organization. It requires administrative privileges (sudo) and SSH access to perform system-level instrumentation and configuration. All behaviors align with its documented purpose as a Datadog Labs utility.

  • Socket2d

    2 alerts: gptAnomaly

  • Snyk2d

    Risk: LOW · No issues

  • Runlayer6mo

    1/1 file flagged

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Provenance

Signed by skilld at 9bcb3ce. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago

Capability

alwaysApply
true
Other metadata
metadata
{
  "version": "1.1.0",
  "author": "datadog-labs",
  "repository": "https://github.com/datadog-labs/agent-skills",
  "tags": "datadog,apm,tracing,performance,distributed-tracing,dd-apm,install,onboarding,instrumentation,ssi,agent",
  "globs": "**/ddtrace*,**/datadog*.yaml,**/*trace*"
}

Topics

  • Performance
  • datadog
  • apm
  • tracing
  • distributed-tracing
  • instrumentation
  • ddtrace
  • kubernetes
  • linux
  • service-remapping

README badge

README badge for datadog-labs/agent-skills/dd-apm

What it does

Installs the Datadog agent, enables single-step instrumentation (SSI) for automatic tracing, and provides commands to search traces and view service maps. Use this skill for any Datadog APM setup, onboarding, or performance analysis task on Kubernetes, Linux, or to rename services in APM.

Generated from the current SKILL.md.

Frequently asked

Does this skill handle both Kubernetes and Linux host APM setup?
Yes. The skill routes to Kubernetes-specific sub-skills (k8s-ssi) when a cluster orchestrator is mentioned, and Linux-specific sub-skills (linux-ssi) for single hosts or VMs with no orchestrator.
Can I use this skill to rename services in Datadog?
Yes. The skill includes a service-remapping sub-skill that rewrites service names at ingestion time without requiring a deployment rollout.
Does this skill set up Single Step Instrumentation (SSI)?
Yes. SSI auto-instrumentation is covered in both the k8s-ssi and linux-ssi sub-skills; SSI requires no code changes and is enabled via agent installation flags or init container injection.
What if my request doesn't match Kubernetes, Linux, or service remapping?
The skill supports trace searching, service analysis, and metrics queries via pup commands. If your request still doesn't fit, the skill asks you to clarify rather than guessing a workflow.
Do I need to install Datadog Pup separately?
Yes. Datadog Labs Pup must be installed before using this skill; setup instructions are in the main agent-skills repository.

Generated from the current SKILL.md. These answers refresh after source changes.