Datadog Labs

Install the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.

Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says "run an auto experiment", "hill-climb this code", "iteratively improve X and measure the delta", "optimize this prompt/file against my traces", "auto-optimize against LLM-Obs", or wants the local equivalent of the auto_experiments worker. Works from an ml_app, a dataset_id, an annotation_queue_id (a queue of human-labelled interactions), a list of trace_ids, or (by exception) a local dataset file. The corpus and its val/test splits live in Datadog LLM-Obs Datasets, created once per run with a timestamp in their names.

Fit a Datadog LLM-Obs evaluator to human labels. Takes an annotation queue, works out where in the trace the labelled property actually lives, drafts an LLM-judge that predicts the human label, scores that judge against the already-labelled rows with a metric agreed with the user, then hill-climbs it — inspect the errors, make one focused change, re-score, keep it only if it beats the best — for a bounded number of iterations, and finally publishes the winner to Datadog as a DISABLED evaluator (not a Datadog draft — a real evaluator with `enabled: false`). Use when the user says "build an eval from my annotations", "build an evaluator from the annotation queue", "turn my annotations into an evaluator", "learn an evaluator from my labels", "fit a judge to the annotation queue", "auto-label", "auto labelling", "automate this annotation queue", "scale up my human labels", or wants the rest of a queue graded the way the humans graded the first rows. Needs an annotation queue with at least two classes present in the human labels (e.g. one true and one false for a boolean).

Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.

End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a "continue" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says "run the eval pipeline", "go from traces to evals", "bootstrap evals end to end", "classify then RCA then bootstrap", "build an eval set from scratch", "onboard me to datasets and experiments", "walk me through experiments", "I have an ml_app, now what", "Agent Observability onboarding", "guided experiment setup", "from traces to experiments", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after <phase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at <phase>`.

Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says "analyze experiment", "compare experiments", "analyze against baseline", or provides one or two experiment IDs for analysis.

Bootstrap a reproducible LLM Observability experiment through the Python ddtrace SDK or the Node dd-trace SDK. Use for experiment, dataset, evaluator, benchmark, regression, or LLM-as-a-judge scaffolding. The legacy Python invocation remains supported.

Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace <trace-id> [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace <id> with <change>"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace / LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser "Replay" button (that's agent-observability-replay-experiment), building an experiment from a dataset/CSV, writing evaluators, root-causing failed traces, or RUM/HTTP session replay.

Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions/traces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.

Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says "what's wrong with my app", "why is my eval failing", "analyze errors", "root cause analysis", "diagnose failures", or wants to understand production failure patterns.

Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.

Audit what the Bits AI assistant (MCP server) has done in your Datadog org — tool calls by user, resources accessed, and anomaly flags for AI governance.

Generate auditor-ready compliance evidence from Datadog Audit Trail for SOC 2 and PCI DSS. Maps framework controls to specific query patterns and produces formatted output.

Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when/what spiked) with Audit Trail config changes (who changed what in the preceding window).

Guides developers building Datadog Apps with TypeScript, React, the @datadog/apps scaffolder, and @datadog/vite-plugin. Use when a user wants to scaffold, run, debug, upgrade, build, upload, publish, upload without publishing (draft upload), add an upload-no-publish script, set up CI/CD, use OAuth or API/application key auth, trigger/poll Workflow Automation, choose DDSQL or Action Catalog for backend data access, or query app datastores with DDSQL, including backend function troubleshooting.

Ensure the user has an authenticated Datadog account with a valid DD_API_KEY on the right region before any Datadog setup or instrumentation. Detects existing DD_API_KEY / DD_APP_KEY / DD_SITE, validates them against the Datadog API, and fixes the common wrong-region 403. If no usable key exists, signs the user in (OAuth) or creates a new account, then obtains and validates a key. Use this whenever a user needs a Datadog account or API key, hits a 403 / wrong-region error, or is about to run any Datadog *-setup or instrumentation skill.

APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.

Audit Trail investigations - who changed what, key compromise, cost spike root cause, compliance evidence (SOC 2/PCI), and AI activity auditing.

Set up the Datadog AWS integration with Terraform - creates the cross-account IAM role Datadog assumes (external ID, no stored credentials), attaches the permission policies Datadog publishes, and registers the account through datadog_integration_aws_account so AWS metrics, the resource catalog, and CSPM findings start flowing. Use when the user has AWS resources they want to monitor, wants to connect an AWS account to Datadog, asks to set up or repair the AWS integration, or needs the Datadog IAM role and external ID provisioned. Does not set up log forwarding.

Set up the Datadog Azure integration with Terraform - creates an Entra ID app registration and service principal, assigns Monitoring Reader across the chosen subscriptions and management groups, grants the Microsoft Graph permissions Datadog needs for resource discovery, and registers the tenant so Azure metrics and resource collection start flowing. Use when the user wants to monitor Azure VMs, App Service, SQL Database, or AKS, wants to connect an Azure subscription or management group or tenant to Datadog, or asks to set up or repair the Azure integration. Does not set up log forwarding.

Datadog Browser SDK — RUM, Logs, Session Replay, profiling, product analytics, and error tracking setup, configuration, and migration. Use when upgrading Browser SDK versions, setting up RUM or Logs, or troubleshooting browser-side Datadog instrumentation.

Datadog docs lookup using docs.datadoghq.com/llms.txt and linked Markdown pages.

Set up the Datadog Google Cloud integration with Terraform - creates a service account in the host project, lets Datadog's delegate principal impersonate it via roles/iam.serviceAccountTokenCreator (no service-account keys), enables the required APIs, grants the monitoring roles across the chosen projects and folders, and registers the account through datadog_integration_gcp_sts. Use when the user wants to monitor GCP resources such as Compute Engine, Cloud SQL, GKE, Cloud Run, or Pub/Sub, wants to connect a GCP project or folder or organization to Datadog, or asks to set up or repair the GCP integration. Does not set up log forwarding.

Instrument browser-based web applications with Datadog Browser RUM. Detect the application framework, router, package manager, bundler, entrypoint, credentials, and existing RUM setup; add or safely complete classic Browser RUM instrumentation for React, Next.js App or Pages Router, Angular, Vue, Nuxt, Svelte, vanilla JavaScript, SPAs, and iframe-hosted apps; avoid duplicate initialization; and verify the application still builds. Use when asked to add, set up, instrument, repair, or verify Datadog RUM, Browser Monitoring, Session Replay, or framework-specific Browser RUM plugins.

Log management - search, archives, metrics, and cost control.

Monitor management - list, search, file-based create, and alerting best practices.

Set up the Datadog Oracle Cloud Infrastructure (OCI) integration with Terraform - verifies ~/.oci/config, then applies Datadog's official oracle-cloud-integration module to create the Datadog service user, group, IAM policies, and API key in the tenancy and register it with Datadog, optionally including log collection. Use when the user has Oracle Cloud resources, wants to monitor an OCI tenancy, wants to connect OCI to Datadog, or asks to set up or repair the OCI integration.

Entry point for Datadog onboarding. Takes a developer's plain-language goal, ensures a valid Datadog account with dd-account-setup, asks dd-product-recommender which products fit, detects the project's platform and cloud, then composes an ordered plan across the existing skills (agent install, product enable, verify, and optional cloud integration) and dispatches to each by source URL — honestly flagging products with no skill yet. Use when the user says "set up Datadog", "onboard my app / this repo to Datadog", "instrument my project", or states a monitoring goal without naming a specific product or skill.

Recommends the right Datadog products for a codebase and/or a stated goal — grounded in a tech-stack→product map and a use-case→product map built from Datadog product capabilities and common technology patterns. Recommendation only; no setup instructions. Use when a user asks which Datadog products fit their app, what to monitor, or which products serve a goal like security, cost, or LLM observability.

Datadog CLI (Rust). OAuth2 auth with token refresh.

Enable Single Step Instrumentation (SSI) on Kubernetes — automatically instruments applications for APM without code changes. Only use if the Datadog Agent is already running on the cluster — if not, use agent-install first.

Investigate a potentially compromised Datadog API key — timeline of actions, geo/IP breakdown, endpoints called, anomaly flags, and remediation steps.

Generate a live Single Step Instrumentation (SSI) onboarding confirmation report — verifies APM instrumentation is working end-to-end with deep links into the Datadog UI. Only use after agent-install and enable-ssi have both completed successfully.

Generate a BYOD ownership preferences reference table for a customer. Walks through preference types, generates CSV, and provides upload instructions (UI, API, cloud storage, or Terraform). Use when asked about BYOD setup, preferences reference table, k9_ownership_preferences, or ownership customization.

Answer "who did what" security questions from Audit Trail — deletions, config changes, login activity, permission changes, actions from a specific user or IP.

Create and manage APM service remapping rules — rewrite service names at ingestion time to collapse noisy inferred entities, clean up auto-generated names, handle org renames, or normalize naming conventions. Use for any request involving service renaming, service mapping, inferred service cleanup, peer.service normalization, or collapsing fragmented service names.

Configure build-time JavaScript sourcemap upload to Datadog using the Datadog build plugin (@datadog/esbuild-plugin, @datadog/rollup-plugin, @datadog/rspack-plugin, @datadog/vite-plugin, @datadog/webpack-plugin), so RUM and Error Tracking show un-minified stack traces with git metadata attached. Use when errors in Datadog show minified stack traces, when asked to "upload sourcemaps", "set up sourcemaps", "unminify errors", or "configure the Datadog build plugin", or when a project has a bundler config and @datadog/browser-rum but no sourcemap upload.

Load when investigating a specific flaky test. Gets history, failure pattern, and category, then recommends fix, quarantine, or escalate.

Diagnose and fix Single Step Instrumentation (SSI) issues on Kubernetes — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are already configured but traces are missing or instrumentation is not working.

Load when investigating a failing PR CI pipeline or checking PR health. Attributes each CI failure as flaky, infra, or regression, proposes a targeted action, and reports code coverage and quality/security status.

Upgrade Datadog Browser SDK from v4 to v5. Use when encountering removed options like proxyUrl, sampleRate, replaySampleRate, premiumSampleRate, allowedTracingOrigins, or deprecated APIs like addRumGlobalContext, removeUser, or when a project references datadoghq-browser-agent.com CDN with /v4/ paths.

Upgrade Datadog Browser SDK from v5 to v6. Use when encountering removed options like useCrossSiteSessionCookie, sendLogsAfterSessionExpiration, or when dropping IE11 support, or when a project references datadoghq-browser-agent.com CDN with /v5/ paths.

Upgrade Datadog Browser SDK from v6 to v7. Use when encountering removed options like betaEncodeCookieOptions, allowFallbackToLocalStorage, trackBfcacheViews, usePciIntake, changed APIs like forwardErrorsToLogs, startDurationVital, stopDurationVital, or when a project references datadoghq-browser-agent.com CDN with /v6/ paths.

Verify Single Step Instrumentation (SSI) is working end-to-end on Kubernetes — SSI automatically instruments applications for APM without code changes. Only use after enable-ssi has run.