Index of every page on this site: llms.txt
All skills
datadog-labs avatar

/dd-instrument-llmo

@618fae3 official
Run it yourself

Instrument the current project with Datadog LLM Observability for Python or Node.js/Next.js backends that call LLMs or run AI agents. Detects the runtime and LLM framework, provisions credentials, adds SDK init (ddtrace/dd-trace) with the correct kwargs, persists the dependency into the deploy manifest, and audits session-ID plumbing for gaps — fixing them when found. Use when the user says "instrument this project with LLM Observability", "add LLM Observability", "monitor my AI app in Datadog", "add LLM spans", "add agent session tracking", or "verify/repair my LLM Observability setup".

Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/dd-instrument-llmo

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesllmobs-python.md

≈4.3k tokens on demand. Your agent reads this file only when SKILL.md points to it.

LLM Observability: Python

Provision DD_API_KEY first via common-credentials.md if you haven't already.

Install — and persist ddtrace into the dependency manifest the deploy installs from (the one identified in SKILL.md Phase 1e), not just the local environment. A local-only install makes the app run for you, but a clean deploy install reads the manifest and will crash with ModuleNotFoundError: No module named 'ddtrace' if the package isn't declared there.

Persisting installs (preferred). These commands record ddtrace in the manifest automatically — nothing else needed:

poetry add ddtrace      # writes pyproject.toml [tool.poetry.dependencies]
uv add ddtrace          # writes pyproject.toml [project.dependencies] + uv.lock
pdm add ddtrace         # writes pyproject.toml
pipenv install ddtrace  # writes Pipfile

Non-persisting installs (you must also hand-edit the manifest). These commands touch the current environment only and do NOT write any manifest — after running one, add ddtrace to the manifest by hand:

pip install ddtrace              # env only — does not edit requirements.txt/pyproject.toml
uv pip install ddtrace           # env only — prefer `uv add`
conda install -c conda-forge ddtrace

Where to hand-edit: a ddtrace line in requirements.txt, or "ddtrace" in the [project.dependencies] (or [tool.poetry.dependencies]) array of pyproject.toml — matching whichever manifest Phase 1e identified.

In-code init

Copy the import and LLMObs.enable(...) call verbatim. The module is ddtrace.llmobs — there is no ddtrace.llm_observability module, and "correcting" the terse name to the spelled-out one is the most common way this instrumentation crashes on deploy (ModuleNotFoundError: No module named 'ddtrace.llm_observability'). Use exactly the kwargs shown below (ml_app, api_key, site, agentless_enabled); don't substitute env/service or add a from ddtrace import tracer, config line from memory.

Must run before any other application imports:

# any top-level imports, mainly os or dotenv
import os
import dotenv

# import and enable LLM Observability _before_ any other module imports
from ddtrace.llmobs import LLMObs

LLMObs.enable(
    ml_app=os.environ["DD_LLMOBS_ML_APP"],
    api_key=os.environ["DD_API_KEY"],
    site=os.environ.get("DD_SITE", "datadoghq.com"),
    agentless_enabled=True,
)

# all other application-specific imports, and remaining application logic
from openai import OpenAI
# ...

Alternative: no code change

Wrap the run command with ddtrace-run and pass config via env vars:

DD_SITE=datadoghq.com DD_LLMOBS_ENABLED=1 DD_LLMOBS_ML_APP=<app-name> DD_API_KEY=<key> DD_LLMOBS_AGENTLESS_ENABLED=1 ddtrace-run app.py

For FastAPI/uvicorn: ddtrace-run uvicorn main:app --reload.

Beyond SDK init: a well-formed trace

Run this section even if LLMObs.enable() already exists in the project. SDK init alone only turns on auto-instrumented llm spans — it does not by itself produce a session ID, a root span, or a RUM/APM link. Don't let Phase 1d's "SDK already present" finding stand in for this checklist.

Enabling the SDK produces auto-instrumented llm spans for supported providers, but a useful trace also needs a root span, session ID, and annotations. Work toward these four goals; the first two are always achievable, the last two depend on the environment:

# Goal Always achievable? What it requires
1 Well-formed LLMObs trace Yes SDK initialized; a root span with the right kind; nested child spans; input_data/output_data annotated
2 Agent Session ID Yes A stable session_id on the root span, scoped to one conversation/operation — it propagates to children automatically
3 RUM session linking Opt-in, needs RUM SDK Correlating a RUM session with LLMObs works by forwarding the RUM session ID (datadogRum.getInternalContext().session_id) to the backend and using it as the session_id. That is the only documented link, so it makes the LLMObs session span the whole browser session — several conversations in one browse collapse into one. Prefer the conversation-scoped session_id from goal 2 by default; opt in only when that whole-browser grouping is acceptable
4 APM trace linking Partial — always apm_trace_id is set on every span automatically. Full navigable linking in the UI needs a dd-agent; without one the tag is present but the APM trace won't exist

Detect what's achievable before instrumenting: check for @datadog/browser-rum on the frontend (goal 3, opt-in), and for DD_AGENT_HOST/DD_TRACE_AGENT_URL/ddtrace-run/an agent sidecar (goal 4). Never promise RUM or navigable APM linking that the detected environment doesn't support — state the gap and the upgrade path instead.

Goal 4 config knob: the init snippet above hardcodes agentless_enabled=True. Flip it to False whenever a dd-agent is detected — leave it True otherwise (the apm_trace_id tag is still set either way, but only becomes a navigable APM trace with an agent present).

Adding spans

All imports come from ddtrace.llmobs (public package) — never import from underscore-prefixed submodules.

Decorators (preferred for child functions):

from ddtrace.llmobs.decorators import agent, workflow, task, tool, retrieval, embedding

@agent(name="my_agent", session_id=session_id)   # orchestrates LLM calls — set session_id here only
@workflow(name="my_workflow")                    # multi-step, no direct LLM calls — inherits session_id from root
@task(name="my_task")                            # pure computation
@tool(name="my_tool")                            # wraps a capability
@retrieval(name="my_retrieval")                  # RAG / vector search
@embedding(name="my_embedding")                  # embedding generation

Context manager (preferred for the root span when session_id is a runtime value):

with LLMObs.agent(name="my_agent", session_id=session_id) as span:
    LLMObs.annotate(span=span, input_data=user_message, output_data=response)

LLMObs.agent(...) returns a Span, which only supports the sync context manager protocol — always use with, even inside an async def. Using async with raises 'Span' object does not support the asynchronous context manager protocol.

Annotation:

LLMObs.annotate(
    span=span,                  # None = current active span
    input_data="user message",  # str or messages list
    output_data="agent response",
    metadata={"key": "value"},
)

Session ID rule: set session_id only on whichever decorator/context-manager is the root span of the trace — typically @agent/LLMObs.agent(...), but a @workflow can be the root if the entry point has no direct LLM orchestration (see the LangChain/LangGraph note below). Descendants inherit it via ddtrace context propagation — never pass session_id to a non-root decorator.

Session name (optional display title)

session_id has no companion "display name" field in the SDK — the UI otherwise shows a truncated session ID. If the application already has (or can cheaply derive) a human-readable label for the operation — a job type, a conversation topic, a ticket title — set it as a session_name tag on the same root span that carries session_id, via LLMObs.annotate(tags=...):

with LLMObs.agent(name="my_agent", session_id=session_id) as span:
    LLMObs.annotate(span=span, tags={"session_name": "Playlist for Milo (cat)"})

Only set it once, on the root span — same rule as session_id. Skip this if there's no natural label available; don't fabricate one from generic defaults (e.g. f"session-{session_id}") since that's no more useful than the raw ID.

Session ID intake by environment

Scope the session_id to one conversation/operation — that is the unit the LLM Obs UI groups on. Don't default it to the browser/RUM session (see the opt-in below for why).

Default — web app (conversation-scoped) — the frontend owns a conversation ID: mint one when a conversation starts (a chat thread, a task), reuse it across that conversation's turns, and start a fresh one when the user begins a new conversation. Send it with each request under whatever key fits this project's existing request-body naming; read it on the backend with the same request-parsing accessor the project already uses for its other fields — do not assume one framework's access pattern (FastAPI's Pydantic body model gives attribute access like body.session_id, with headers read off a separate Request param; Flask needs request.get_json()["session_id"] or a schema's .load(); see Framework-specific notes below):

import uuid
from fastapi import Request

# FastAPI: body fields (session_id, message) come off the Pydantic model; the
# x-session-id header comes off the Request object — a body model has no `.headers`,
# so take both. Substitute the accessors your framework/route actually uses.
@app.post("/chat")
async def chat(body: ChatRequest, request: Request):
    session_id = body.session_id or request.headers.get("x-session-id") or str(uuid.uuid4())
    with LLMObs.agent(name="my_agent", session_id=session_id) as span:
        LLMObs.annotate(span=span, input_data=body.message)
        result = do_work(body.message)
        LLMObs.annotate(span=span, output_data=result)

Frontend side (show as instructions, never edit frontend files yourself in this reference):

body: JSON.stringify({
  message: userMessage,
  // one ID per conversation, regenerated when the user starts a new one — NOT the RUM/browser session ID
  session_id: conversationId,
})

Opt-in — reuse the RUM session ID (enables the RUM↔LLMObs pivot) — the only documented way to pivot between a RUM session and its LLMObs traces is to forward the RUM session ID and use it as the session_id. The trade-off: the LLMObs session then spans the whole browser session, so multiple conversations in one browse collapse into a single LLMObs session. Choose this only when that grouping is acceptable (setting up the RUM SDK itself is out of scope for this skill):

body: JSON.stringify({
  message: userMessage,
  session_id: datadogRum.getInternalContext()?.session_id ?? crypto.randomUUID(),
})

No web frontend (CLI / background job) — reuse an existing per-operation identifier if the code already generates one before the pipeline runs (job_id, request_id, task_id, etc.) instead of minting a second ID; otherwise one UUID per invocation:

session_id = job_id or str(uuid.uuid4())

Multi-agent pipelines and sub-agent orchestration

When one function invokes multiple distinct agents (or the same agent multiple times) as sequential stages of a single logical operation — a background job that runs a profiler agent, then a theme agent, then a scout agent, then a curator agent, for example — wrap the entire orchestrating function once. One root span per logical operation, not one per agent.run()/agent.iter() call:

async def run_pipeline(job_id: str) -> None:
    with LLMObs.workflow(name="playlist_pipeline", session_id=job_id) as span:
        vibe = (await profiler_agent.run(...)).output
        theme = (await theme_agent.run(...)).output
        candidates = await scout_agent.run(...)
        selections = (await curator_agent.run(...)).output
        LLMObs.annotate(span, input_data=job_id, output_data=selections)

Use workflow here, not agent — run_pipeline doesn't call an LLM directly, it orchestrates other agents that do (see the decision table below). Each agent.run() call auto-instruments as a nested agent span under this one root; all four land on a single trace and inherit session_id through ddtrace's context propagation. This applies to any auto-patched agent framework (Pydantic AI, LangChain/LangGraph, OpenAI Agents SDK) — the rule is about where you put the one enclosing span, not which framework is calling the LLM.

Anti-pattern — produces one trace/session per stage instead of one per operation: calling agent.run() for each stage directly inside the pipeline function with no enclosing span. Without an active parent span, there's nothing for the auto-instrumentation to attach each call to, so every agent.run() gets its own root span — a separate trace, and (with no explicit session_id) a separate session in the UI. This is easy to miss because the SDK is "enabled" and traces do show up — they're just fragmented one-per-agent instead of grouped one-per-operation. Symptom in the trace list: what should be one conversation instead shows up as N separate single-agent conversations (e.g. curator_agent, scout_agent, theme_agent, profiler_agent each listed on their own, instead of one session containing all four).

Detecting this while verifying: if a single logical operation (one job, one request, one pipeline run) produces more than one root span/trace in the LLM Obs UI, the fix is to add one enclosing workflow/agent span around the whole orchestrating function — not to sprinkle the same session_id across each stage's own root span. Matching session_ids across separate traces groups them into one session, but a single trace with nested children is the stronger, simpler, and intended fix; prefer it whenever the stages share one caller.

A deliberate split is fine — the anti-pattern is the accidental one. The failure above is unintentional fragmentation: no enclosing span and no shared session_id, so the trace splits and the pieces never regroup. When the agents are genuinely independent or long-lived — not tight sequential stages of one operation — a separate trace per agent is a legitimate call; put the same session_id on each root span and they still read as one session in the UI. One combined trace stays the default — it's simpler, and it's usually what a single operation is — so reach for per-agent traces when you actually want them, not because the enclosing span got dropped.

Span kind decision table

Code pattern Span kind
Top-level function orchestrating LLM calls and tools agent
Multi-step pipeline not directly calling LLMs workflow
Function calling an LLM directly (only if not auto-instrumented) llm (manual)
Function wrapping a capability (web search, calculator, DB) tool
Pure data transformation / business logic task
RAG / vector search retrieval
Text embedding generation embedding

Auto-instrumented providers (no manual llm span needed): OpenAI, Anthropic, LangChain/LangGraph, AWS Bedrock, Google Vertex/Gemini, LiteLLM, CohereAI. Decorate the calling function with tool/agent instead of the provider call itself.

Never add spans to: web routing/middleware, data model/config classes, DB/cache clients, logging/metrics utilities, test files, pure utility functions.

Framework-specific notes

  • OpenAI Agents SDK: coexists with its own trace(); Runner.run() is auto-instrumented — wrap the calling function with @tool/@agent; align session_id via group_id in trace() and session_id in the LLMObs context manager.
  • Pydantic AI: auto-patched on LLMObs.enable(); wrap the function calling agent.run()/agent.iter() with a context manager passing session_id. If that function calls more than one Agent (a multi-stage pipeline), wrap it once, enclosing every stage — see "Multi-agent pipelines and sub-agent orchestration" above; don't wrap each stage's .run() call separately.
  • LangChain/LangGraph: fully auto-instrumented; wrap the top-level invoking function with @workflow(session_id=...) or a context manager — session_id is valid here only because this @workflow is the root span; don't add it to any nested @workflow.
  • FastAPI: LLMObs.enable() where app = FastAPI() is created; session ID from request body or x-session-id header with UUID fallback.
  • Flask: LLMObs.enable() before app.run() or in create_app(); session ID from request.get_json()["session_id"] (or a schema's .load()), not attribute access — Flask's request has no Pydantic-style fields.
  • CLI / background job: prefer an existing per-operation ID if the code already tracks one (session_id = job_id or str(uuid.uuid4())); otherwise str(uuid.uuid4()) per run. RUM not applicable.

Reporting goal status

Alongside the standard report shape in common-verify-report.md, summarize goal status in the human-readable message, e.g.:

✓ Well-formed LLMObs trace
✓ Agent Session ID (conversation-scoped; UUID fallback)
~ RUM session linking: not applicable (no RUM SDK detected)
~ APM trace linking: apm_trace_id set, but not navigable (no dd-agent detected)

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts13d3 checks · Risk SAFE
  • Gen Agent Trust Hub13d

    This skill instruments projects with Datadog LLM Observability by installing official SDKs and configuring environment variables. It follows standard development practices for observability setup and credential management.

  • Socket14d

    No alerts

  • Snyk13d

    Risk: LOW · No issues

Signed by skilld at 618fae3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 2 months ago
Other metadata
metadata
{
  "version": "0.1.0",
  "author": "datadog-labs",
  "repository": "https://github.com/datadog-labs/agent-skills",
  "tags": "datadog,llm-observability,llmobs,instrumentation,python,nodejs,nextjs,ddtrace,dd-trace,agents",
  "alwaysApply": "false"
}

README badge

README badge for datadog-labs/agent-skills/dd-instrument-llmo