All skills
datadog-labs avatar

/agent-observability-experiment-bootstrap

@47d0cf5 official

Bootstrap a reproducible LLM Observability experiment through the Python ddtrace SDK or the Node dd-trace SDK. Use for experiment, dataset, evaluator, benchmark, regression, or LLM-as-a-judge scaffolding. The legacy Python invocation remains supported.

Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/agent-observability-experiment-bootstrap

This session only. Nothing lands on disk.

referencespythonproviderslangchain.md

≈608 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Provider: LangChain

Triggered by introspection (Workflow step 2.5) when the call-site function imports from langchain / langchain_openai / langchain_anthropic / etc. and uses one of:

  • langchain.*.invoke(...)
  • ChatOpenAI(...), ChatAnthropic(...), ChatVertexAI(...), ChatBedrock(...), etc.
  • LLMChain(...)

{{PROVIDER_ASSERTS}} substitution

LangChain is a meta-framework — it wraps a specific provider. Walk one level deeper: the chat-client class names the provider. Read the user's function (and immediate imports) to identify which Chat* class is instantiated, then emit the assert for THAT provider per the table below:

LangChain class Underlying provider Reference file
ChatOpenAI, OpenAI, AzureChatOpenAI (with azure_endpoint=) OpenAI / Azure OpenAI providers/openai.md or providers/openai.md (Azure: also AZURE_OPENAI_ENDPOINT)
ChatAnthropic, AnthropicLLM Anthropic providers/anthropic.md
ChatVertexAI, ChatGoogleGenerativeAI Vertex / Gemini providers/gemini.md
ChatBedrock, BedrockLLM AWS Bedrock providers/bedrock.md
ChatLiteLLM LiteLLM (auto-routes) providers/litellm.md

Emit the assert for the underlying provider, not LangChain itself. Example: if the user's function has from langchain_anthropic import ChatAnthropic, emit:

assert os.getenv("ANTHROPIC_API_KEY"), "ANTHROPIC_API_KEY is required for the wired task_fn (LangChain ChatAnthropic)."

If the call-site uses multiple chat clients (rare), emit asserts for each.

Adapter notes

  • chain.invoke({...}) returns either a string (for simple chains) or an AIMessage (for chat-based chains). Extract .content if it's the latter.
  • LangChain LCEL chains (prompt | llm | parser) return the parser's output type directly — usually a string. Trust the user's function signature.
  • Async: chain.ainvoke(...) — wrap with asyncio.run(...).

Common gotchas

  • LangChain configures provider via env vars by default but ALSO supports per-instance kwargs (ChatOpenAI(api_key="sk-...")). If the user's function passes a key explicitly, the env var is irrelevant — emit a # Note: comment instead of an assert.
  • LangChain prompts/ directory conventions vary widely; the wrapped function should encapsulate prompt loading, so task_fn just calls function(input_data) and trusts the chain.

Source: SKILL.md on GitHub

No alerts1mo3 checks · Risk SAFE
  • Gen Agent Trust Hub1mo

    This skill provides scaffolding for LLM observability experiments using Datadog's official SDKs. It adheres to security best practices by explicitly forbidding the hardcoding of credentials, implementing PII scrubbing for datasets, and utilizing official vendor libraries and documentation.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: LOW · No issues

Signed by skilld at 47d0cf5. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last month

README badge

README badge for datadog-labs/agent-skills/agent-observability-experiment-bootstrap