Index of every page on this site: llms.txt
All skills
langchain-ai avatar

/langchain-agent-experiments

@fb22e06 official
Run it yourself

Use when testing an agent or LLM application offline in LangSmith before it ships, including building and maintaining evaluation datasets, writing the target function and evaluators, aligning LLM judges with human labels, running and comparing experiments, and diagnosing why an experiment scored badly.

Use this Skill: https://skilld.dev/gh/langchain-ai/langchain-skills/langchain-agent-experiments

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesremote-agent-tracing.md

≈793 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Reference: trace a deployed agent into an experiment

Use this when the experiment's target function calls the agent over HTTP instead of running it in process.

The problem

The experiment and the agent service record two unrelated traces. The experiment's trace has only the target's root run, with no model runs under it, so:

  • token and cost columns stay empty (they add up the model runs inside each example's trace), while feedback and latency still show;
  • evaluators and anyone reviewing the experiment cannot see the agent's steps.

Join the two with distributed tracing. Both sides change: the caller sends the trace context, and the service applies it.

Caller: the target function

Inside the target, get the current run tree and merge its trace headers into the request, following the distributed tracing docs. Merge them with existing headers such as auth rather than replacing them. Read them in the target itself and pass them along if the request is made elsewhere: a standard thread pool does not carry tracing context, and on Python older than 3.11 async code may not either.

Service: apply the incoming context

When a request carries langsmith-trace (and baggage), run the agent inside tracing_context(parent=<those headers>). Requests without them behave as before. The baggage header carries the experiment's project, so the service does not need to know it.

import langsmith as ls

trace_headers = {
    k: request.headers[k] for k in ("langsmith-trace", "baggage") if k in request.headers
}
if "langsmith-trace" in trace_headers and is_trusted_internal_caller(request):
    with ls.tracing_context(parent=trace_headers):
        return await graph.ainvoke(payload)
return await graph.ainvoke(payload)

An agent deployed on LangSmith Agent Server has its own setup; search the docs for Agent Server distributed tracing. For a plain ASGI service where all inbound traffic is trusted, the SDK's tracing middleware does the same for every request.

Trust: these headers decide where runs are recorded and what metadata they carry. Apply them only for trusted internal callers, such as the experiment runner, using the service's existing service-to-service auth. Strip both headers from public or end-user traffic at the gateway.

Where the runs land

Once the service applies the headers, experiment requests are recorded only in the experiment project; normal traffic still goes to the service's project. If the service's project must also keep experiment traffic, follow experiment-trace-replicas.md.

Verify

  1. Run the experiment on one example.
  2. Open that example's trace: it should nest the target, the agent, its nodes, and the model calls.
  3. The experiment's token and cost columns should fill in.
  4. No new trace for that request should appear in the service's project.

If the service still records a separate root trace, the headers are not arriving or the agent is not invoked inside the tracing_context block (for example, it was handed to a background worker). Tokens without cost usually means the model name has no pricing entry in the workspace.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning1d3 checks · Risk SAFE
  • Gen Agent Trust Hub1d

    This skill provides a comprehensive framework for testing AI agents with LangSmith, incorporating security-conscious practices for distributed tracing and data handling. It contains minor security considerations regarding indirect prompt injection from the processing of external datasets, which are mitigated by instructions to treat inputs as untrusted and use specific boundary markers for LLM judges.

  • Socket1d

    No alerts

  • Snyk1d

    Risk: MEDIUM · 1 issue

Signed by skilld at fb22e06. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated yesterday

README badge

README badge for langchain-ai/langchain-skills/langchain-agent-experiments