Index of every page on this site: llms.txt
All skills
langchain-ai avatar

/langchain-agent-experiments

@fb22e06 official
Run it yourself

Use when testing an agent or LLM application offline in LangSmith before it ships, including building and maintaining evaluation datasets, writing the target function and evaluators, aligning LLM judges with human labels, running and comparing experiments, and diagnosing why an experiment scored badly.

Use this Skill: https://skilld.dev/gh/langchain-ai/langchain-skills/langchain-agent-experiments

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesexperiment-trace-replicas.md

≈989 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Reference: keep experiment traces in a tracing project too

By default, every run an experiment produces is recorded only in the experiment's project. Teams sometimes also want those runs in their usual tracing project, for example to search them alongside production or staging traffic, or to run the same dashboards over them. Use trace replicas: the SDK writes a copy of each run to every listed destination.

Leave this off unless someone needs it. Copies double the trace volume for experiment traffic, and the experiment's own results never need them.

Rules that are easy to get wrong

  • Replicas replace the default destination. When replicas is set, runs go only to the listed destinations. A single replica pointing at the tracing project moves the runs out of the experiment, so its results lose their traces. Always list the experiment's own project as well.
  • Keep the experiment copy primary. The primary replica keeps the original run IDs, which the experiment's results and evaluator feedback refer to. Every other replica gets new, deterministic run IDs.
  • Feedback goes to the run ID it names. The experiment's evaluators score the original run IDs, so their feedback lands on the primary copy. To also attach feedback to a secondary copy, compute its run ID with compute_run_id_for_secondary_replica.
  • Reroot copies whose parent lives elsewhere. If the copied runs start below the experiment's root (as in a remote agent service), mark that replica with "updates": {"reroot": True} so it becomes a standalone trace in the tracing project.

Replica options change between SDK versions; check the trace replicas docs and the replica type in langchain-reference against the installed langsmith version before writing this. The examples below pass replicas as dicts.

When the experiment runs the agent in process

Wrap the evaluate() call. A primary replica with no project name falls back to the experiment's project.

from langsmith import Client, tracing_context

with tracing_context(
    replicas=[
        {"primary": True},                        # experiment copy
        {"project_name": "my-tracing-project"},   # extra copy
    ]
):
    Client().evaluate(target, data="my-dataset", evaluators=[...])

When the target calls a deployed agent

The agent service applies the experiment's trace headers (see remote-agent-tracing.md), so its runs nest under the experiment's target run. To also keep a copy in the service's own project, set replicas where the service applies those headers, behind a flag that is off by default:

import os
import langsmith as ls
from langsmith.run_trees import RunTree

def experiment_replicas(trace_headers: dict) -> list[dict] | None:
    if os.getenv("MIRROR_EXPERIMENT_TRACES", "false") != "true":
        return None
    parent = RunTree.from_headers(trace_headers)
    if parent is None:
        return None
    return [
        {"project_name": parent.session_name},  # experiment copy, unchanged
        {"project_name": os.environ["LANGSMITH_PROJECT"], "updates": {"reroot": True}},
    ]

with ls.tracing_context(parent=trace_headers, replicas=experiment_replicas(trace_headers)):
    return await graph.ainvoke(payload)

Here the experiment's project comes from the incoming baggage header, so the service does not need to know it. The service-project copy is rerooted at the agent's root run, because its parent exists only in the experiment project. Requests without trace headers are unaffected.

Verify

  1. Run the experiment on one example.
  2. The experiment looks the same as without replicas: same trace, scores, and token and cost columns.
  3. The tracing project has a new trace for that example, rooted where you expect (the target in process, or the agent's root run when rerooted).
  4. With the flag off, no new trace appears in the tracing project.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning1d3 checks · Risk SAFE
  • Gen Agent Trust Hub1d

    This skill provides a comprehensive framework for testing AI agents with LangSmith, incorporating security-conscious practices for distributed tracing and data handling. It contains minor security considerations regarding indirect prompt injection from the processing of external datasets, which are mitigated by instructions to treat inputs as untrusted and use specific boundary markers for LLM judges.

  • Socket1d

    No alerts

  • Snyk1d

    Risk: MEDIUM · 1 issue

Signed by skilld at fb22e06. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated yesterday

README badge

README badge for langchain-ai/langchain-skills/langchain-agent-experiments