Index of every page on this site: llms.txt
All skills
langchain-ai avatar

/langchain-eval-engineering

@fb22e06 official
Run it yourself

Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.

Use this Skill: https://skilld.dev/gh/langchain-ai/langchain-skills/langchain-eval-engineering

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencessynthetic-data.md

≈739 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Synthetic Data

Create structured truth with deterministic code. Use a model only for fields that need natural language or semantic variation.

Build structured truth first

Define IDs, relationships, permissions, dates, state, constraints, and edge cases in code. Use a fixed seed. Materialize the result so each trial gets the same data. Validate foreign keys, event order, permissions, counts, and required scenario conditions before generating prose.

Create a relationship-complete subset. Each record must support the intended decision, a named distractor, or a required relationship. Do not use answer-coded names, insertion order, a single obvious choice, or unrelated noise.

Generate prose from facts

Give the generator one structured request per output. Include:

  • stable request and target IDs;
  • allowed source facts with fact IDs;
  • document type and workflow position;
  • required and prohibited claims;
  • time bounds and output fields; and
  • a rubric version.

The generator must use only supplied facts and return the fact IDs it used. It must not invent keys, permissions, source facts, or the planned Task answer. Keep the request, output, generator identity, and content hash.

Separate generation from review

When volume warrants it, generate independent records in parallel. Validate shape with code, then use a separate reviewer for meaning. Do not show the reviewer another review.

Review every output for:

Check Question
Facts Does each factual claim match an allowed fact?
Time Are dates and event order valid?
Relations Are people, teams, records, and products connected correctly?
Fit Does the content fit its document type and workflow point?
Distinctness Is it specific and not a near-duplicate?
Safety Does it avoid private data, hidden answers, and agent directions?

Require pass or fail and evidence for each check. Use code for JSON shape, IDs, date and length bounds, prohibited literals, duplicate hashes, strong overlap, and content-hash matching. Use a reviewer only for meaning that code cannot settle.

Merge only accepted content. Use one transaction. Reject missing, changed, repeated, or failed requests. Validate the complete world after the merge.

Test Task quality

Before model runs, inspect ordinary, boundary, and rejected records with their source facts. Confirm that the Task is solvable through normal discovery and that data does not reveal the expected result. Run the reference path and the Verifier fixtures. When model contrast answers a named uncertainty, inspect the selected traces for confusing records, accidental clues, and unsupported shortcuts. Do not require weak-and-strong model runs for every Task.

Put reusable project schemas, generators, validation rules, and safe examples in the project World Skill. Keep a Task's focal records and exact expected answer in its collocated Task.md and Environment assets.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warningtoday3 checks · Risk SAFE
  • Gen Agent Trust Hubtoday

    This skill provides a robust framework for engineering agent evaluations, including tools for discovery, environment building, and verifier design. It includes scripts for database snapshotting and tool schema comparison. The skill demonstrates high security awareness by implementing read-only SQL constraints and providing detailed guidance on isolating untrusted agent data. No significant security risks were detected.

  • Socket5d

    No alerts

  • Snyktoday

    Risk: MEDIUM · 1 issue

Signed by skilld at fb22e06. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated yesterday

README badge

README badge for langchain-ai/langchain-skills/langchain-eval-engineering