Index of every page on this site: llms.txt
All skills
Run it yourself

Use when measuring how well an AI agent handles a recurring kind of task. Builds a small eval set, runs it, and raises the difficulty once the agent passes.

  • 2 files
  • 2 KB
  • Updated 2 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/erkamyaman/dhh-p-bloom-agent-skills/agent-evals

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈42 tokens for metadata: the name and description. ≈324 when used: this file. ≈135 more on demand in 1 file.

Description uses 2.1% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈170 characters, or 2.1%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Agent evals

Inspired by DHH's Rails World 2026 keynote: https://www.youtube.com/watch?v=vDjW_dRyKXY

From the talk

DHH mentions agent evals for Rails run by Evil Martians for the Rails Foundation. Agents saturated the first version (about 95% completion), so a harder one was made. The steps, task counts and thresholds below are our additions, not from the talk.

The talk mentions an agent benchmark for Rails that agents saturated quickly, so it was made harder. The lesson: evals have a shelf life, and you need to keep raising the bar.

When to use

  • You rely on agents for a recurring task (for example, adding features to your app).
  • You want to compare models, prompts or skills.

Steps

  1. Pick 10 to 20 realistic tasks from your own work.
  2. For each, write a pass/fail check that doesn't depend on a human reading the output (tests, a script, a diff against expected behaviour).
  3. Run the agent on every task from a clean state and record pass or fail.
  4. Look at failures first: was it the agent, the prompt, or a bad check?
  5. When the pass rate is consistently high (say above 90%), add harder tasks.

Verification

Run the same eval twice. If results differ a lot between runs, the checks or tasks are too noisy to trust.

See references/eval-template.md.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 7fcc555. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last month.

Activeupdated 2 months ago

README badge

README badge for erkamyaman/dhh-p-bloom-agent-skills/agent-evals