---
name: eval-author
description: Use when an author needs to add, review, audit, or improve SkillRoll cases for a skill behavior or prompt regression.
title: eval-author
canonical_url: https://skilld.dev/gh/hagaiw/skillroll/eval-author
last_updated: 2026-11-15T17:45:26.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [evals/bounded-repository-pass.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/bounded-repository-pass.eval.md), [evals/defer-artifact-heavy-skill.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/defer-artifact-heavy-skill.eval.md), [evals/diagnose-shared-output-limit.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/diagnose-shared-output-limit.eval.md), [evals/leave-a-final-model-turn.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/leave-a-final-model-turn.eval.md), [evals/offline-context-complete-handoff.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/offline-context-complete-handoff.eval.md), [evals/ordinary-single-skill-case.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/ordinary-single-skill-case.eval.md), [evals/script-boundary.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/script-boundary.eval.md), [evals/unobservable-capability.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/unobservable-capability.eval.md), [evals/world-file.eval.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/evals/world-file.eval.md), [references/context.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/references/context.md), [references/repository-pass.md](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/references/repository-pass.md).
>
> If the user asked to install this Skill, run `npx skilld install hagaiw/skillroll/eval-author`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/eval-author`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/hagaiw/skillroll/eval-author) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/eval-author`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/eval-author --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/eval-author --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Author SkillRoll evals

SkillRoll is a drop-in eval harness for Agent Skills. A case is a small Markdown
file with three parts:

- **Input** is the realistic request the skill must handle.
- **World** is the Dungeon Master's private brief. The evaluated agent cannot
  read it and learns its facts only through simulated action results.
- **Success criteria** describe the behavior that should be visible in the
  response or action transcript.

This makes prompt TDD practical: preserve a failure as a case, confirm it,
change the skill, and rerun the case to protect the behavior from regression.

Before authoring or diagnosing a case, read [authoring context](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/references/context.md)
for the supported case format, limits, evidence boundaries, and failure diagnosis.

For advice or review of supplied material, use that material and the packaged
guidance. Inspect or modify a repository only when the requested task needs
repository-specific evidence or file changes; a request for an explanation is
not a request to execute the authoring workflow.
When returning a proposed case without file changes or live runs, label it an
offline draft and state what verification remains.
When summarizing a completed authoring pass, name each deferred capability and
the real check that must supply the missing evidence outside the harness.

For a broad repository request such as “add evals to this repository,” read and
follow the bounded [repository pass](https://skilld.dev/api/skills-raw/hagaiw/skillroll/eval-author/references/repository-pass.md) before
selecting skills or writing cases. Select at most three skills for the first
batch. Defer skills whose primary behavior needs a capability the case cannot
exercise, even if it is possible to write a generic refusal case. In the
handoff, name the deferrals and state separately whether live inference was
authorized and whether it ran.

## Write one useful case

1. Read the repository instructions and the skill's `SKILL.md` and references.
2. Choose one important decision, boundary, or outcome owned by the skill. For
   boundary cases, separate facts available in Input or skill context, facts
   discoverable through actions, and facts or capabilities that are unavailable.
3. Write a realistic Input. Do not tell the agent the expected answer or
   workflow. When the behavior is recognizing unsupported evidence, give Input
   the observed facts but do not tell it that the conclusion is unknown,
   unconfirmed, or unsupported; put that absence in World.
4. Give the Dungeon Master only the external facts and action results needed
   for this scenario. The agent can discover a limitation when the Dungeon
   Master returns an action failure based on World prose. Exact rules are
   optional and useful when the response must be fixed. World prose can
   describe one failure for all attempts to create a task, including
   dynamically named tasks. Do not require an exact match for each name.
   State a limitation in Input when the user would already know it.
5. Write observable Success criteria. Accept equivalent wording and action
   choices.
6. Run `skillroll validate` before any live eval. Run inference or repository
   commands only when the user and environment authorize them.

## Improve without hiding the failure

When repairing a reported skill failure, add the realistic regression case
before editing `SKILL.md`. Confirm that the current skill fails for the reported
reason, make the smallest responsible change, then rerun the new case and
nearby cases. Keep technical errors distinct from behavioral failures, and
never report an unrun revision as passing.

When diagnosing a run that exhausted its model turns, preserve the original
artifacts. Leave room for a final response and review the new diagnostic's
transcript and verdict before claiming that behavior passed or failed.

For a whole-skill review, map its important behaviors to existing cases or
other tests and suggest the next one to three useful cases.

Treat prompt structure as a source of hypotheses, not behavioral proof. Missing
prerequisites, undefined empty results, conflicting sources, unsupported
capabilities, and absent recovery guidance can motivate a case. Only a completed
run can show how the evaluated model behaved in that case.

## Keep the claim as small as the evidence

The Dungeon Master can simulate files, tools, services, errors, and other
external interaction. It cannot prove that real code compiled, a command ran,
or a visual artifact is correct. Use a deterministic script test, trusted
repository check, or external integration for those claims.

If the requested coverage depends on the finished artifact and the required
external check is unavailable, defer that coverage and name the missing check.
For a PDF, that means running an actual renderer and inspecting its output
outside the simulated case. A case about a decision during rendering is useful
only if that decision independently matters to the requester.

Preserve the intended behavior when choosing that boundary; an evaluation
request is not a request to redesign the output. When a skill mixes model
judgment with fixed rendering, recommend extracting only the deterministic
renderer into a separately tested script. Keep content choices in the prompt
and evaluate their meaning, not the renderer's fixed formatting.

An offline-valid case is a draft, not evidence that the skill behaves well. A
completed run is evidence about the named behavior, not the whole skill. End
with a short report that names what changed, what ran, what remains untested,
and the next useful action.
