/skill-writer

Skill Writer Evaluation

Use this file to evaluate changes to skill-writer itself. This is a maintainer eval, not a runtime reference: do not link it from SKILL.md.

Framework Choice

Use AXIS as the skill-eval harness.

Do not add custom run-codex.* scripts. Add a small AXIS scenario, fixture, or artifact pattern instead.

Quick Run

Run the working set:

npx @netlify/axis run --config skills/skill-writer/evals/axis.config.json

Run one case:

npx @netlify/axis run \
  --config skills/skill-writer/evals/axis.config.json \
  --scenario small-inline-workflow

Inspect results:

npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json
npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json --html

Compare to a stored baseline:

npx @netlify/axis run \
  --config skills/skill-writer/evals/axis.config.json \
  --compare-baseline

Set or refresh a baseline only after reviewing the report:

npx @netlify/axis baseline set --config skills/skill-writer/evals/axis.config.json

Evaluation Principles

Use these principles when adding or changing skill-writer evals:

1. Start with 2-3 realistic cases before expanding the suite.
2. Compare candidate output against a baseline instead of trusting one-off scores.
3. Write observable judge checks: files created, forbidden files absent, validation mentioned, generated instructions concise.
4. Use LLM-as-judge for open-ended quality: concision, routed structure, source coverage, progressive disclosure, and overfitting risk.
5. Require concrete evidence in judge checks and manual review: paths, artifacts, diffs, transcript snippets, or validation output.
6. Track timing, token use, and interaction waste from AXIS reports.
7. Review transcripts when a run fails or is slow; wasted steps usually point to bloated or ambiguous skill instructions.
8. Use blind old-vs-new comparison for qualitative regressions when scores are close.

Case Set

AXIS scenarios live in evals/scenarios/.

Add provider-specific cases only when changed rules affect provider-specific mechanics.

What To Test

AXIS Setup

evals/axis.config.json configures:

1. agents: [{"agent": "codex"}] so Codex is the agent under test.
2. skills: ["./.."] so AXIS installs this local skill-writer skill for the run.
3. scenario limits so a stuck generation fails quickly enough to debug.

Each scenario configures artifacts for generated skill files, registration files, and validator output when present.

Each scenario copies the repository into AXIS's isolated workspace before Codex starts. That gives skill-writer the same repo conventions it sees in normal use without polluting this checkout.

Rubric

Score each dimension pass, warn, or fail during manual review. AXIS supplies the transcript, artifacts, and judge scores; this rubric decides whether to adopt the change.

Critical dimensions: trigger precision, artifact minimality, runtime concision, progressive disclosure, and validation.

Adoption Gate

Adopt a skill-writer change only when:

1. AXIS runs complete for the working set.
2. no critical dimension regresses on holdout cases.
3. at least one target dimension improves on the case that motivated the change.
4. structural validation passes for generated skills or the transcript explains a legitimate blocker.
5. runtime SKILL.md remains a router and does not link to eval files.
6. source and maintenance notes are updated only when the skill contract changes.

Result Template

## Eval Summary
- Baseline:
- Candidate:
- AXIS report:
- Cases run:
- Critical regressions:
- Decision: adopt | revise | reject

## Case Results
| Case | Score | Structure | Quality | Regression | Notes |
|------|-------|-----------|---------|------------|-------|

## Required Fixes
- ...

## Evidence
- AXIS report:
- generated artifacts:
- validator output:
- transcript notes: