/skill-writer

Sources

This file tracks source material synthesized into skill-writer, plus iterative changes over time.

Current source inventory

Decisions

1. skill-writer is a meta-router: it must choose both a skill class and an execution shape before authoring.
2. Default to the simplest adequate shape. Advanced mechanics require evidence and an explicit reason simpler shapes were rejected.
3. Skill class and execution shape are independent axes. Class drives coverage requirements; shape drives runtime mechanics and artifact layout.
4. SKILL.md remains the orchestration/index layer; references, scripts, assets, hooks, and subagents are leaves selected by that router.
5. Provider-specific Claude Code features are valuable but not default. Use them only when justified and record portability implications.
6. Claude-specific frontmatter guidance should track the current code.claude.com/docs/en/skills fields, including when_to_use, arguments, effort, paths, shell, context, agent, and hooks.
7. Router, parallel/orchestrator, subagent-fork, and hook-backed shapes each require explicit contracts, not just prose.
8. Hooks are deterministic enforcement and need narrow scope plus security notes because command hooks run with full user permissions.
9. Multi-agent guidance should distinguish manager/orchestrator, handoff, and isolated subagent execution instead of collapsing them into one pattern.
10. Reasoning-model guidance is a design option, not a universal default: planner/doer splits are useful when complexity warrants them.
11. Architectural choices are reviewed qualitatively during authoring, not inferred by validator heuristics.
12. Reference files remain split by lookup need rather than topic buckets, even as the set of supported shapes expands.
13. Runtime references stay flat under references/; related leaves use filename prefixes, and every bundled reference is directly discoverable from SKILL.md.
14. The validator should enforce durable structural guarantees and required fields, but should not hardcode name style, path-portability scans, skill classes, source-coverage schemas, SPEC headings, trigger-quality heuristics, or provider-specific optional frontmatter keys.
15. skill-writer should default to dense structures such as tables, checklists, templates, and I/O examples, and should cut explanatory prose unless it prevents a concrete mistake.
16. SKILL.md should stay a thin router; repeated policy belongs in routed references rather than always-loaded step prose.
17. Validation stays lightweight and structural; qualitative precision remains an authoring judgment.
18. Missing referenced bundled files are structural failures because they break runtime loading.
19. Generated skill quality should be evaluated with repo-level repeatable task cases, baseline comparison, and a rubric rather than by adding semantic heuristics to quick_validate.py.
20. Eval cases for skill-writer itself belong outside runtime SKILL.md routing so they do not affect normal skill use.
21. The prescriptive default for skill evals is AXIS with agents: ["codex"] when the eval must exercise Codex; AXIS uses the Codex codex exec --json harness while adding scenarios, artifacts, reports, judge checks, and baselines.
22. Promptfoo is not part of the prescribed skill-eval path; AXIS is the only framework choice for skill evals in this repo.
23. skill-writer needs runtime guidance for authoring evals for other skills, but its own maintainer eval files must stay out of SKILL.md routing.
24. Skill evals should combine deterministic assertions with LLM-as-judge for qualitative dimensions and require concrete evidence for every grade.

Coverage matrix

Open gaps

1. Advanced-shape contracts still rely on author judgment and review rather than deterministic validation.
2. This repository still has few real shipped examples using hooks, context: fork, when_to_use, arguments, paths, or effort.
3. Public repo docs outside skill-writer may need follow-up updates to fully reflect the current Claude-specific skill fields.
4. The initial skill-writer eval cases are synthetic; they should be replaced or expanded with real generated-output regressions over time.
5. AXIS is now the eval harness, but the initial scenarios are still synthetic and should be calibrated against real generated-output regressions.

Changelog

• 2026-03-05: Initialized SOURCES.md with baseline source pack (local canonical, Codex upstream, Claude upstream, spec, and repo conventions).
• 2026-03-19: Clarified path-resolution guidance so bundled skill references stay skill-root-relative while registration steps are resolved from the repository's active layout.
• 2026-03-19: Made portability a default authoring rule and emphasized avoiding host-specific absolute filesystem paths.
• 2026-04-19: Updated path guidance to preserve repository-standard root variables such as ${CLAUDE_SKILL_ROOT} instead of banning them outright.
• 2026-05-09: Replaced ${CLAUDE_SKILL_ROOT} guidance with skill-root-relative paths. Environment variable path prefixes cause permission friction and runtime failures in Claude Code.
• 2026-04-19: Restored .agents/skills as the default authoring target and kept repository-specific layouts as an inspected override rather than the default.
• 2026-04-19: Added explicit prior-art inspection and user-confirmation guidance when the correct skill root is unclear.
• 2026-04-26: Added reference architecture, source discovery, and iteration evidence guidance; updated synthesis, authoring, and iteration paths to prevent overloaded SKILL.md and catch-all reference files.
• 2026-04-26: Added SPEC.md as the canonical skill-writer maintenance specification and added references/spec-template.md for future skills.
• 2026-04-26: Removed fixed integration reference filename validation and added length-based reference warnings.
• 2026-04-26: Reworked reference architecture around concrete lookup needs instead of generic topic buckets.
• 2026-05-01: Reworked skill-writer around explicit execution-shape routing, added shape-specific example profiles, refreshed Claude Code provider mechanics from current official docs, and added source guidance for routing, delegation, and hooks.
• 2026-05-01: Replaced generic pattern bucket references with routed leaf files and made SKILL.md enumerate every bundled reference file with a direct open-when reason.
• 2026-05-01: Removed the validator's hardcoded optional frontmatter allowlist so provider-specific field drift does not create noisy false warnings.
• 2026-05-01: Reduced prose-heavy guidance in skill-writer, rewrote the main runtime refs into denser tables/checklists, and made compact runtime guidance an explicit contract.
• 2026-05-01: Thinned SKILL.md back toward a true router and removed duplicated execution-shape detail from mode-selection.md.
• 2026-05-05: Flattened runtime reference files under references/, kept SKILL.md as the complete material index, removed unused runbooks, added source-adaptation guidance, and made precision passes part of every skill create/update flow while keeping validation lightweight.
• 2026-05-05: Removed validator skill-class inference, integration coverage parsing, SPEC heading checks, SOURCES schema checks, and description-style heuristics.
• 2026-05-05: Removed validator name-style checks and machine-specific path scanning; kept only format, required fields, directory-name match, referenced-file existence, and a high-threshold size warning.
• 2026-06-30: Added skills/skill-writer/EVAL.md, evals/axis.config.json, and AXIS scenarios with baseline comparison guidance, working/holdout cases, artifact capture, judge checks, and semantic grading that stays out of the structural validator and runtime SKILL.md.
• 2026-06-30: Replaced framework-first and custom-runner eval guidance with AXIS as the prescriptive open source skill-eval harness. AXIS satisfies the Codex-harness requirement through its built-in codex adapter while adding reports, artifacts, transcripts, and baselines.
• 2026-06-30: Added references/skill-evals.md so skill-writer can author skill-specific eval playbooks and cases without routing its own maintainer evals.