verifying-doc-parity evals
Behavior test cases for the verifying-doc-parity skill, in the skill-creator schema.
behavior-eval.json holds cases exercising what happens when the skill fires. Covers both primary success paths (session and pull-request review), every outcome the skill claims to produce (Update, Attest, Dismiss), and each recently-added instruction (below-component source-embedded surfaces, placement follows what the doc describes, root-level placement for a repo-wide capability, the out-of-ancestor-chain restatement search, and the condensed Step 4 report shapes).
behavior-baseline.json is the recorded benchmark that intentional skill changes must not regress silently.
Do not add trigger evals for this skill
It was found that the run_eval.py's methodology incorrectly failed to detect skill triggers due to requiring immediate skill execution rather than context exploration, which models almost always do regardless of description changes.
This skill's load-bearing invocation paths are the doc-parity Stop hook and the ai-review workflow's documentation pass. Natural-language triggering is a nice-to-have fallback, not the critical path.
Eval cases
Behavior cases run against the Seeder subsystem of bitwarden/server (util/Seeder/, util/SeederApi/, util/SeederUtility/), chosen because its code-to-docs mapping is known.
These evals are executed against a pinned server ref, d6c84a7562cc6b464de910dbf829690885500137. To increase stability in testing, evaluate against that reference point.
Each case is constructed so a run that skips a documented scope during the walk, token-edits its way past the gate, or emits a report shape that mishandles the outcome type (Update list, one-line Attest summary, or one-line Dismiss) fails a named expectation. Behavior cases assume the run happens inside a bitwarden/server checkout so the ground-truth docs are readable.
Case 10 exercises the review context (out-of-repo discovery against the contributing-docs corpus); all other cases are session cases. Cases involve live edits to a scratch checkout, so run them against a disposable clone or a git worktree, never a working tree you care about.
Each case's expectations array is the pass criterion — every expectation is graded independently. Denominators differ per case because they count expectations, not runs.
Running
Run with /skill-creator:skill-creator in benchmark mode (with-skill vs. without-skill) with a config-blind grader. Install the runner from the claude-plugins-official marketplace if it is not already present:
/plugin install skill-creator@claude-plugins-officialThe AI Review Guidelines want at least three iterations per case per configuration so pass rate and variance are both captured.
Regression check against the current baseline compares the aggregate summary and per-run pass rates; timestamps, models, and per-run evidence prose vary on every execution and are not signal:
proj='{run_summary, runs: [.runs[] | {eval_id, configuration, run_number, pass_rate: .result.pass_rate}]}'
diff <(jq -S "$proj" behavior-baseline.json) <(jq -S "$proj" result.json)An empty diff on that projection means no regression on the graded data. When a change is intentional and the new numbers are the new desired state, replace the baseline with the new results in the same PR as the skill change.
Updating the test surface
Update behavior-eval.json when the test surface needs to evolve: a new load-bearing rule in the skill, a Seeder doc restructure that invalidates a case's ground truth, or an expectation that turned out to be ambiguous. The eval set encodes intent, not implementation — when a run fails, fix the skill, not the case, unless the case itself is wrong.