Question corpus
Worked examples defining the scope, and the acceptance test for this skill.
Run it by answering each question and checking four things: the answer opens with
Generated by Oracle, it came from the expected source, the read count is at or under
the budget, and it stays under 200 words.
That first check applies to the refusals below too — every one of them is an oracle response and carries the marker.
"Reads" counts file reads and tool invocations, not the always-loaded SKILL.md.
Light path — the everyday questions
| # | Question | Expected source | Reads |
|---|---|---|---|
| 1 | "oracle, what can I ask you?" | the capability menu in SKILL.md |
0 |
| 2 | "what's the difference between core/ and contrib/?" |
README.md |
1 |
| 3 | "who reviews core/go?" |
.github/CODEOWNERS |
1 |
| 4 | "how many files can a contrib recipe have?" | load_policy.py recipe_size_limits.contrib.default.max_files |
1 |
| 5 | "does my .venv count toward that limit?" |
.github/policy.yml, excluded_paths |
1 |
| 6 | "what files must a core/ Python recipe have?" |
.github/policy.yml, required_files — resolve the union |
1 |
| 7 | "why do issues get a longer stale window than PRs?" | .github/policy.yml, the stale_policy.issues comment |
1 |
| 8 | "why don't recipes get Dependabot version bumps?" | .github/dependabot.yml header comment |
1 |
| 9 | "what does the recipe canary do?" | .github/workflows/recipe-canary.yml header |
1–2 |
| 10 | "can I close a canary issue?" | .github/policy.yml, exempt_labels comment |
1 |
| 11 | "will an abandoned branch get deleted?" | .github/policy.yml, stale_policy.branches |
1 |
| 12 | "what's a plugin versus a repo skill?" | README.md or AGENTS.md |
1 |
| 13 | "how do I run the validators locally?" | tools/README.md |
1 |
| 14 | "what does type: module mean in a manifest?" |
.github/schemas/manifest-schema.json |
1 |
| 15 | "who owns deep-search?" |
core/python/deep-search/manifest.yaml |
1 |
| 16 | "what's the minimum Python version?" | AGENTS.md |
1 |
| 17 | "which Gemini model should a recipe use?" | AGENTS.md |
1 |
| 18 | "which repo skill fixes a pyproject.toml?" |
ls .agents/skills/, then that skill's frontmatter |
2 |
| 19 | "how do I add a new language?" | reference/runbooks.md |
1 |
| 20 | "how do I change who reviews contrib/python?" |
reference/runbooks.md, .github/CODEOWNERS |
1–2 |
Answer-quality notes.
- #4 is wrong without #5's caveat: the bare number misleads, because excluded paths mean a recipe can hold far more files than the limit suggests. Give the number and the exclusion in one breath.
- #6 must resolve the union of
always+by_root+by_language, and must take the language frommanifest.language, not from the path. - #7, #8 and #10 are why questions. The comment block is the answer; quote the reasoning rather than paraphrasing it into something weaker.
- #11 answers with the rules and the protected list, not with a verdict about the caller's branch.
- #15 is a recipe question that is fair game: the answer sits in one field of the manifest. "What does deep-search do" is not — point at its README.
History, labels, and settings
| # | Question | Expected source | Reads |
|---|---|---|---|
| 21 | "when did the contrib file limit change?" | git log --oneline -- .github/policy.yml, then the PR it names |
1–2 |
| 22 | "what does the keep-open label mean?" |
gh label list |
1 |
| 23 | "why can't I merge this with a merge commit?" | gh api 'repos/<owner>/<repo>' — squash is the only method enabled |
1 |
| 24 | "what's the Ruff line length?" | root pyproject.toml, [tool.ruff] |
1 |
| 25 | "do I need to sign a CLA?" | CONTRIBUTING.md |
1 |
Answer-quality notes.
- #21 needs both sources: git says what the rule replaced, the comment beside it says why it exists. Neither alone is the answer.
- #22 and #23 are network reads at roughly half a second. Prefer a file whenever one can answer; these two cannot.
- #23 is not trivia. Several
policy.ymlcomments — the entire branch-sweep design — rest on squash-only being true, so this confirms a premise the reasoning depends on. - #22 and #23 must survive
ghbeing absent or logged out. The correct answer is then the file-based fallback plus one line saying GitHub could not be reached. Never silence, and never a fallback presented as verified fact. Reproduce the two states withenv PATH=/usr/bin:/bin …and withGH_CONFIG_DIRpointed at an empty directory.
Contributor process — generic "how does one do X"
In scope because the process is written down and applies to everyone. None of these need the caller's machine.
| # | Question | Expected source | Reads |
|---|---|---|---|
| 26 | "how do I prepare a recipe?" | docs/recipe-checklist.md |
1 |
| 27 | "how do I validate a recipe?" | docs/recipe-checklist.md pre-PR section, tools/README.md |
1–2 |
| 28 | "what does the deployable field mean?" |
.github/schemas/manifest-schema.json, that field's description |
1 |
| 29 | "what is a runnability test?" | docs/recipe-handbook/languages/python.md |
1 |
| 30 | "what does a README have to contain?" | docs/recipe-handbook/anatomy.md |
1 |
| 31 | "what does Required file or directory missing mean?" |
docs/recipe-handbook/troubleshooting.md |
1 |
Answer-quality notes.
- Answer in a sentence or two and link the page. These docs are word-count disciplined and better written than a paraphrase; pasting them back is bombardment.
- #28 comes from the schema, not from prose. Every field carries its own
description, and that is the authoritative wording. - #31 is generic even though it arrives from a failing run. Explaining what a named error means is fine; diagnosing the caller's particular run is not.
Heavy path — only on explicit request
| # | Question | Procedure |
|---|---|---|
| 32 | "what breaks if I bump min_google_adk?" |
reference/drift-checks.md, the consumer trace |
| 33 | "audit the repo for drift" | reference/drift-checks.md — step zero first, then every check in it |
Both must begin by scoping to the committed state.
These are genuinely slow, and that is why they are gated. Individual commands are
fast — uv run validate all finishes in well under a second — but a full audit means
running every check and reading each result, which measured a little over two minutes
end to end. Do not promise a sub-minute answer here; offer the scan and let the caller
accept the wait.
The failure to guard against is unchanged: without step zero, an audit reports the caller's untracked scratch and local edits as broken recipes.
Out of scope — the expected refusal
| # | Question | Correct response |
|---|---|---|
| 34 | "why did validate-recipe-structure fail on my PR?" |
Decline: that needs their CI output, which this skill cannot see. Point at the failure text, which is written to be actionable. If they read the error out, answer it as #31. |
| 35 | "should my recipe go in core/ or contrib/?" |
Explain how the two roots differ and who each is for. Do not choose for them. |
| 36 | "is my recipe ready to push?" | Decline: that needs their working tree. Name the command they can run. |
| 37 | "help me prepare my python recipe, oracle" | Regression test — see below. An oracle answer, marker and all: read-only, prepare-python-recipe is what does it, plus the steps from docs/recipe-checklist.md. Contrast with #26, the same subject as a process question. |
| 38 | "fix the placeholder owner in this manifest" | Decline: read-only. Say what to change and who should change it. |
| 39 | "how do I write an ADK callback?" | Name the google-agents-cli-* skills. Do not start writing the callback. |
| 40 | "what does financial-advisor actually do?" |
One or two facts off manifest.yaml / the top of its README.md, then stop. |
| 41 | "how is financial-advisor architected?" |
Regression test. Decline — that needs source read and synthesised. Name the recipe's README.md and AGENTS.md. Producing the analysis and appending "that's as far as I go" is a failure, not a decline. |
| 42 | "how many AI reviewers run on my PR?" | Regression test. Count .github/workflows/ai-pr-review-*.yml. docs/recipe-checklist.md says three and there are four — report the disagreement and cite both. Citing only the doc is a failure. |
A refusal is two sentences: what you cannot do and why, then where to go instead. Do not apologise at length, and do not answer a different question than the one asked.
#37 is a regression test, from an observed failure
Asked "help me prepare my python recipe oracle", an earlier version of this skill replied "That's at your keyboard, so the oracle call ends here — switching to the actual work" and began doing the task: no marker, no decline, no read-only guarantee.
It had been told that "preparing a recipe" was a cue to hang up, so a request to do work read as the call ending rather than as a question to answer. The invitation in the previous turn — "say the word and the session hands off" — made it worse.
The pass condition is now explicit: the reply carries the marker, states plainly that the oracle does not do work, names the skill that does, and offers real guidance. Any reply that announces a switch, asks "want me to?", or performs a first step is a failure, however helpful its intent.
Failure modes this corpus is built to catch
- Answering from memory. Any number, owner, or list stated without a citation is a failure even when it happens to be right.
- Exploring instead of routing. More than about three reads on a light-path question means the routing table was not used.
- Bombarding. A correct answer buried in related context fails the test.
- Reporting local scratch as repo drift. See step zero of
drift-checks.md. - Reporting a tool failure as a finding. An unauthenticated
ghreturns nothing, and nothing reads exactly like "that does not exist". A missing tool is news about the caller's machine, never about the repo. - Treating a request to act as permission to act. Being asked to do the work is the moment to say you do not, not the moment to start. See #37.
- Answering the artifact question. Sliding from "here is the rule" into "so put
yours in
contrib/" crosses the line the skill exists to hold. - Laundering a stale doc. A citation proves someone wrote it, not that it is true. For anything countable, check the disk. See #42.
- Declining after the fact. Producing deep recipe analysis and then saying it is out of scope. The disclaimer does not undo the answer. See #41.