18-dogfood-testing (Phase 5: Dogfood Testing)
Receipt entry (required):
"$PRINTING_PRESS_BIN" phase-receipt enter --file "$PHASE_RECEIPT_LOG" --run-id "$RUN_ID" --phase "18-dogfood-testing"MANDATORY when an API key is available. Do NOT skip or shortcut this phase.
Shipcheck verified commands start and return exit codes. Dogfood verifies the CLI produces correct, useful output for real workflows. These are different checks.
Step 1: Ask the user for depth
Present via AskUserQuestion:
"Shipcheck passed. How thoroughly should I test against the live API?"
- Full dogfood (recommended) — Complete mechanical test matrix across every leaf subcommand, including help, happy-path, JSON parse validation, output-mode fidelity, and error paths. Includes write-side lifecycle only with an approved disposable fixture/sandbox plan.
- Quick check — A compromise subset when the user explicitly wants speed or full dogfood would consume unapproved real-world cost/side effects.
Recommendation rule: Full dogfood is the default recommendation. Do not downgrade because of ordinary time cost; a few extra minutes is cheap compared with the generation run and the cost of shipping a broken CLI. Recommend Quick only when the user asks for speed or when full live testing would create unapproved real-world cost/side effects (paid credits, outbound messages, public posts, real orders, irreversible deletes, invites, bookings, charges). Potential mutation is not itself a reason to downgrade: if the user approves a test account/workspace/calendar/project or the CLI can create and clean up disposable fixtures, Full dogfood remains recommended.
There is no skip option when an API key is available. Phase 5 auto-skips ONLY
when the API requires auth AND no key is available: display "No API key
available — skipping live dogfood testing. The CLI was verified against exit
codes and dry-run only." Record that skip with the Step 4
dogfood --write-skip command. Do not hand-write phase5-skip.json.
For APIs with auth.type: none (or no auth section in the spec), Phase 5
is MANDATORY — the API is freely testable without any credentials. Do not
skip testing just because no API key was detected. No-auth APIs are the
easiest to test and the most embarrassing to ship untested.
LAN-only no-auth carve-out. Some no-auth APIs are real hardware or
private-network APIs that are testable only from the user's LAN (SSDP, mDNS,
RFC1918/private hostnames, localhost-shaped appliance endpoints). If Phase 5
cannot reach the hardware because the generation host is not on that LAN, do
not fabricate an API-key skip and do not hand-author phase5-acceptance.json.
Ask the user whether to hold the CLI or skip live dogfood and promote anyway.
Only when the user explicitly chooses the skip/promote path, record it with
the Step 4 command for lan-unreachable-from-generation-host.
Do NOT proceed without asking. Do NOT substitute an ad-hoc smoke test. If some commands cannot be exercised because fixture values are missing, classify them as BLOCKED_FIXTURE and file/fix the machine gap; do not use that as a reason to recommend Quick.
Step 2: Run the binary-owned test matrix
Full dogfood is not a judgment call about "enough." Run the Printing Press-owned live matrix so command enumeration, exit-code capture, JSON parsing, and acceptance-marker writing are deterministic:
cli-printing-press dogfood --live \
--dir "$CLI_WORK_DIR" \
--level full \
--research-dir "$API_RUN_DIR" \
--json \
--write-acceptance "$PROOFS_DIR/phase5-acceptance.json"research.json is written to $API_RUN_DIR (Phase 1.5e), the same directory Phases 11 and 12 pass.
$RESEARCH_DIR is $API_RUN_DIR/research and does not contain that file. Live dogfood
also reads the parent directory when <dir>/research.json is absent, but a missing
or unparseable file after that lookup fails the run instead of reporting novel-feature
coverage as passed.
Use --level quick only when the user selected Quick Check in Step 1.
Cookie / composed / session-handshake auth. The runner executes the binary
in a sandboxed HOME with no captured browser session, so a CLI whose auth is a
cookie jar will 401 every command unless you hand it the session. To exercise
the matrix for real, point the CLI's config-override env var (the
<PREFIX>_CONFIG-style override, e.g. <API>_CONFIG) at a config.toml that
carries the captured session, and export it before the dogfood --live call:
<API>_CONFIG="$SESSION_DIR/config.toml" cli-printing-press dogfood --live \
--dir "$CLI_WORK_DIR" --level full --research-dir "$API_RUN_DIR" --json \
--write-acceptance "$PROOFS_DIR/phase5-acceptance.json"If no captured session is available to the harness, the runner does not count
the 401s as failures: it records a clean skip verdict
(skip-cookie-auth-no-session) and writes phase5-skip.json with
skip_reason: "cookie-auth-no-harness-session" (see Step 4). The promote gate
accepts that skip marker, so do not hand-author or fabricate a pass marker for a
cookie-auth CLI you could not exercise.
OAuth2 refresh-token rotation (auth.type: oauth2_refresh). Pass
--auth-env <VAR> when the refresh token lives in the environment. When that
variable is refresh-token-shaped (REFRESH_TOKEN or *_REFRESH_TOKEN), the
runner copies it into a shared sandbox credentials.toml and strips those
rotating env vars from subprocesses so the first refresh persists for later
commands. A client-ID --auth-env (the oauth2_refresh publish default) is
left in place and is not written as refresh_token. Do not expect a static
env value to survive rotating identity providers across N subprocesses. If
invalid_grant still appears after a live pass, the runner aborts the matrix
with that diagnosis instead of recording hundreds of command failures; the
operator's stored refresh token may have been revoked and needs a fresh login.
The live dogfood runner enumerates the CLI's agent-context command tree,
runs help, happy-path, JSON-fidelity, error-path, and write read-back checks where applicable,
captures subprocess exit codes directly without shell pipes, and emits a
structured report with pass/fail/skipped counts. Before running the matrix, it
atomically rebuilds a selected binary that is older than the CLI's Go sources;
binaries at least as new as the sources are reused. Save the JSON report to:
$PROOFS_DIR/<stamp>-dogfood-results.json
A declared nonzero exit can count as a successful happy path and JSON check when it returns a nonempty JSON outcome without an error envelope (for example a documented NO-GO verdict). A usage error or silent declared exit does not establish real coverage. HTTP 403/404 responses that explicitly say a feature is not enabled for the current account are skipped with the live response retained as evidence; an ordinary missing resource remains a failure. Inspect that evidence rather than manually changing a failed row to skipped.
If the command exits non-zero, inspect the structured failures, fix the CLI, and
rerun live dogfood. The runner writes phase5-acceptance.json on every outcome
(status: "pass" on success, status: "fail" with a failure_summary block on
failure), so the Phase 5.6 gate always has a marker to read. Do not hand-edit
phase5-acceptance.json; it must come from the runner. Re-running shipcheck after a passing marker does not pass verify --fix while that fingerprint matches the tree. shipcheck --fix still repairs source and warns that the marker must be re-minted.
When hollow_features lists a mutating novel command that only previews unless
a confirm flag is passed, annotate it pp:preview-happy-path (see
Phase 3) and rerun without --allow-destructive; the
runner runs that preview for real. When a failure is an endpoint input no
portable fixture can satisfy (a continuation cursor, an account-specific path,
a resource URL), declare it as --flag=example-value in pp:happy-args so the
row lands as blocked-fixture instead of baking real account data into the CLI.
When publish validate reports hollow coverage for a command the live matrix
skipped on a positional, the detail names pp:happy-args. Set that annotation
to a real positional. When every positional is optional, flags alone are enough.
A novel command with no fixture stays hollow.
Quick check (auto-selected test subset):
doctor— auth valid, API reachable.- 3-5 list commands — return data, not empty.
sync --full→ data appears in local store.search "<term from synced data>"— finds results.- One list command with
--json,--select <fields>,--csv— all produce correct output. - One transcendence command — produces output that relates to the query (not just non-empty: verify relevance by checking output content contains query tokens or expected shape).
Full dogfood adds to the matrix:
- Every approved feature in the Phase 1.5 manifest gets a sample invocation with domain-realistic args.
- For every command that takes an arg, one error-path test.
- For every command that supports
--json, one JSON parse validation. - For write-side commands (when API key + user consent): create test entity with obviously-test data, verify in subsequent list/get, test one mutation, verify change.
- A live create/update that exits 0 records a
write_readbackrow. The row passes when a sibling get/show of the created or targeted id shows one submitted field holding that value, or a modification timestamp advanced while the field stayed unchanged. It fails withmutation reported success but read-back shows field "<name>" unchangedonly when a pre-read shows every comparable field still holding its previous value. The row is skipped, and does not fail the matrix, when no sibling read exists, the id cannot be determined, the submitted field is absent from the read, the value was normalized or transformed, another submitted field shows the write landed, or the read itself fails.
Context budget for long matrices
A full live matrix on a 35+ command CLI can outgrow a single agent's reliable context
window (~200k tokens). When the testing agent approaches that budget: finish the current
test row, append a handoff entry to $PROOFS_DIR (matrix position, failures found so
far, fixes applied, acceptance-marker path), and continue in a fresh agent that reads
only the handoff plus the dogfood results file — not the prior conversation. The
runner's acceptance-marker workflow is unchanged by a handoff: markers are written by
the runner on runner outcomes, never authored by an agent.
Step 3: Fix issues inline
When a test fails, fix it immediately — do not accumulate failures. Tag each fix:
- CLI fix — specific to this printed CLI
- Printing Press issue — should be fixed in the Printing Press (note for retro)
Step 4: Report and gate
Write a structured acceptance report and a machine-readable gate marker. The
JSON marker is required — Phase 5.6 and publish validate check for it
before promoting or publishing.
Acceptance Report: <api>
Level: Quick Check / Full Dogfood
Tests: N/M passed
Failures:
- [command]: expected [X], got [Y]
Fixes applied: K
- [each fix]
Printing Press issues: J
- [each issue for retro]
Gate: PASS / FAILRedact PII while authoring the report. When live API responses include an organization or workspace name, user email, assignee/collaborator name, or any other human-identifying string, describe the result generically instead of quoting the literal value:
- organization or workspace name -> "the test workspace"
- authenticated user email/name -> "the authenticated viewer"
- assignees or collaborators -> "the highest-loaded assignee" / "the project lead"
- team identifiers such as
ENGorT2are OK when they are structural keys
The Phase 5.6 manuscript scan and publish-skill PII scan are defense in depth; keep PII out of the acceptance report from the moment you write it.
Acceptance threshold:
- Quick Check: 5/6 core tests must pass. Auth (
doctor) or sync failure is automatic FAIL. - Full Dogfood: every mandatory test in the matrix must pass. A single broken flagship feature is automatic FAIL. Auth/sync failures are automatic FAIL.
Bugs surfaced in Phase 5 must be fixed now, not deferred. Do not offer the user a "ship as-is and file for v0.2" option when the fix is a 1-3 file edit. Present a "Fix now" (default), "Fix critical only", "Hold (don't ship)" set. Deferring bugs to a v0.2 backlog is an anti-pattern — context is freshest in-session, and a backlog that may never be revisited ships known-broken CLIs.
Gate = PASS: proceed to Phase 5.5 (Polish).
Gate = FAIL: fix issues inline (Step 3) and re-run failing tests, up to 2 fix loops. If the gate still fails after 2 loops, put the CLI on hold:
"$PRINTING_PRESS_BIN" lock release --cli <api>-pp-cliThe working copy remains in $CLI_WORK_DIR. Record the 18→20 hold handoff in
the completion block below, then proceed to Phase 5.6
to archive manuscripts (archiving still happens on hold). Do not follow the
canonical Next: pointer. Tag the failure reason in the acceptance report so
the next run can learn from it.
See references/dogfood-testing.md for additional guidance on common failure patterns and what NOT to test.
Write:
$PROOFS_DIR/<stamp>-fix-<api>-pp-cli-acceptance.md
For every outcome (PASS or FAIL), the runner writes:
$PROOFS_DIR/phase5-acceptance.json
{
"schema_version": 1,
"api_name": "<api>",
"run_id": "<run-id>",
"status": "pass",
"level": "quick|full",
"matrix_size": 42,
"tests_passed": 42,
"tests_failed": 0,
"auth_context": {
"type": "none|api_key|bearer_token|cookie|composed|session_handshake",
"api_key_available": true,
"browser_session_available": false
}
}On Gate: FAIL the same path is written with status: "fail" and a
failure_summary block grouping failures by category
(transport_error / http_4xx / http_5xx / exit_nonzero /
output_mismatch / other) plus the list of contributing commands. The
Phase 5.6 gate routes this marker to the hold path; do not promote.
For level: "quick", tests_failed may be 1 only when the Quick Check
threshold still passed (matrix_size: 6, tests_passed >= 5) and the miss was
not auth or sync related. For level: "full", tests_failed must be 0.
Proof-backed coverage for state-dependent features. A novel command
annotated pp:verified-by-proof (see Phase 3) is
listed as "proof_covered_features": [{"command": "<cmd>", "proof": "<file>.md"}]
instead of in hollow_features when its proof file sits in $PROOFS_DIR beside
the acceptance marker. Write that proof before running dogfood --live. It
must record:
- every command run for the write lifecycle, verbatim, with its exit code (the write that creates the state, the state-dependent command itself, and any verification read);
- the sandbox used (a disposable folder, test account, or scratch resource, described generically, never real account data or PII);
- the cleanup performed and its result.
Use it only for features that need state a prior real write created. It is not
a substitute for a runnable happy path, and the gate fails if a listed proof
file is missing beside phase5-acceptance.json.
If Phase 5 is legitimately skipped because the API requires API-key, bearer,
or OAuth2 auth (api_key, bearer_token, oauth2, oauth2_refresh) and no
credential was available, record the skip with the
Printing Press. Do not hand-author the marker and do not compute a digest.
<AUTH_ENV> is the credential variable from .printing-press.json
(auth_env_vars). The command refuses when that variable, or any manifest
credential variable, is set, and it refuses this reason for none, cookie,
composed, and session_handshake auth. The marker includes
source_fingerprint and source_files. A later .go edit makes it stale.
Replacing only the stamped var version = "..." string in internal/cli/root.go,
internal/cli/version.go, or cmd/<name>-pp-mcp/main.go does not; any other
edit in those files does. Rerun the command after the tree is final.
"$PRINTING_PRESS_BIN" dogfood \
--dir "$CLI_WORK_DIR" \
--write-skip "$PROOFS_DIR/phase5-skip.json" \
--reason auth_required_no_credential \
--auth-env "<AUTH_ENV>"If Phase 5 is legitimately skipped because a no-auth API is LAN-only and the
generation host cannot reach the user's LAN hardware, record that skip the
same way. The command sets auth_context.local_network_only and the
fingerprint.
"$PRINTING_PRESS_BIN" dogfood \
--dir "$CLI_WORK_DIR" \
--write-skip "$PROOFS_DIR/phase5-skip.json" \
--reason lan-unreachable-from-generation-hostIf a cookie / composed / session-handshake CLI could not be exercised because
no captured session was available to the harness (the sandboxed HOME has no
cookie jar, so every command 401s), dogfood --live writes phase5-skip.json
on a skip-cookie-auth-no-session verdict. Do not hand-author that marker,
and do not pass cookie-auth-no-harness-session to --write-skip.
Prefer the inject-session path in Step 2 (pass the captured session via the
<API>_CONFIG override) so the matrix runs for real; the skip is the floor when
no session is available.
Do not write a skip marker for ordinary auth.type: none cloud/public APIs.
No-auth APIs are testable and require phase5-acceptance.json unless they match
the LAN-only carve-out above. Do not use missing API key
(auth_required_no_credential) as the skip reason for cookie, composed, or
session-handshake auth; inject the session, or rely on the runner-emitted
cookie-auth-no-harness-session skip when none is available.
Run exactly one of the two completion blocks.
On a hold, after the gate still fails and the lock is released, record the
archive handoff instead of following Next:. Point --evidence at the failed
acceptance marker:
"$PRINTING_PRESS_BIN" phase-receipt complete --file "$PHASE_RECEIPT_LOG" --run-id "$RUN_ID" --phase "18-dogfood-testing" --next "20-promote-and-archive" --evidence "$PROOFS_DIR/phase5-acceptance.json" --note "hold: <one-line reason>"Then proceed to Phase 5.6.
On a pass, or an allowed skip, record the canonical handoff before following
Next:. Point --evidence at the live dogfood acceptance JSON. If dogfood used
an allowed skip, record it with --skip --note "<allowed reason>" and point at
the skip marker instead:
"$PRINTING_PRESS_BIN" phase-receipt complete --file "$PHASE_RECEIPT_LOG" --run-id "$RUN_ID" --phase "18-dogfood-testing" --evidence "$PROOFS_DIR/phase5-acceptance.json"Next: phases/19-polish.md