ci-speedup - Architecture
Contributor/maintainer-facing guide to how the ci-speedup skill is built
and why. Read SKILL.md first (the canonical contract); this doc explains the
implementation behind it so you can extend the skill safely.
The conceptual heart of the skill is the wall-clock lever model - a cascade of physical bounds that decides whether a finding actually shortens a developer's wait. It gets its own prominent section below.
Contents
- Purpose & scope
- Pipeline overview (scan → collect_runs → render → gap-fill → verify) — incl. §2.1 adaptive run sampling, §2.1a config-era boundary, §2.2 fetch orchestration
- Data model — findings.json
- The two metric axes
- The wall-clock lever model — a cascade of physical bounds — incl. §5.4 OPT81, a runner-class gap and the last-resort runner advisory
- Admission gate & advisory routing
- Evidence, provenance & verification
- Reproducibility & determinism — incl. §8.1 what the data pass does NOT re-fetch
- Testing strategy
- Status of the planned bounds
- The structural / critical-path track
- The blocking-path report — incl. §12.0 the Long pole map cascade, §12.3 leaf detectors, §12.7 the coverage-gap fallback
- Cross-references
1. Purpose & scope
ci-speedup audits a repository's GitHub Actions workflows against a 109-pattern catalog — 97 hygiene/data-driven patterns (OPT1–OPT9, OPT11–OPT66, OPT68, OPT69, OPT76, OPT77, OPT79–OPT82, OPT86, OPT88–OPT91, OPT93, OPT94, OPT96–OPT98, OPT100–OPT103, OPT105–OPT114) plus 12 structural / critical-path patterns (OPT70–OPT75, OPT78, OPT83–OPT85, OPT87, OPT104; routed from the measured long pole; see §11) — and produces a root-cause-analysis markdown report with measured impact on two axes: developer wall-clock wait (the ranking axis) and runner-minutes (the cloud bill). Detection, ranking, and every measured number are deterministic — no LLM in detection, scoring, the critical-path spine, or the cross-run checks. The one exception is the gap-fill (SKILL.md phase 4a): when a drilled pole's log matches no catalog detector, the agent running the skill writes a labelled, log-grounded root-cause reading so the pole still yields a breakdown + prompt instead of a dead-end — it never touches detection, ranking, or the magnitudes. ci-speedup does not prescribe fixes: each finding ships a ready-to-paste agent prompt the user hands to their own coding agent, which sees the repo and reasons out the fix. Everything here is self-contained in this skill directory; the report links out only to the public catalog and to GitHub job-log permalinks.
2. Pipeline overview
findings.json is the single artifact threaded through every stage. Each stage
reads it, enriches it in place, and atomically rewrites it (.partial →
rename), so a crash mid-write never replaces a good file.
repo clone (.github/workflows/*.yml)
│
▼
┌──────────────┐ static detectors (per-file / declarative / cross-workflow /
│ scan.py │ repo-file / source-grep). Parses the catalog, emits a
│ (Phase 2) │ finding per hit. Reports catalog_patterns_without_detector.
└──────┬───────┘
│ findings.json (findings[], scanned_workflows, scan_incomplete, …)
▼
┌──────────────┐ gh sampling (_sample_runs, _monthly_volume) + data-driven
│ collect_runs │ detectors (OPT24/25/43/48/51) + STRUCTURAL routing
│ (Phase 3) │ (OPT70-75: decompose long pole, required-checks, §11) +
└──────┬───────┘ two-axis sizing (_SIZING / _size_finding) + the wall-clock
│ bound cascade. --with-logs anchors cache findings in logs;
│ --data-dir also captures the blocking-path drill bundle (§12).
│ ── run.py orchestrates Phases 2-3 in ONE process ──
▼
┌──────────────┐ renders the measured-spine RCA report: a metadata header + a
│ blocking_ │ Contents TOC, then the critical-path spine — per gating pole a
│ path.py │ gate → step-timeline → root-cause drill (ASCII), each ending in
│ (Phase 4) │ a ready-to-paste AGENT PROMPT (no prescribed fix). Queue
└──────┬───────┘ time gets its own "Pre-start wait" wall-clock section;
│ measured+certified, source-backed runner-minute findings
│ render in "Runner-minute reductions (wall-clock-neutral)"; residual
│ hygiene demotes to "Also noticed". Strips em-dashes at the
│ render boundary.
▼
┌──────────────┐ LLM gap-fill (§12.7): for any drilled pole whose log matched
│ agent + │ NO catalog detector, the agent running the skill reads the
│ --analysis │ captured log and writes a grounded, LABELLED root-cause analysis
│ (Phase 4a) │ + tailored prompt, passed back via blocking_path.py --analysis.
└──────┬───────┘ The ONLY non-deterministic step; never touches the measured
│ numbers. No pole ships as a "no pattern matched" dead-end.
▼
┌──────────────┐ PASS/FAIL invariant checks on the rendered report + JSON
│ verify_report│ (Phase 5): header arithmetic, anchors, no-emdash, provenance,
└──────────────┘ every finding carries a prompt, diagnosis never dead-ends.run.py collapses the deterministic phases (scan → collect_runs) into a single
process and one findings JSON, then prints a summary.py digest to stdout (the
gating resolution + addressable poles + the exact blocking_path.py render
command, so the agent acts on it instead of re-spelunking the JSON or re-probing
gh), and the agent calls blocking_path.py to render.
The report renders to an internal/session path beside the scratch
findings.json (run.py's --report-out default) and is rendered + verify-gated
there on every run. Issue #18 made the artifact opt-in: it is surfaced into the
user's working tree (./ci-speedup-findings-report.md) only when they pick "save
the full report" at the phase-6 close — the honesty gate (verify_report) fires
regardless, so opting in only copies an already-verified .md.
There is no fix-generation phase. ci-speedup stops
at the measured root cause; the report's per-pole agent prompts are the
hand-off — the user pastes one into their own coding agent, which sees the repo
and reasons out the fix. Because that prompt is the entire contract, the
renderer owns two blocks in it that no builder may omit or paraphrase: the
"does NOT prescribe the fix" disclaimer and the no-weakening rail
(_NO_WEAKENING_LINES). Any new prompt builder must emit both —
verify_report.py counts each once per rendered prompt, so a rail-free path
fails the gate on a real audit, not just in unit tests. Detection + the measured report are deterministic and
reproducible from the findings JSON; the only non-deterministic step is the
phase-4a LLM gap-fill for poles the catalog can't analyse (labelled + log-
grounded, never touching the measured numbers). record_timing.py lets the orchestrator
merge a measured duration for any non-scripted phase into the JSON's timings
block, so "where did the time go?" is answerable from the artifact alone.
blocking_path.py (§12) is the single, data-first renderer: it reads the
findings JSON plus an optional drill bundle (logs/timelines/magnitude samples)
and answers "why is the merge slow?" as a gate → step-timeline → root-cause
drill-down. (An earlier catalog-spine renderer, report.py, was retired when
the measured critical path became the report's spine; blocking_path.py is now
the only renderer.)
2.0 The gh choke point (GhClient) — no silent drops
Every GitHub byte the skill reads passes through GhClient.json() / .text()
in collect_runs.py, so the correctness rules that protect the sample are
stated once, there:
The one rule underneath all of them: a truncated or incomplete sample must never be presented as clean. A failed fetch is a coverage gap (disclosed); an empty result is a fact (rendered). Collapsing the first into the second is the skill's cardinal bug, and it has a dozen possible sites — this is the list of them.
Full pagination, or a loud gap (
_paginate). The list endpoints (…/actions/workflows,…/runs/{id}/jobs,…/commits/{sha}/check-runs) are walked to completion. Requestingper_page=100and keeping page 1 silently truncated the sample: measured on better-auth/better-auth @ 6f20f44, check-runs reportedtotal_count = 103and returned 100, so 3 checks vanished from the critical-path sample — any of which could have been the merge pole, which would have made the computed gate wrong. The walk stops on a short page — one GitHub could not fill, which by definition ends the collection.total_countis a warn-only cross-check, never a stop: it is a field we don't control, and if an endpoint ever under-reports it (thefilter=alljobs view is the one to worry about), stopping onlen(items) >= total_countwould silently drop pages 2+ — reintroducing the exact truncation this exists to prevent. Cost: one extra call on an endpoint whose item count lands exactly on a page boundary. Everything else FAILS the whole fetch (None, a disclosed gap), never a short-but-plausible list: a page that errors, a page that is malformed, and the_GH_MAX_PAGESceiling — reaching that means the stop condition is broken, so the accumulated items are of unknown completeness and the honest answer is a gap, not a list. The one exception is a page 1 that is explicitly empty (total_count == 0, or the list key present and empty) — a genuinely empty collection, which every caller already reads as "nothing ran". A body that merely lacks the key is not evidence of emptiness and fails like any other malformed page:_fixture_nameis lossy, so in replay a check-runs fixture can be served some other endpoint's body ({"default_branch": "main"}— nocheck_runskey, nototal_count), and the old "no key and no total" guard turned that into[]witherrors == 0: a clean critical path built on nothing. (Record mode now raises on a fixture-name collision rather than overwriting, and the one collision the pipeline actually issued —created=>=vscreated=<=, whose operators both transliterated to_— is spelled out (gte/lte) so the two opposite windows can no longer share a file.) Page 1's endpoint string is byte-identical to the pre-pagination spelling, so the record/replay fixture corpus (_fixture_name) still resolves; page 2+ appends&page=<n>(a re-recorded corpus must therefore carry those pages).A failed fetch never becomes an empty result (
_list_workflows,_run_listand the four run-list samplers over it). These returned[]on failure, which reads downstream as "this workflow has no runs" — so a rate-limited fetch made a workflow vanish from the audit while the report still rendered as though it had been measured, and a failed workflow-list made the audit believe the repo runs nothing at all. They now returnNone. A failed workflow list aborts the gh pass with a disclosed reason; a failed run list marks that workflowunavailable, skips the detectors that would otherwise clear it off an empty sample, and is disclosed by name (see the disclosure path below). A body that comes back MALFORMED — a dict with no (or a non-list)workflow_runs— is a failure too, not an empty workflow:_run_listused to launder that shape into[], witherrorsunbumped and no gap noted, which is the same silent deletion by a different door.The disclosure path: data → renderer → verify invariant. A gap the collector records but the ARTIFACT doesn't show is not a disclosure. All three links are load-bearing, and the middle one is where this broke:
- Data.
collect()stamps the two ways a workflow can leave the MEASURED sample —data_sources.run_list_fetch_failures(its run-list fetch died: no runs) anddata_sources.job_fetch_failures(its per-run job fetches ALL died: runs, but no job timing) — pluspartial_reason(the human sentence, which NAMES them) andpartial_kind(the SEVERITY, as a machine-readable key:sample_thinned|workflow_missing|collection_failed|gh_unavailable|static_only). Both sentences come from ONE helper (_partial_reason/_partial_kind, fed the UNION of the two gap lists) —collect()re-stamps coverage at the end (more gh calls happen after the first stamp), and an earlier re-stamp that re-derived the sentence inline clobbered the by-name disclosure back down to a bare "N gh API call(s) failed during collection". Never re-derive adata_sourcesvalue a helper already owns. All four keys are stamped on EVERY exit fromcollect(), even when empty — an early return that omitted them made the verify invariant SKIP (see link 3).gh_unavailablemeans the gh pass could not start (gh absent, logged out, or a sandboxed shell with no keychain) — NOT an API refusal.available()is agh auth statusprobe, and that probe makes a live API call to verify the token, so it ALSO returns non-zero when gh is installed and a token is stored but the API does not accept the credential — a secondary-rate-limit 403, a transport / 5xx error, or an invalid (expired / revoked / too-narrowly-scoped) token. Stamping that casegh_unavailable(a NOT-MEASURED kind) shipped a silent static-only report that read as a complete audit of a quiet repo while the collection had actually been refused. The up-front branch now asksGhClient.diagnose_unavailability()(OFFLINEgh --version+gh auth tokenprobes, so the diagnosis can't itself be rate-limited): a present binary + present token STRING meansapi_blocked→ the LOUDcollection_failedkind (withgh_availableTrue, since gh IS available — the API refused it), the same severity the mid-collectiongave_upabort already uses; only a truly absent binary/token keeps the quietgh_unavailable. The offline probes cannot tell a rate limit from a dead token (gh auth tokenproves a token is stored, never that the API would accept it), so the disclosure names both remedies — wait out a rate limit, or re-authenticate — rather than asserting one; and an ambiguous probe failure (a hunggh auth token,TimeoutExpired) also routes LOUD, never back to the quiet path that would re-open the silent drop. A sandboxed / no-network shell can land on EITHER side (no keychain →gh_unavailable; a readable token →api_blocked), so both reasons quote gh's own error line (_gh_probe_error_line, which skips gh's bare host-name line) and name the "retry with network / host access" remedy.gh_unavailabletherefore never reads as a quiet repo: the report renders a "run history was not read" banner, not the dormant-repo one. - Renderer.
blocking_path._coverage_noterenders it at the severity the DATA says, branching onpartial_kindand never string-matching the prose. Onlysample_thinnedmay carry the minimizing suffix ("a few runs/jobs are absent … marginally fewer runs"); stapling that onto aworkflow_missing/collection_failedreason ("NO workflow could be measured") had the report contradicting its own disclosure. The note fires whenever apartial_reasonis set (not only whengh_error_countis non-zero — a gap is not always a failed CALL), and it NAMES every workflow in EITHER gap list, re-derived from the stamp. The static-only banner is the load-bearing one: "an archived, brand-new, or low-activity repo whose run history aged out" is a confident WRONG diagnosis of a broken fetch that invites the reader to conclude their CI is quiet. It is now gated on the INVERTED predicate_measurement_is_broken— the quiet-repo banner may render ONLY when the gh pass never ran (static_only/gh_unavailable, which the report already announces up top); EVERY other kind (including a merely thinned sample with no spine, and any severity key the renderer has never heard of) defaults to the loud "Collection FAILED — this is not a quiet repo" banner. Gating onpartial_kind == collection_failedalone (as an earlier round did) closed the hole for ONE of the five kinds and left a total run-list wipeout — which stampsworkflow_missing— rendering the dormant-repo lie under a passing verify gate. The no-static-findings sub-case (which returned "" → the bare one-line note) now renders the banner too. - Verifier.
verify_report.check_run_list_gaps_namedfails any report that does not name every workflow in a non-empty gap list (bothrun_list_fetch_failuresandjob_fetch_failures). That is the invariant that makes the clobber uncatchable-again: the pre-existing gh-error guard only required the error COUNT to appear, which the clobbered sentence satisfied. It fails CLOSED on a missing stamp whenever the gh tier ran — it used to skip, and since the stamps were only written on the main path, the invariant SKIPPED on all six committed worked examples and ran only in its own unit tests (a guard that skips corpus-wide is a guard that isn't there).workflow_activity: {"status": "unavailable"}is a machine-readable stamp on findings.json for the same gap; no renderer reads it, and it is NOT the disclosure.
- Data.
A job-fetch WIPEOUT cannot headline off a queue-inflated span. When every per-run job fetch for a workflow fails, its
critis empty, so its checks would fall back to CHECK-RUN SPAN timing — which is queue-inflated (an 80s job whose check-run reads 1871s, because the span starts at check CREATION). That inflated number can outrank the true gate and headline the report — off a value we failed to measure. So while anyjob_fetch_failuresentry stands, a check with no sampled job timing is DROPPED from the critical-path spine (_bar_queue_inflated_fallback) rather than timed by its span. The bar is spine-wide because a check-run carries no workflow path to attribute it back to the wiped workflow; every wiped workflow is NAMED andpartial_kindis severe, so the spine reads as incomplete instead of quietly headlining the wrong gate. With no wipeout, this is exactly the old fallback behaviour.A sustained FAILURE ENDS the run; it never hangs it (the global breaker). The attempt budget is per CALL, so on its own it bounds nothing in aggregate: with a bucket exhausted (or the API 5xx-ing, or hung) and no terminal condition, every remaining call burns its attempts, gives up — and the next starts fresh at attempt 1. Several hundred queued calls is HOURS of wall-clock, i.e. a fast wrong answer traded for a hang. The timeout case is the worst: each attempt costs the full
timeout(60s json / 90s text), so a hung API is a multi-hour grind. So the "better to fail loudly than wait" rule is applied at the CLIENT: after_GH_MAX_GIVEUPSconsecutive exhaustions of ANY retryable class (rate limit, 5xx, OR timeout — an earlier round fed the breaker rate limits only, which left the 5xx/timeout outage completely unbounded), or once cumulative backoff across BOTH the rate-limit and transient paths passes_GH_TOTAL_BACKOFF_BUDGET_S, the client is terminallygave_up— every later call short-circuits toNone+ a counted gap with no sleep and no subprocess, andcollect()aborts through the SAME disclosed-coverage-gap path the workflow-list failure uses (tiers_run: [],partial_kind: collection_failed, scan.py's static findings intact). A success resets the consecutive count, so the breaker fires on a sustained failure, not on two unlucky calls. Measured over 200 endpoints: every class now trips in ≤ 6 subprocess calls.An EXPECTED absence is not an error.
allow_missing(bothjson()andtext()) means "a 404 here is expected" — an admin-only rulesets endpoint, or a job log GitHub has deleted (retention-days, default 90; a pinned--created-beforewindow can sit entirely outside it). Before this, 4 guaranteed 404s on better-auth's skipped jobs becamegh_error_count: 4and a phantom partial-coverage banner on a fully-successful run, which devalues the banner exactly when it's real.The log skip gates on whether the job STARTED, not on what it concluded (
_job_has_log). No log: askippedjob, a job stillqueued/in_progress(conclusion: null— in-flight runs ARE sampled), and a job cancelled before it started. A job cancelled MID-RUN does have a partial log that GitHub serves 200 — and since the drill picks its representative job by DURATION and never filters on conclusion, a long cancelled job can BE the drilled pole. Skipping it on conclusion alone would delete that pole's drill-down: a coverage regression inside a coverage fix.A rate limit is loud, retried, shared, and counted (
_classify_gh_failure,_invoke). A 429 / secondary-limit 403 used to be indistinguishable from a 404: no retry, no backoff,Nonereturned,logger.debug(invisible at the default INFO level). A throttled run therefore shrank its own sample and still rendered a confident, plausible, wrong report. Now: every live call runsgh api -i, so the failed response's OWNretry-after/x-ratelimit-resetarrives with the failure — one request, no extra probe fired at the endpoint that just blocked us. Retried up to_GH_MAX_ATTEMPTS, floored at GitHub's documented one minute (anx-ratelimit-resetdescribes the PRIMARY bucket; a secondary block needs ≥60s regardless, or the whole attempt budget burns in seconds), jittered, capped at_GH_MAX_BACKOFF_S, and published to every worker (_blocked_until) so one thread's discovery pauses the shared fetch pool instead of eight threads hammering a blocked endpoint in lockstep. The rate-limit KEYWORD alone is decisive — it does not wait for the HTTP status to parse, since a false positive costs one retry and a false negative costs a silently truncated report._classify_gh_failurealso reads the response HEADERS-ialready returned: a 403/429 carryingx-ratelimit-remaining: 0or aretry-afteris a rate limit regardless of gh's prose. Without that, a 403 whose message gh renders without the keywords classified asforbidden— no retry, and on anallow_missingendpoint (every job log) not even counted: measured, 200 rate-limited job logs, 0 errors, silent. 5xx and timeouts (the data exists; we didn't get it) retried at 1/2/4s + jitter, and their waits are charged to the same run-wide backoff budget the breaker reads. Everything is logged at WARNING, and exhausting the retries counts towarderrorseven on anallow_missingendpoint — being blocked from data that exists is a real coverage gap, not an expected absence. This legibility is the precondition for raising the fetch concurrency: a wider pool is only safe if throttling can't hide.
2.1 Adaptive run sampling (shallow → deepen-to-convergence)
The dominant gh cost is one GET /runs/{id}/jobs per sampled run. Sampling every
workflow at full --max-runs (default 20) is wasteful: a check's identity is
stable within ~10 runs while only its exact p50 needs the full sample. So
collect_runs runs two passes (--shallow-runs, default 10, clamped ≥1):
- Shallow — fetch jobs for the first
--shallow-runsruns of every workflow (concurrent), and stash each workflow's un-fetched deeper runs. - Deepen, iterating to convergence — rank every candidate's concurrent-check
keys (each job's p50, plus the long-pole p95 so a fast-median/high-tail bimodal
gate surfaces) across the eligible workflows, deepen the owners of the top
_DEEPEN_TOP_CHECKS(12 ≥ the renderer'ssrc[:9]chart) to--max-runs, recompute, and re-rank — repeating until the top region is entirely full-depth (_deepen_candidates,_deepen_check_keys, the loop incollect()).
Eligibility is full-breadth (events_by_wf ∩ _DEVELOPER_EVENTS), NOT the
shallow event_scope — so a [push, pull_request] workflow whose newest runs are
all push can't be hidden from deepening and measured against the wrong run
population. Only the per-run JOB fetch is depth-gated; the run-list (1 call/wf),
_monthly_volume, events, and the PR-sha pool stay at full breadth.
When the PR check-run sample proves a declared-PR check ran on sampled PRs but
the workflow-job sample for that workflow still has no developer-event run,
collect_runs keeps the check in the PR spine using the sampled PR check-run
duration. It stamps the pole timing_source=pr_check_runs and withholds the
workflow-job step drill, because the only available job steps came from
all-events push/schedule timing and would fabricate a PR root cause. The
structural router follows the same rule: no OPT70/72/75 root cause is inferred
from those all-events job steps.
Guarantee + the one honest limit. Convergence makes the gate, drilled poles,
the cross-workflow floor, and bimodal flags depth-invariant (== a full pass) —
validated by Monte-Carlo subsampling of depth-20 ground truth on five repos
(gate/drill/floor/bimodal = 100%) plus end-to-end diff vs a forced full pass.
PR33/#174 adds the cost-axis equivalent after the PR19/PR32 panel gate passed:
collect_runs first builds the shallow runner_minute_spine, then
_cost_deepen_candidates_from_spine ranks workflows by summed
billable-equivalent minutes/month, fetches the selected workflows' remaining
sampled runs to --max-runs, and rebuilds the runner-minute source block from
that deeper sample. This deepens the cost-spine clone only; it does not mutate
the already-produced wall-clock timing, queue, or finding-level evidence.
Off-path hygiene findings (OPT24/33/43/45) aggregate over ALL workflows, so
for capped workflows outside the wall-clock-deepened set they still rest on the
shallow sample and are approximate (queue-time p90 can swing run-to-run).
The runner-minute source block tracks its own cost-spine full-depth/shallow
sets separately. This is disclosed, never silent: the provenance cost line
states the shallow/deepened split, and the report carries one ⚠️ "re-run with
--shallow-runs <max> to confirm" note whenever a capped workflow was left
shallow on either axis — normally inside the Pre-start-wait and/or Also-noticed
sections, and emitted standalone when neither of those host sections renders
(PR-P2/#186 closed that corner, where an honest render silently dropped the
note). The cost-spine half is verifier-enforced:
verify_report.check_cost_spine_shallow_disclosed re-derives the disclosure's
workflow count from data_sources.cost_spine_shallow_workflows (the names
list, never the count stamp), so a renderer regression that drops it is a
verify FAIL, not a style drift. --shallow-runs >= --max-runs forces a single
full pass. Provenance fields: shallow_runs,
max_runs, shallow_capped, capped_workflows, pole_candidates,
deepened_workflows, deepen_converged, cost_deepen_candidate_workflows,
cost_deepened_workflows, full_depth_workflows,
shallow_remaining_workflows, cost_spine_full_depth_workflows, and
cost_spine_shallow_workflows.
PR19 makes that threshold reproducible without changing shipped behavior:
maintainers/ci-speedup/scripts/measure_bill_pole_convergence.py compares an
adaptive findings JSON against a forced full-depth findings JSON, applies the
PR18 selector to the adaptive spine, and simulates replacing selected workflows
with full-depth rows. The default gate requires the selected workflows to cover
at least 95% of full-depth billable-equivalent minutes and the simulated
deepened total to sit within 5% absolute error of the full-depth total. It also
fail-closes unless the paired artifacts have the expected adaptive/full
max_runs/shallow_runs provenance, the same repo, the same non-empty pinned
sampling window, complete cost-spine coverage, and zero cost-spine job-fetch
failures. Missing or malformed panel artifacts, wrong-side filenames, and empty
panels are measurement failures, not clean passes.
PR35 adds the local discovery counterpart for bill-only gaps. When the
collect_runs.py CLI runs from a tracked maintainer source checkout and the
final runner_minute_spine is render-ready, it writes the top billable-minute
workflows that have no source-backed Tier-2 finding to
.ci-speedup-gaps/bill-workflows/. The ranking is the same cost-spine axis
used for bill-pole deepening: summed billable-equivalent minutes/month, with
top job rows and repo/commit/window provenance copied into bill-gap.json.
This does not render a report section and does not promote a detector. The
namespace is deliberately separate from log-backed phase-4b captures so
draft_detector.py ignores it until a later human-reviewed detector/test PR is
grounded in real logs or equivalent deterministic evidence.
PR20 adds the maintainer-local producer for those paired artifacts:
maintainers/ci-speedup/scripts/produce_bill_pole_panel_artifacts.py. It scans
each local panel checkout once, then runs collect_runs.py twice from the same
scan output with the shared pinned window: adaptive (--max-runs 20 --shallow-runs 10) and forced full-depth (--max-runs 20 --shallow-runs 20).
It verifies the checkout's GitHub origin/clean git root and writes a manifest
after a complete producer run; panel measurement requires that manifest and
checks the exact adaptive/full paths it declares, so a smoke subset or stale
retry cannot masquerade as a full-panel pass. Generated findings live under the
gitignored .ci-speedup-bill-pole/; the script is an input producer for the
PR19 gate, not a collection-behavior change.
Run-list triage + relative recovery (skip the job fetch for workflows that can't hold the pole). Before the
shallow pass spends a GET /runs/{id}/jobs on a workflow, its run-LIST (1 call/workflow,
already fetched) gives each run's wall-time for free (run_started_at|created_at →
updated_at). _TRIAGE_WALLCLOCK_FLOOR_S (90s) is a coarse, fetch-cheap pre-filter: a
workflow whose slowest sampled run finishes under it is provisionally triaged (its per-run
job fetch skipped). It can still be a concurrent sibling on the PR, so the triage stub
carries concurrent_wall_p50 (its run-list wall) and the cross-workflow floor
(wall_clock._concurrent_workflows) keeps counting it — the floor is preserved, not
dropped, so a saving can't overstate by up to ~90s when the binding sibling is itself sub-90s.
The 90s constant is absolute, but the merge pole is relative — so on a seconds-scale repo
(measured gate at/under the floor) a sub-90s workflow CAN own a near-gate secondary pole. Once the
shallow pass has measured the gate, a RECOVERY pass (before _deepen_candidates) job-fetches
any triaged workflow whose run-list wall reaches _TRIAGE_RECOVER_GATE_FRAC (0.5) of the gate's
long pole, so it is ranked + drilled like any pole rather than dismissed — triage relative to the
measured pole rank, not the absolute constant. Disclosed via
recovered_fast_workflows/recovered_fast_count. A minutes-scale gate sits far above the floor,
so nothing under it qualifies and triage is fully preserved there (zero behavior change, zero extra
gh). If a recovery fetch yields no job timing (total fetch failure), the workflow stays triaged
with its stub floor intact, surfaced via jobs_fetch_failures, rather than mislabeled recovered
with empty data.
Headline-crown recovery. The relative-recovery pass references the gate long pole, which
is the rare-giant's p50 when a heavy pole is present-but-minority. So on a repo where required
checks are unreadable AND every heavier pole is minority-present, the crown (critical_path_check
= the slowest typical check) can fall to a sub-floor lint that is still triaged — its wall
never reaches 0.5 of the giant gate. The report would then HEADLINE a workflow it also discloses
as triaged-fast, and that headline pole dead-ends ("no captured log" / "NO CATALOG PATTERN
MATCHED"). So after the spine is ranked, _crown_recovery_wf resolves the crown to its workflow
(via the scanned job graph, since the timing mapper misses a triaged workflow) and, if that
workflow is triaged with retained runs, job-fetches it once — un-triaging the headline so it is
drillable and dropping it out of triaged_fast_workflows/into recovered_fast_workflows. Bounded
and single-shot (one workflow, no re-rank: a fast-lint crown's job p50 ~= its check-run p50, so
its rank is unchanged — only its per-step drill was missing). No-op on a healthy repo (the crown
is never triaged) and on a fileless/external crown (no workflow to recover — a legitimate no-drill
headline).
Conservative on purpose: gated on the max wall-time over the sampled window (a workflow
that ran long anywhere in that window is fetched), the floor is far below any plausible pole,
and a workflow with any sampled run of unknown duration (missing/unparseable timestamps)
is fetched rather than triaged — so a fast run can't mask an unmeasured long one — so the
gate/pole/floor/bimodal stay exact (validated by measured before/after: on a
dogfood repo this cut total gh calls ~23%, 408 → 314, with an unchanged pole). Only the
skipped workflow's
job-level hygiene/queue degrade to run-list-only, disclosed in
data_sources.triaged_fast_workflows (+ a provenance footnote) — never silent. PR-sha pool,
events, and volume still come from the run metadata, so the sample population is unaffected.
A triaged workflow's check-run still rides along on the sampled PRs, so its check-run p50 can
land in pr_checks_tuple and even rank into the structural top-N — but it has no sampled job
to decompose, so it is excluded from the drilled pr_critical_path.poles
(_structural_pole_candidates resolves each candidate's workflow with the same _pole_mapping
the decomposer uses and drops it when that workflow is triaged). Without this it would render as
a bare long pole (no dominant_step/steps → "no captured log" + "NO CATALOG PATTERN
MATCHED"), contradicting the very triage coverage note above. The dropped check stays on the
checks spine (its honest fast-check disclosure); it is simply never a drilled pole. The class
guard verify_report.check_speed_poles_complete re-derives this from the findings JSON in two
places: the poles-keyed _triaged_pole_offenders (a triaged workflow_file appearing among the
poles) AND the crown-keyed _crown_triaged_offender (the critical_path_check mapping — via its
checks[*].workflow_file — to a triaged workflow), so both a bare drilled pole and a triaged
HEADLINE (which never enters poles) are caught on every report.
Pole-log fetches are concurrent. The cross-run magnitude check (_magnitude_sample)
validates the drilled run's load-bearing number against a few other runs' logs; those logs
(often multi-MB) are fetched per batch concurrently (bounded by the batch; the adaptive
probe→escalate decision stays sequential between batches), so the downloads overlap instead
of running back-to-back.
Why per-run REST job timing, not a GraphQL CheckRun projection. A recurring
instinct is to collapse the per-run GET /runs/{id}/jobs cost by reading check
state from GraphQL instead. We don't: the CheckRun projection is lossy for
timing — it doesn't expose the per-run job fields the drill and queue-time
analysis depend on (runner label, queued/started timestamps, per-step timing), and
its check-run span folds in queue + re-run boundaries, re-introducing the
queue-inflated-duration bug the REST path was written to avoid. The per-run REST
call is the price of an accurate, queue-clean p50 and a step-level drill. The
adaptive two-pass above is the sanctioned way to cut that cost, because it was
proven not to move the answer. The durable rule: any change that trades gh calls
for an approximation — a smaller sample depth, a cheaper endpoint, a coarser
projection — must be justified by measured convergence against full-depth ground
truth (the Monte-Carlo subsampling / forced-full-pass diff above), never by the
code's general reasoning that "it should be enough." Reproduce the
gate/drill/floor/bimodal signal first, then adopt the cheaper path.
2.1a Config-era boundary — a sample never blends two CI configurations (issue #66)
The sample window is a time window, but a workflow file can CHANGE inside it. When it does, the runs before the change and the runs after it measure two different CI configurations, and blending them is unsound: it produces a stale headline, a drill that mixes the old step layout with the new (the fabricated "the guard runs twice, once whole and once sharded" cross-era synthesis — no PR ran both), and a recoverable ceiling drilled from the retired config that exceeds the typical wait under the current one. This is the universal second-run journey (audit → the user's own fix lands → re-audit), and the skill's own description advertises "re-auditing after upstream CI changes," so it must handle a mid-window change first-class.
The boundary. For each workflow file with ≥2 sampled runs, _workflow_change_boundary
issues ONE commits?path=<wf>&per_page=2 REST call — the two most-recent commits that
touched the file, returned as (last, prev) — pinned &until=<created_before> so an
edit LATER than the audited window is never mistaken for the boundary (a regen samples
the same window). last is the config-era boundary; prev is the boundary of the era
immediately before it, used only to keep the disclosed-pre fallback single-era when a
workflow changed TWICE in the window (see below). API cost: one call per
straddle-eligible workflow (K calls, K = workflows with ≥2 runs) — per_page=2 is still
a SINGLE call, so the budget is unchanged. This is the frugal choice, sanctioned by the
adaptive-sampling ethos above: the runs API exposes no workflow-content hash, so the
only content-diff alternative is one /contents/ fetch per run (N ≫ K) — the commit
lookup is O(1) per workflow and strictly cheaper. allow_missing=True; any failure/empty
history → (None, None) → a byte-identical no-op.
The partition (_partition_config_era), applied to each workflow's spine + drill
run population (runs_for_spine, threaded into the PR-sha pool, the shallow/deepen job
plan, and the triage-recovery stash — NOT sampled_runs_by_wf, which the runner-minute
/ relative-recovery consumers keep at full sample; the partition is scoped to the PR
spine/drill, L2):
The straddle state space is total — every straddle resolves to exactly one of three rules, each with stamps that survive and a disclosure that matches what the report enumerates (issue #74). The blended-while-claiming-purity state is unreachable, not caught after the fact:
| sample vs. boundary | rule | kept era | rendering |
|---|---|---|---|
| all runs one side (or no boundary) | — | full sample | byte-identical no-op (a workflow that did not change is untouched) |
straddle, post-change sample ≥ _RARE_PRESENCE_MIN_PR |
post_only |
post | measures the CURRENT config on a narrowed window; a light "narrowed to the current configuration" note |
| straddle, post-change sample too thin, pre era has a gate-bearing check | disclosed_pre |
pre | measures the PREVIOUS config; a PROMINENT ⚠️ disclosure near the headline ("ci.yml changed N ago — this audit measures the previous configuration; re-run once history accumulates") |
| straddle, post-change sample too thin, pre era has no gate-bearing check in the sample (the gate PRs are all post-change) | post_only_thin |
post | measures the CURRENT config from the thin post sample; a PROMINENT ⚠️ disclosure ("ci.yml changed N ago — measures ONLY the new configuration on a thin sample; only N runs … treat these numbers as provisional; re-run as history accumulates") |
In every straddle branch the change's OWN before/after never blend — one side is always dropped whole — so the fabricated retired-vs-current synthesis (the "runs twice, once whole and once sharded") is structurally impossible, not merely caught after the fact.
The disclosed_pre → post_only_thin flip — the timing spine flips WITH the rule (issue #74,
direction (a)). _partition_config_era chooses disclosed_pre from RUN counts alone, but only the
PR gate sample (per_sha_checks, available after _select_repr_shas) reveals whether the kept PRE
era actually produced a gate-bearing check IN THE SAMPLE. When it did not — the live #74 shape, where
the sole gate-bearing sampled PRs were all post-change so the pre era is check-empty — "measure the
old config" is UNAVAILABLE (there is nothing pre-era to show). The original never-empties fallback
skipped the enumeration cut whole: it left the blend intact AND cleared the stamps the enumeration
guard re-derives from, so the report rendered test @ 8m58s + guard shard 3/4 under a "reflect the
configuration BEFORE it" disclosure while the guard went blind.
_era_resolve_thin_flip closes it BEFORE the spine is consumed. The decision is a pure function of
the sample; the crucial part is that when it fires it does not merely relabel the rule — it
re-drills the workflow's whole spine from the POST runs (the injected _era_redrill re-fetches
that one workflow's post-era job listings and rebuilds crit_by_wf/jobs_per_run_by_wf), so
pr_check_p50, the poles, the representative-run links, and the makespans all derive from the new
configuration — never a pre-era run under a "measures the new configuration" claim (the defect a
naive enumeration-only flip left behind: crit_by_wf still at the retired 538s). This runs in the
pipeline order (partition → shallow/deepen drill off runs_for_spine → _select_repr_shas →
flip + re-drill → pr_check_p50 → _era_scope_enumeration), so every downstream consumer inherits
the post-era spine. Only then are rule/kept_era rewritten to post_only_thin/post; the
disclosure becomes the provisional _CONFIG_ERA_THIN_MARKER line rather than the pre-only caveat; and
_era_scope_enumeration binds the already-resolved fact (keeping the post checks, other_era_checks
empty). The re-drilled post sample is below _RARE_PRESENCE_MIN_PR, so the presence-dependent
machinery (minority demotion, populations, the presence-causal headline forms) stays inert on the POST
timings and the thin disclosure carries the reduced confidence — no sub-floor pretend-confident
output. A timing-provenance guard leg (_stamp_pole_repr_run_era +
check_era_disclosure_matches_enumeration) stamps each pole with its earliest drilled-run timestamp
and FAILs a post-claiming disclosure that carries a pole drilled from a pre-boundary run.
Era classification is content-keyed (issue #77). A run is placed pre/post by the workflow-file
CONTENT its head_sha actually executed, not by its created_at. A pull_request run runs the
workflow from the PR's OWN head, so the two PRs that CARRY a CI fix run the NEW config from their own
heads — often minutes BEFORE the fix merges — and a stale branch merged AFTER the boundary runs the
OLD config; timestamp classification misreads both, and on the live internal-dev-repo shape it
rendered the fix-PRs' new-config makespan under a pre-only disclosure and SUPPRESSED the thin-flip
(the fix-PRs looked like kept-side pre gate PRs). The mechanism: _workflow_change_boundary returns
the two boundary COMMIT SHAs; _resolve_content_eras fetches the workflow BLOB sha on each side
(POST at the boundary commit, PRE at its predecessor) and, for each UNIQUE sampled head_sha, the blob
that head carried — matching POST→post, PRE→pre, NEITHER→timestamp fallback. _partition_config_era,
_era_pr_side (thin-flip / enumeration / re-drill), and the pole stamps all consult this content era
first and fall back to created_at when it does not resolve. API cost — the worst-case bound:
≤2 blob calls for the boundary + ≤1 per unique sampled head_sha, and ONLY for a workflow whose sample
TIMESTAMP-straddles the boundary (the cheap _timestamp_straddles gate). A workflow that did not
change, or whose whole sample sits one side, fetches zero blobs — byte-identical to a
non-straddling repo. When the POST blob fails to resolve, content classification is skipped whole
(every run falls to timestamp) without paying the per-head fetches. Classification bookkeeping
(content vs timestamp sampled-run counts, boundary-blob resolution) is stamped on the era fact as
classification, and each straddling pole gains repr_run_era / repr_run_era_basis /
repr_run_head_sha alongside repr_run_created_at, which the verify guard's timing-provenance leg
reads basis-aware (a content-post pole with a pre-boundary timestamp is no longer a false FAIL, and
the pole's era stamp is cross-checked against the fact-level content map).
Multi-boundary: if a workflow changed TWICE inside the window, the runs before last
themselves span two older eras; the disclosed_pre fallback narrows the kept set to the
single [prev, last) era (using the prev boundary from the same one call), so the pre-side
is a single era too — multi_change/kept_count record when that narrowing fires. (In the
common single-change case prev predates the whole window, so the narrow keeps every pre
run — byte-identical to the un-narrowed path.) In the rare ≥3-change corner where [prev, last) holds no sampled run — only the two most-recent boundaries are fetched, not deeper
history — the kept set falls back to a disclosed wider pre set that may span older eras
(best-effort, always disclosed, never silent). The sufficiency floor reuses
_RARE_PRESENCE_MIN_PR (the same sample-size floor below which a check's frequency is
already treated as noise). Per-workflow era facts (boundary, prev boundary, kept era,
rule, multi-change flag, pre/post/kept counts) are stamped in
pr_critical_path.config_eras; verify_report.check_config_era_boundary re-derives that
no drilled pole (nor the spine) bound to a kept_era == "pre" workflow ships without the
disclosure. Bill-scope (L2): the partition is scoped to the PR spine/drill; the
runner-minute / cost-spine figures deliberately keep the FULL sample (they size total
compute, not the critical path), so a straddle co-renders a caveat that those figures
still include the earlier configuration and a duration- or structure-changing edit blends
both layouts there. A recoverable-within-wait coherence guard
(check_recoverable_within_wait, a bounds-family sibling of §7's #24/#25/#30) further
requires any rendered "up to ~X" ceiling that exceeds the typical merge wait to
co-render the slow-mode/worst-case reconciliation (the excess is the pole's conditional
figure on the PRs where it IS the pole, not the median).
Runner-label-only eras are disclosure-only. A gating job can move between runners without a
workflow-file change (for example, through a repository variable or runner-group update). When the
same sampled gate has exactly two non-empty runner labels with one chronological transition, the
collector appends a rule: runner_label_change fact with kept_era: all, disclosure_only: true,
the job name, and both labels' run counts. The renderer names that split and the verifier requires
both populations in the note. These facts never enter the content-era enumeration or per-PR spine
doors: both runner populations remain measured, and no check, run, pole, or timing-spread sample is
partitioned at the synthetic label boundary.
Enumeration binding — one report describes one configuration (issue #69). The partition
above scopes the SPINE TIMING (runs_for_spine) to one era, but the enumerated CHECK SET —
the PR-gate check-run sample that seeds pole candidacy, the populations, and the Level-1 chart
— is drawn from the sampled PR heads, which can include post-boundary PRs carrying the NEW
config's checks. Left unbound, a disclosed_pre report renders the new config's jobs (guard shard 1/4..4/4, observed on a couple of post-change PRs) as poles and Level-1 bars BESIDE the
pre-era test timing — a configuration that never ran, and the seed of the same fabricated
cross-era redundancy #66 kills, through a different path. _era_scope_enumeration closes it:
for each straddle it splits the sampled PRs at the boundary (_era_pr_side, mirroring the
run-partition's kept-side selection incl. the multi-boundary [prev, last) window), attributes
each spine check to its workflow (the same timing-then-scanned-graph mappers the collector uses everywhere else to bind a check to its job),
and drops any check bound to the straddling workflow that was observed ONLY on the dropped
side. Those dropped checks are the other configuration's adds/removes: stamped on the fact as
other_era_checks (with kept_checks alongside) and NAMED in the era note ("the new
configuration adds checks not measured here: guard shard 1/4…guard shard 4/4" for
disclosed_pre; the converse "the previous configuration ran checks not measured here" for
post_only). Because pole candidacy, presence, populations, and the chart all re-derive from
pr_check_p50, dropping there binds every enumeration surface at once. _era_scope_enumeration runs
on ALREADY-RESOLVED facts — the disclosed_pre → post_only_thin flip and its spine re-drill happen
earlier, in _era_resolve_thin_flip (above), so this binder never sees the empty-enumeration case a
disclosed_pre straddle used to hit (it has been re-drilled to post first). The only residual
empty-spine path — a post_only straddle whose CURRENT era is check-empty while the retired one
was not — leaves the enumeration intact but KEEPS its stamps (never the old skip-whole clear, the #74
blindness), so the guard FAILs loudly on the leak rather than skipping blind. It is a pure no-op when
nothing straddles (L2 byte-identity).
verify_report.check_era_enumeration_bound re-derives the bind from the stamped other_era_checks:
no check the report enumerates (from checks / poles / populations) may be a member of any
era's other_era_checks — a check absent from the kept era's runs. A second, independent guard
(check_era_disclosure_matches_enumeration, issue #74) keys on the RENDERED disclosure marker: a
pre-only "measures the previous configuration" caveat must not co-exist with an all-post / hollow
(empty-kept_checks) measurement — re-derivable now that the flip makes stamps always survive. Both
run on every config_eras fact, the same stamp the renderer's era banner reads; a fact predating the
enumeration stamps contributes no other-era names to the set-dependent legs (said in the detail),
and only findings with no config_eras stamp skip.
Per-PR spine door — the makespan/chain layer is bound to the kept era too (issue #80). The
enumeration bind above scopes the CHECK SET (pr_check_p50) by NAME, but a check name survives a
config change, so the layer built from the raw per-PR SAMPLE — the chain_facts (→ chain_summary
→ makespan_p50_s, the "a typical PR waits N" headline and the §7 #24 physical-bound cap), the
populations, and the presence denominators — still blended a dropped-era PR's latest-attempt
interval for a kept-NAMED check (test is still test after it gets 3× faster). On the live
internal-dev-repo run two dropped-era PRs' 166s makespan crowned the headline under a disclosure
whose pre-era test p50 was 538s — physically impossible. _era_scope_pr_spine_sample closes it:
under a straddle it scopes the per-PR sample to the kept side, surgically per straddling
workflow — a PR on workflow W's DROPPED side (_era_pr_side, content-first via head_sha) loses
only its W-attributed checks (check_wf_of); era-neutral checks from non-straddling siblings
survive, and a row left with no gate-bearing check drops whole. The scoped sample feeds
_rank_spine_present_first (presence), _segment_pr_populations, and _stamp_chain_facts;
sampled_pr_count becomes the kept-side count and era_dropped_pr_count stamps the drop for the
sampling caveat. It runs AFTER _era_resolve_thin_flip (so an emptied kept side has already flipped
to post_only_thin and its post PRs are kept here — never an empty-spine render under a pre claim)
and deliberately does NOT touch the flip's decision input, which still sees the full sample. Pure
no-op when nothing straddles. verify_report.check_era_chain_spine_bound_to_kept_era re-derives the
bind offline in three legs: chain_summary.n ≤ sampled_pr_count; no chain fact for a
content-classified dropped head carries a kept-era chain member; and a pre-claiming makespan may
not sit below a kept gate present on EVERY kept sampled PR (the unanimous-presence restriction keeps
it false-positive-free; a non-unanimous kept gate is left to the first two legs + the engine door).
2.2 Fetch orchestration (one pool, bounded prefetch waves, a token-wide governor)
§2.1 decides which gh calls to make. This decides when they are issued — and nothing else. Every call in the pass is an idempotent, latency-bound GET, so the whole family is safe to overlap; the constraint is that the results must land in the same order the serial code saw, and the call multiset must not change.
One shared pool. _fetch_pool() is a single process-wide ThreadPoolExecutor
at _FETCH_CONCURRENCY (8). Every concurrent site maps over it, so the in-flight
ceiling for the whole run is exactly the width — not (number of live pools × width),
which is what a pool-per-call-site gives you.
Prefetch waves (GhClient.prefetch_json / prefetch_text). The parallelism seam.
A caller about to issue a known set of endpoints hands them over first; they are
fetched through the shared pool and parked in a buffer, and each original
json()/text() call site then finds its response already there. The consuming code
is untouched — same iteration order, same results — which is what makes the
"identical output" claim checkable rather than hopeful. Two rules keep the call
multiset invariant:
- Prefetch only what will certainly be requested — never be MORE EAGER than the call
site. A parked response nobody consumes is a gh call the serial path never made.
drain_prefetch()counts any leftovers at the end ofcollect()(parked JSON, parked logs, and the tail of the bounded log window that was never reached), logs them loudly, and writes the count intodata_sources.prefetch_unconsumed. That count is then both disclosed (blocking_pathrenders a prefetch-drift line when it is non-zero) and checked (verify_report.py'scheck_prefetch_plan_consumedfails any freshly collected report whose count is non-zero) — so a drifted plan leaves a mark a human and the gate both see, not a lost scrollback. (The guard earns its keep: it caught 31 real unconsumed prefetches while this pass was built.) NB the committed worked-example reports predate this key and carry noprefetch_unconsumedat all; the key is a guarantee about future collected reports, and the serial-vs-parallel equivalence test — not the report diffs — is what proves the parallel path. This is why the rerun-attempt plan prefetches thefilter=allleg only and never thefilter=latestleg: that leg's call site is the one actively changing, and a plan that assumes it would keep paying for N calls per workflow that nobody consumes. - Consumption is POP-ONCE, not a cache. A second request for the same endpoint
re-issues live, exactly as before. So a call site that legitimately fetches an
endpoint twice still costs two calls, and
gh_query_countis unchanged. Responses queue FIFO per key and a wave never skips an already-parked endpoint — skipping is how a pop-once buffer quietly turns into a cross-phase cache, serving a later call site a value fetched in an earlier phase. - The buffer key is
(endpoint, allow_missing), not the endpoint alone.allow_missingis the accounting rule the fetch was made under (does a failure count towarderrors, and so toward the partial-coverage banner?), and it is applied on the pool thread at prefetch time. Keyed by endpoint alone, a value fetched under the permissive rule could be served to a call site passing the strict one — the failure would already have gone uncounted and the consumer would just seeNone. A mismatched consumer therefore MISSES the buffer, fetches live under its own rule, and the disagreement is logged at WARNING.
The plans and the call sites share their endpoint builders (_volume_endpoint,
_run_list_endpoint, _status_count_endpoint, _run_jobs_endpoint,
_job_log_endpoint) and their guard predicates (_tier2_scope_event,
_opt57_timeout_job_specs, _on_has_event, _recovers) precisely so the two cannot
disagree about which URL, or whether a call happens at all. A predicate re-stated inline
next to the plan that computed it is a drift bug waiting to happen; there is exactly one
copy of each.
The waves, in order: the run-list family for every workflow → the whole shallow job
sample flattened across workflows → one wave per deepen round → the triage-recovery
job fetches → the detector loop's run-list family → the rerun-attempt filter=all
listings → the workflow contents reads, the cache-evidence logs, the cache push-probe
logs, the OPT79 net-negative-cache probe logs (up to 24 job logs in one prefetch
wave, issued without --with-logs), and the pole drill logs.
Peak memory: the JSON waves are flat, the LOG waves are a bounded window. The JSON
responses (run lists, job listings) are retained by the consumer for the rest of the pass
anyway — jobs_per_run_by_wf holds the entire sample — so buffering a whole wave of them
adds no new peak. Job logs are the opposite: multi-MB each (a big test job's log runs to
tens of MB), and the serial path they replaced held exactly one at a time — fetch, parse,
discard. A flat pool.map over the whole log plan would materialise every planned log
before the first is consumed and hold them all live until popped — peak memory O(plan),
which on a large repo (every cache-family finding × cap runs, plus the pole drills) is
hundreds of MB to GB. So the three prefetch_text plans drain through a window
(_TEXT_WINDOW = 16): at most 16 logs are parked at once, topped back up as the call site
consumes them, so peak log memory is O(window), independent of plan length, while each
refill is still a full-width wave. The one directly-pooled log map (_magnitude_sample)
is chunked to the same window for the same reason. A test asserts the parked log set never
exceeds the window across a plan far longer than it.
What must stay serial. The deepen rounds (§2.1). Each round's ranking is
computed from the previous round's corrected p50s — that feedback IS the convergence
guarantee — so the rounds never overlap. Workflows within one round do fan out.
(test_gh_concurrency.py pins this: fusing the rounds changes the answer.)
_select_repr_shas also keeps its bounded, newest-first, stop-early batching: it is
capped to the PRs still needed so it never over-fetches.
Width 8 bounds concurrency; the governor bounds the rate (they are different things). GitHub's primary limit (5,000 req/hr) is indifferent to concurrency — the pass issues the same calls either way. The binding rule is the secondary limit. Its clauses, verbatim from "Rate limits for the REST API → Secondary rate limits":
- "No more than 100 concurrent requests are allowed. This limit is shared across the REST API and GraphQL API." — width 8 is comfortably inside this.
- "No more than 900 points per minute are allowed for REST API endpoints" (a GET = 1 point). This is an AGGREGATE budget for the token across the whole REST API, not a per-route allowance. (The giveaway is the sibling clause: GraphQL scores 2,000/min against "the GraphQL API endpoint" — a single endpoint. There is no per-route allowance anywhere in GitHub's rate-limit docs; the only other secondary limits are the 100-concurrent one above and "no more than 80 content-generating requests per minute", both per-token.) An earlier version of this section claimed "900 points per minute per REST route" — that was a misreading, and it is corrected here and in the governor below.
Width 8 is deliberately not a new number: it is exactly the width the three pre-existing pooled call sites already used, so peak concurrency is unchanged by the re-orchestration. But peak concurrency is not the whole story: finishing the same ~700 calls in roughly a third of the wall time triples the sustained request rate, and the binding secondary limit is a rate. So the honest claim is "peak concurrency unchanged at 8; sustained request rate up ~3×, paced by the governor" — not "zero new rate-limit exposure". The governor below is what makes the higher sustained rate safe; the width is what bounds the blast radius when it isn't. The value here is parallelising the right group (the run-list family, hoisted out of the per-workflow loops), not driving a wider pool.
⚠️ Raising
_FETCH_CONCURRENCYabove 8 is unblocked but not free. Rate-limit detection, retry/backoff,Retry-Afterhandling and the circuit breaker all live in_invoke(§2.0) — so a secondary-limit 403 is retried, shared across the pool, and counted rather than silently lost. What a wider pool still costs is exposure: more of the sample is in flight per incident, and an exhausted retry budget is still lost data. The bump is a one-line follow-up that must ship with a fresh before/after measurement; a test pins the ceiling at 8 until then.
The token-wide governor, and why capacity ≠ budget. _RestRateLimiter is a single
token bucket for the whole REST API on the token — because that is the thing GitHub scores.
It refills at _REST_RATE_PER_MIN (600/min = 67% of the documented 900) and holds at most
_REST_BURST (100) tokens. The remaining third is real headroom, not timidity: the budget
is the token's (the user's gh is usually also driving their shell and editor), GitHub
says some endpoints cost more than 1 point and does not publish which, and a 403 here is
lost data once the retry budget is exhausted — riding at 94% of the ceiling is not a safety
mechanism. It costs nothing measurable because the pass is latency-bound at width 8 (see
the measured numbers in the CHANGELOG): the cap binds only on a repo whose responses come
back fast enough for the pool to sustain >600/min, which is exactly where it should bind.
_route_key still exists, but only as a DEBUG label ("which family is the governor pausing
on?") — it does not select a bucket, because there is only one. It now collapses every
parameter in the path, not just numeric ids: owner/repo, SHA-shaped hex
(commits/{sha}/check-runs is one route, not one per commit) and everything under
/contents/ (one route, not one per workflow file). The old numeric-only version left
those two families keyed per-value, which — under the old per-route bucket — meant they got
a fresh full burst on every call, i.e. no pacing at all; the global bucket fixes that
regardless, but the keying was simply wrong and narrower keying is not "more
conservative" (more buckets = more admissions).
The two knobs are separate on purpose. A token bucket admits, in the worst 60-second
window, capacity + rate × 60 requests — so a bucket whose capacity is the per-minute
budget admits twice the budget from a cold start and only begins pacing in the second
minute. That is precisely the wrong shape: GitHub's secondary limit is burst-sensitive, and
a large repo's job-listing wave (40 workflows × 20 runs) IS a cold-start burst. With the
burst decoupled:
worst case in any 60s = _REST_BURST + _REST_RATE_PER_MIN = 100 + 600 = 700 < 900 ✅A test states exactly that property, and — the point the old single-route test missed — offers the infinite load across many routes, because the budget is shared: N routes × one bucket is not a partition of anything GitHub scores. On a small repo the burst covers the whole pass and the governor never blocks; on a large one it — not the width — sets the pace.
A failed fetch is never silent. allow_missing=True means "a 404 here is
expected" (branch protection / rulesets on a repo you don't own; a workflow file that no
longer exists) — it does not mean "any failure here is expected". Every non-404
non-zero exit is counted in errors even at an allow_missing call site, and logged at
WARNING (visible at the default INFO level). This matters most exactly where the
concurrency work put the most pressure: the workflow contents reads are all
allow_missing=True and are now one late wave. A swallowed 403 there drops a workflow
from _wf_docs, and every consumer (on: block, matrix/shard recognition, OPT57 timeout
specs, OPT65 event scope, the schedule block) then evaluates against an empty dict — a
report that renders clean and complete with a workflow's entire file-level signal missing.
The classification lives in one function (_classify_gh_failure — 403-vs-429-vs-5xx by
status line + header evidence, rate-limit keywords as a fallback) and every live call
funnels through one choke point (GhClient._invoke), which applies the retry/backoff, the
shared rate-limit block (_blocked_until), and the global give-up breaker (gave_up).
allow_missing only ever exempts a genuine 404; a rate-limit / 5xx / timeout exhaustion is
counted even there. The concurrency pass runs _invoke on the shared pool's threads, so
that same choke point also carries the token-wide rate governor (_RestRateLimiter, acquired
before every live attempt) and catches an OSError/UnicodeDecodeError so one bad fetch
never discards a whole wave.
3. Data model - findings.json
scan.py emits the document; later stages add keys. The top-level shape:
| Key | Set by | Meaning |
|---|---|---|
findings |
scan + collect_runs | the finding list (see below) |
scanned_workflows |
scan | count of workflow files seen |
scan_incomplete |
scan | files that could not be read/parsed (coverage gaps); every record carries a stable machine kind as well as its human reason |
catalog_patterns_without_detector |
scan | coverage-honesty list |
scanned_at / skill_commit_sha / commit_sha / repo |
scan main | provenance |
data_sources |
collect_runs | tiers run, gh counts, sampled_runs_created_before, partial_reason, adaptive-sampling fields (shallow_runs, max_runs, shallow_capped, capped_workflows, pole_candidates, deepened_workflows, deepen_converged, full_depth_workflows, shallow_remaining_workflows — §2.1), run-list-triage fields (triaged_fast_workflows, triaged_fast_count, recovered_fast_workflows, recovered_fast_count — §2.1), cost-spine-only triage/fetch fields (cost_spine_triaged_workflows_included, counts for workflows/runs/jobs, cost_spine_job_fetch_failures, cost_deepen_candidate_workflows, cost_deepened_workflows, cost_spine_full_depth_workflows, and cost_spine_shallow_workflows), and cache_dist_probe ({cache_poles, push_logs_fetched, pr_logs_reused} — the cache-distribution provenance + the schema stamp verify_report's cache checks SKIP on when absent, §12.4) |
pr_critical_path |
collect_runs | the merge-wait spine: critical_path_check/critical_path_s (the headline pole), checks[] (name, p50_s, present_on, pole_n = PRs where this check is the actual pole, the recurrence-ranking signal, §11; workflow_file, bimodal?), check_present_n_pr (presence denominator), sampled_pr_count (the kept-side PR count post the §2.1a #80 spine door) + era_dropped_pr_count (PRs the door removed as dropped-era; stamped only on a straddle that dropped >=1), poles[] (decomposed drilled poles; a cache pole also carries cache_dist = the per-event, fork-aware miss distribution + verdict, §12.4), populations[] (per-PR check sets, M2), dropped_non_pr_checks / dropped_non_required_checks (visible exclusions), fileless_status_checks[] (name, span_s, basis — the fileless/managed status checks EXCLUDED from the crowning basis, disclosed as PR-lifetime status-gating latency; issue #12, §12.2a) + all_checks_fileless (the degenerate flag), config_eras[] (content eras carry workflow_file, boundary, kept_era, rule, pre_count, post_count, plus prev_boundary / multi_change / kept_count on disclosed_pre; disclosure-only runner transitions instead carry rule: runner_label_change, kept_era: all, job, and runner_label_counts[] and never partition the sample; empty when neither kind is detected; §2.1a), chain_facts[] (ENG-1: per sampled PR the longest needs: chain — spine-scoped members, capped per-member spans, chain_s, co_longest_n, runner_up_s, the attempt-scoped makespan_s cross-check) + chain_summary (modal chain, chain_p50_s, runner_up_p50_s, signed divergence_pct) — when the modal chain has >=2 members the renderer's headline names the CHAIN and its summed p50 instead of the slowest single check (headline_chain claim, re-derived by check_headline_chain_matches_stamp) |
per_workflow_timing |
collect_runs | {wf: {long_pole_job, long_pole_p50, floor_p50, job_p50, events}} (events = the workflow's OBSERVED trigger events, so verify_report can re-derive merge-path eligibility — §12.6a); a run-list-triaged workflow (§2.1) instead carries {long_pole_p50: 0, …, concurrent_wall_p50} (its run-list wall, read only by the cross-workflow floor) |
per_workflow_monthly_volume |
collect_runs | {wf: 30-day run count} for the final cost-spine coverage denominator: finding-backed workflows plus any detector-only probes that emitted cost-spine rows. Detector-only probes that emit no finding and no rows are sampled for evidence but omitted from this map so they do not masquerade as incomplete cost-spine coverage. |
runner_minute_spine |
collect_runs | source block for the cost-spine table: stable per-workflow/job/runner rows, raw compute minutes, billable-equivalent minutes, totals, repo visibility, and render_ready: true only when complete workflow coverage is proven; PR16 stamps sampled_workflows_in_play_with_job_data plus workflow_coverage for positive-volume workflows in play, omitted workflows, unknown-volume workflows, triaged workflows included, and source-row fetch failures |
dropped_unprovable |
collect_runs (--with-logs) |
cache findings dropped at the admission gate |
data_bundle |
collect_runs (--with-logs --data-dir) |
{logs_dir, logs:[{job, file, steps_file, mag_file, sample, selected:"nearest-p50"}]} — the blocking-path drill bundle (§12): per pole, the nearest-P50 run's log + step timeline + cross-run magnitude sample |
timings |
scan/run/report | per-phase durations + run_start_epoch |
skill_tree_dirty |
orchestrator | injected so the report can stamp <sha>-dirty |
A single finding dict carries (load-bearing fields in bold):
id(f1, …),pattern(a catalog OPT-id),pattern_class(static|data-driven|structural),severity(HIGH/MEDIUM/LOW/MANUAL),title,workflow_file,line,affected_jobs,subject(the detector-owned stable identity),evidence,fix_strategy,fix_recipe_anchor.Structural findings only (
structural: true, OPT70–OPT75 — see §11):risk(LOW/MEDIUM/HIGH),guardrail,rollout,failure_mode,decomposition({dominant_step, dominant_category, dominant_share, redundant_ratio, …}), andrequired_status(required/not-required/unknown).risk+guardrailare load-bearing:_validate_findingspulls out any structural finding missing them, so a high-leverage change can never render as if it were safe hygiene.Sizing:
wall_clock_p50_s(the ranking metric;null= qualitative,0.0= sized-as-zero),runner_min_saving,tier(1 direct lever / 2 bill-only-or-negative / 3 reliability),realization(direct/tail/none),size_note, andwall_clock_uncapped_p50_swhen a cross-workflow cap fired.Evidence/provenance:
measured_signal(required on data-driven findings),measured_evidence({summary, table:{headers,rows}, note}with links to real job logs),evidence_snippet(verbatim matched YAML, static findings),workflow_activity({runs_30d, last_run, dormant}).Cross-workflow:
concurrent_workflows([{workflow, long_pole_p50_s}]) andworkflow_long_pole_p50_s.Routing:
advisory(excluded from the report, kept in JSON),needs_log_confirmation.
The static scanner also exports INSTANCE_GRANULAR, the pattern ids whose
detectors have cardinality tests proving that each physical occurrence produces
its own finding. Differential consumers such as ci-speedup-guard may enforce
only members of this set. Scanner readers do not follow symlinks outside the
scan root; each such path is disclosed as scan_incomplete.kind = symlink-out-of-tree.
A pattern absent from the _SIZING table is rendered qualitatively (both axes
null) - the scanner never invents a number.
4. The two metric axes
Every finding is sized on two different numbers measuring two different things:
- Δ wall-clock - the per-run developer wait removed from the critical path. A finding can have a large runner-min saving and zero wall-clock (its job sits below the cluster floor and never gated the run).
- Δ runner-min/mo - the cloud-bill saving. Unlike wall-clock it is additive across disjoint step-seconds. For promoted Tier-2 rows the section sums raw credited minutes after the Tier-2 de-overlap pass; residual appendix rows still group by pattern and do not cross-family de-overlap.
The report RANKS on Δ wall-clock (blocking_path.py): the findings table
puts the biggest wall-clock lever first, including all non-advisory findings;
advisory findings are excluded from the report but kept in the findings JSON.
Off-path catalog hygiene (a fix that moves ~0 wall-clock) now has two routes:
measured findings with a tier2_neutrality certificate promote into
"Runner-minute reductions (wall-clock-neutral)" as source-backed candidates,
while modeled, uncertified, advisory, or residual same-pattern members stay in
"Also noticed". Candidate admission and appendix exclusion are per finding
(_is_tier2_finding); visible R-rows additionally require matching
runner_minute_spine source rows. The appendix groups by pattern, so an OPT-id
can straddle both sections. A finding whose fix is wall-clock-negative
(_increases_wall_clock: the explicit wall_clock_negative flag, set by
_size_finding) is disclosed as such in the residual appendix — a bill saving
that adds developer wait, not a speed win.
4.1 Runner-minute reductions promotion gate
blocking_path.py renders a first-class Tier-2 section after Pre-start wait and
before Also noticed. A finding becomes a Tier-2 candidate only when
sizing_basis == "measured" and tier2_neutrality is stamped; that predicate is
the routing source for candidate admission. The visible section uses the
stricter source-backed set: a candidate renders only when
matching render-ready runner_minute_spine rows exist for the finding's
workflow, relevant job identities, and non-conflicting row dimensions when
the cost spine stamps that detector-specific population. OPT64 must bind to its
stamped rerun_dominant_job and all-status/prior source rows; explicit
finding filters cannot weaken those required dimensions. Non-OPT64 savings
must bind to success/latest/all-status rows, so prior-attempt rows are not
allowed to back ordinary latest-attempt savings. Job matching is exact-first:
matrix-base source rows are used only when no exact source row exists for the
stamped job name. Latest-attempt
schedule/failed-run detectors currently bind to the collector's
all-events/success/latest source envelope while their detector-specific stamps
prove the event or failed-run subset. The matched source rows must also cover
the credited raw minutes; OPT65 rounding waste is the billable-minute exception
because it credits per-run billing-rounding delta, not raw compute time.
The Contents count, section lead, bottom-line Tier-2 sentence, and summary.py
handoff line all use that source-backed set. Measured/certified candidates that
fail source binding are suppressed from R-numbered savings cards but fall back
to Also noticed with an explicit source-backing note, so they stay visible
without being presented as source-backed.
The section ranks by billable-equivalent runner minutes and displays raw
credited minutes. All figures are runner-minutes; the section lead ends with the
one-sentence pricing story ("multiply by your runner's per-minute rate to get
dollars"). The renderer shows at most 12 R-rows, but the lead totals cover every
source-backed Tier-2 row in findings.json; overflow rows are disclosed as not
shown and remain in findings JSON rather than moving back into Also noticed. Each
visible card's Source block line shows the current cost-spine raw minutes and
billable minutes for those source rows. The verifier also rejects duplicate
R-row marker IDs, rendered rows without per-finding Tier-2 stamps, non-positive
savings, and visible row savings whose credited raw minutes exceed the matched
source-row totals (billable totals for OPT65).
The verifier re-derives the promoted set, measured basis, certificate class and
margin where applicable, and de-overlapped raw total from
findings.json. The checks SKIP, never FAIL, on genuinely
pre-stamp findings JSON so older worked examples keep verifying until their data
is regenerated; if top-level Tier-2 stamps are present with positive
runner-minute candidates but the per-finding stamp surface is missing, they fail
closed instead of treating the artifact as pre-stamp.
The certificate's proof token picks the re-derivation, and every token is
enumerated in check_tier2_neutrality_derived (an unknown one fails):
below_cluster_floor (the job's p50 is strictly below the workflow's cluster
floor; OPT65, OPT77 — historically, see below — and OPT79, each re-derived by
its own arm), below_long_pole (OPT79 only: the job is at or above the floor
but strictly shorter than the workflow's long pole and, on a pull-request
workflow, in no needs: chain with it, margin long_pole_p50 − job_p50 from per_workflow_timing, the
chain re-derived from workflow_job_graph when the run recorded one),
post_completion_waste (OPT35/46/57/64), checkout_tail_excess (OPT80 only)
and non_pr_event (OPT36). The "affected job is also rendered as a Long pole"
proxy is waived for exactly two shapes whose arm re-derives the neutrality it
stands in for: checkout_tail_excess and OPT77's whole-workflow group. It is
NOT waived for below_long_pole: a job in a needs: chain with the long pole
adds to the merge wait (the chain's sum), which the single-job margin cannot
see, and a rendered pole is how such a job shows up. The renderer's
_tier2_cert_summary names each token in the R-row's proof line.
Whole-run Tier-2 detectors can stamp tier2_sample_run_ids, the sampled
workflow-run IDs behind their credited count. _reconcile_tier2_overlap uses
those IDs to move duplicate sampled-run credit into runner_min_overlap_s
before totals are rendered, so the section lead can show both the
naive and credited-after-de-overlap minute totals. Event-subset certificates
also carry a machine-readable event stamp: OPT36 schedule burn promotes only
tier2_run_subset_events: ["schedule"], and verify_report.py re-derives that
non_pr_event proof from the persisted workflow events plus the detector's
schedule-only evidence. The persisted mirror (events_by_wf) is first built from
the main-pass SUCCESS slice, so a [push, schedule] workflow whose recent
successes are all push would omit schedule and fail that re-derivation;
_fold_observed_events unions the events the dedicated schedule probe actually
observed back into the mirror so it stays a faithful superset of every event the
pipeline saw for the workflow.
When measured OPT36 is emitted, the matching static OPT36 finding for that
workflow is retained in JSON with tier2_superseded_by but excluded from Also
noticed, so the report shows the measured upgrade once.
The same supersession path is used by measured OPT35: explicit
fail-fast: false shard matrices promote only when failed-run job timings prove
sibling shard compute after the first failed shard, and the certificate remains
post_completion_waste because the run result was already decided. OPT35 emits
one measured finding per matrix job and supersedes only the matching static
workflow/job row, leaving unrelated shard matrices in the residual appendix
until they have their own failed-run evidence.
OPT64 uses the same whole-run Tier-2 path for reruns: it compares a
run_attempt > 1 run's filter=all jobs against filter=latest, credits only
the prior-attempt job delta, and requires every credited prior attempt to share
the same unique dominant failed/timed-out exact job name that appears in the
latest attempt. It keeps those prior-attempt seconds additive with OPT36/OPT46:
a bare workflow run ID is too coarse an overlap key because OPT36/OPT46 attribute
at the latest-attempt RUN level (OPT46 credits the cancellable remainder of each
superseded run, issue #89 — not its whole compute) while OPT64 credits
earlier-attempt job seconds. If job payloads omit run_attempt, the job-id fallback is disabled on
capped 100-job pages rather than treating a truncated page as measured waste.
OPT65 is the billing-rounding exception to the normal raw runner-minute
subtractive sizing: it computes the exact per-run billable-minute delta
sum(ceil(job_seconds/60)) - ceil(sum(job_seconds)/60) for tiny matrix legs
from sampled job timestamps. It emits per matrix base only when every credited
occurrence is tiny and on the same runner, and each credited run's combined leg p50 stays
strictly below the workflow cluster floor. It stamps structured
rounding_waste evidence, a below_cluster_floor certificate, and claims no
speedup (wall_clock_p50_s=0, realization=none). It deliberately does not
stamp bare run IDs for de-overlap: the credited unit is per-job billing round-up,
not whole-run elimination. The guardrail text must keep consolidation off the
merge gate; lowering matrix parallelism or adding a serial needs: stage for an
on-spine matrix is wall-clock-negative.
OPT65 is also superseded by OPT77 on any job set both claim, in
collect() right after both have run for a workflow (_supersede_opt65_with_opt77).
The two are not disjoint: OPT65 groups on a trailing parenthetical in the OBSERVED
job name, so three ordinary jobs named lint (eslint) / (biome) / (stylelint)
form an OPT65 base AND resolve to one YAML job each. One edit must render as one
lever, so the consolidation wins; OPT65's round-up minutes for the overlap go
unreported (a deliberate under-statement, since folding a billable-round-up
quantity into OPT77's raw-compute basis would break it) and every drop is
disclosed in findings_doc["superseded_findings"].
OPT77 (repeated fixed setup across independent small jobs) is the other measured
Tier-2 consolidation lever. It splits each sampled job's step timeline into a
LEADING setup prefix and useful work, using the same _SETUP_STEP_RE classifier
the CUT hygiene detectors OPT49/OPT51 were defined against — widened in this wave
to recognise GitHub's Run <owner>/<action>@<ref> rendering of an unnamed action
step and the bare install commands (npm ci, pip install, …), without which
the largest component of a real setup prefix was invisible. The prefix's SHAPE is
read from every declared step and only its duration from the steps that measured
above zero, because GitHub's one-second step granularity otherwise turns
sub-second jitter into a change of signature.
Its neutrality model is deliberately NOT the cluster floor: the projected
consolidated job (max(setup_p50) + max(useful_p50), tasks run concurrently
inside it) is compared against the tallest job that REMAINS after the
consolidation — a floor the group's own members help define would measure the fix
against something the fix removes, and silenced the pattern on its own motivating
shape. The certificate's proof token stays below_cluster_floor (shared with
OPT65 as the dispatch key) and is historical for OPT77; verify_report.py's
_opt77_consolidation_rederived arm re-derives the saving, the margin, the
grouping (each job's own stamped prefix must equal the credited shared one) and
the eligible set the tallest-remaining job was chosen from. When the group is
every job the workflow DECLARES there is no remaining job, and the comparison is
against the gate the group sets today: max_i(job_p50_i), the slowest member's
MEASURED duration. The members run concurrently — the independence gate has
already shown none of them waits on another — so the longest of them is
literally what this workflow makes a pull request wait for. At or below that
gate the merge wait does not lengthen and the runner-minute saving is credited;
above it the group is held back and disclosed, with both numbers on the entry.
That gate used to be re-assembled from each member's step components,
max_i(min(setup_i + useful_i, job_p50_i)). The capped value is not a
duration anything took: three members each measuring 400s were told, in the
report's own prose, that they take 90s today. The component sum survives only
as a coherence check — the projection can never be below any single member's
own setup_i + useful_i, and verify_report reddens if it is, which says the
stamped medians do not come from one set of runs.
For a group that is the whole workflow, collapsing can never SHORTEN the merge
wait: each member's useful work is its measured duration minus its setup, read
off one started_at→completed_at span, so setup_i + useful_i == job_p50_i
exactly, and the collapsed job's max(setup) + max(useful) is therefore at
least max_i(job_p50_i). The saving is credited only when the two are EQUAL —
one member holding both the longest setup and the longest task, so the wait is
unchanged. Otherwise the group is held back and the report says what the
collapse would cost the merge wait. On real data the credited case is the
exception and the disclosed hold-back is the common outcome; that disclosure
is the point of the change, not the credit. The equality is also only as
stable as a one-second median: a member's p50 moving by a second flips a tie
into a hold-back, which is the safe direction — the flip can produce a
disclosed WITHHOLD, never a false FIRE.
"Whole workflow" is decided from the declared jobs, never from the sample, and
needs one declared key per member. What verify_report re-derives from
MEASUREMENT is the gate itself (max over per_workflow_timing[wf].job_p50
for the credited members, which no detector writes) and the workflow's declared
job keys (per_workflow_timing[wf].declared_job_keys, stamped by the workflow
YAML reader before any detector runs). What it takes from the finding's own
stamps, and then checks for internal coherence, is the per-member setup/useful
split and the projection built from it.
Like OPT65 it claims
no speedup (wall_clock_p50_s=0, realization=none), and it shares
_billed_job_runner and _tier2_scope_event with it.
OPT77 and OPT80 each also keep an explicit list of the candidates they measured
but could NOT decide — opt77_withheld_candidates (a formed group that exited on
anything outside _OPT77_VERDICT_GATES) and opt80_withheld_candidates (a
checkout with a measured tail that exited unresolved, outside
_OPT80_VERDICT_GATES; after the log probe, only when the unread, unfetched or
undecided tail runs could still have supplied the missing proof, named by their
commonest reason). The per-gate tallies (opt77_withheld_by_gate,
opt80_withheld_by_gate) are for maintainers; these lists reach the reader.
The held-back disclosure is ONE mechanism shared by OPT77, OPT79, OPT80, OPT81, OPT82, OPT83 and OPT84.
blocking_path._WITHHELD_ROWS is a WithheldRow per pattern — findings-doc
key, Data sources row label, counted noun, "Used for" cell, and entry_shape
("job" for OPT79 / OPT80 / OPT81 / OPT82 / OPT83, "group" for OPT77, "step" for OPT84: a job
entry that also names its one candidate step when the gate was decided on one, rendered
job (step)). The shape is registered
rather than inferred: both twins used to decide it by comparing the key against
OPT77's, so a fourth group-shaped pattern would have rendered (unnamed job)
for every candidate in the renderer AND been re-derived the same wrong way in
verify_report — agreeing, and green. verify_report reads all five fields,
including the "Used for" cell, which was previously carried on both sides and
compared to the report by neither. A job-shaped entry may also carry steps
(OPT83, when the run was found before the gate): the entry then reads
job (step + step), in both twins. _withheld_candidates_line builds the same
sentence for all three:
"N candidate … held back (<jobs>): <reason>." — the entries distinct and sorted,
workflow-qualified as <workflow.yml> / <job> when two workflows share a job
name (an OPT77 entry is the whole group), at most five then ", and K more", every
repo-controlled name escaped for the table (\ | [ ] ( ) _ < >
` * & and ~ backslash-escaped, \ first, so a name can neither
split the row, nor turn itself into a link, nor open raw HTML or a code span.
Escaped, never substituted: swapping backticks and asterisks for an apostrophe
used to render three different job names as one, in a row whose whole job is
saying which candidates were held back. & and ~ are on the list for the
same reason — left alone, A&B and A&B render as the same name. Out of scope, and unfixable by character
escaping: GFM's extended autolink turns a BARE https://... job name into a
live link with no punctuation to escape); the reason a plain-English phrase for the commonest gate,
ties alphabetical, and prefixed "most commonly," whenever more than one gate
contributed, or "equally often," when the top two gates held back the same
number — the count and the job list cover every candidate while the reason
covers only some of them, and on an exact split there is no commonest reason
for "most commonly" to name. The count is of DISTINCT candidates, the same set
the names and the "and K more" overflow come from; counted over raw rows it
disagreed with its own job list whenever two rows deduplicated into one. A pattern contributes only its key, its
row text and its gate→phrase table (registered in _WITHHELD_PHRASES_BY_KEY),
never its own row builder. The phrase tables (_OPT77_WITHHOLD_PHRASES,
_OPT79_HELD_BACK_REASONS, _OPT80_WITHHOLD_PHRASES, _OPT81_WITHHOLD_PHRASES, _OPT82_WITHHOLD_PHRASES, _OPT83_WITHHOLD_PHRASES, and _OPT84_WITHHOLD_PHRASES) map EVERY withhold gate
their collector can record. What a test actually guarantees is narrower than
"a new gate cannot ship without a phrase": it enumerates the gates it can READ
from FOUR sources in the detectors' own code — the literals handed to
_drop_group / _unresolved, the names the independence check returns, the
names _opt80_stall_in_log returns, and the reasons accumulated by .append,
.extend, += or an assignment's fallback default — and fails when one of
those has no phrase, or when a phrase has no gate that records it. The
accumulators are DERIVED from the detector (a list-valued local the function
iterates into a gate sink, plus anything built out of one) rather than listed
in the test: listed, a renamed accumulator silently took its gates out of the
scan's reach while the scan went on reporting the ones it still saw. A gate
recorded in some other shape is still invisible: assembled at runtime, passed
to _drop_group by keyword instead of positionally, or recorded in a helper
other than the four functions the scan reads. OPT81's two detectors record a
listable gate only through _no(gate, job), a shape the four-source scan does
not read, so a separate AST scan
(test_opt81_every_recordable_withhold_gate_has_a_plain_english_phrase in
tests/test_blocking_path.py) reads every _no(...) call that passes a job in
_detect_opt81_measured_runner_gap and _detect_opt81_runner_size_advisory,
drops the verdict gates, and requires the rest to equal
_OPT81_WITHHOLD_PHRASES exactly. The
backstop for that one is verify_report, which fails the report closed rather
than printing a code. verify_report.py's check_coverage_disclosed carries
its own copy of the tables (pinned equal by a coupling test) and re-derives the
whole line — count, jobs, reason — from the findings document, failing a report
that omits the row, misstates any part of it, prints a gate name instead of a
phrase, or carries a row with nothing behind it; a recorded gate with no phrase
fails closed. A group that is every job the workflow declares is no longer a
silent verdict: it fires when collapsing it does not lengthen the merge gate,
and is listed as held back
(collapsing_the_whole_workflow_would_lengthen_the_merge_gate) when it would
not. The three
files name the keys as constants a coupling test pins equal.
OPT80 (checkout stalls on the tail) is the third measured Tier-2 lever, and the
first whose admission rests on a LOG rather than on a timing alone. It measures
one step's distribution across the sample — the checkout step, identified from
the workflow YAML (directly uses: actions/checkout@*, or a local composite
whose body does, read with a transitive local-action walk mirroring OPT76's; an
unreadable or self-referential action fails closed) and never from its timing, because one-second step granularity makes a warm checkout
measure 0s and a timing-based search would drop that run and inflate the p50.
A tail is p95 >= max(3 x p50, p50 + 30s) with at least two runs at or above the
threshold; only THEN are logs fetched, for the tail runs only, newest-first,
bounded by _OPT80_LOG_PROBE_MAX — a job with no tail costs no gh call. The
finding exists only if at least two of those logs show the transfer standing
still — a gap of >= 20s between two consecutive Receiving objects: N% lines at
the SAME N below 100, with both bracketing lines quoted verbatim, and only inside
the checkout step's own time window. A log is split on newlines only, and each
record split again on the carriage returns git writes its progress animation
with, so an advancing transfer cannot masquerade as a stalled one.
A stall is progress that STOPPED, not progress that had not started. The quiet
before the first remote: line, the quiet across pack enumeration, and two
receiving lines whose percentage advanced are all a LARGE REPOSITORY (OPT28's
lever) and are withheld under tail_pause_was_advancing_or_pre_transfer; a pause
at 100% is the runner writing the pack to disk and is withheld under
tail_pause_was_after_the_transfer_completed. The large-repository shapes would
also be aborted by the low-speed timeout this pattern recommends, so reporting
them would hand the reader a fix that reds their CI.
That log gate is the whole reason the pattern is admissible where OPT49 was cut:
OPT49 read a CAUSE ("uncached") out of a DURATION. OPT80 rejects in both
directions — a tail whose logs show a smooth fetch is withheld
(tail_without_log_gap), and no log is fetched for a job with no tail, so a gap
alone can never produce a finding. Every "no proof" case is counted apart, and
every DISTINCT one across the probed runs is counted rather than only the most
common, because progress switched off (log_carries_no_progress_vocabulary) and a log
lost to retention (tail_run_log_unavailable) say nothing about the repository
while a smooth fetch does. Log text is untrusted: only the closed
progress vocabulary is read, and a quoted line carrying a credential shape drops
that run's proof rather than being masked — withholding the finding outright if
that leaves fewer than the two proofs the gate requires.
Sizing is the tail-excess model — mean - p50 of the checkout step, scaled by
the job's own observed run frequency (_effective_volume) — never the full p95
and never the whole step. wall_clock_p50_s is 0 by construction: the median
run has no stall, so capping the tail cannot move the p50 merge gate; the tail
runs' own improvement is bounded above by the longest observed pause, stamped
(tail_run_longest_pause_s, with on_critical_path), named in the rendered
evidence and deliberately left uncredited rather than rendered as a p50 saving.
When the job is the workflow's slowest AND the workflow runs on pull requests
(the detector's is_pr; unknown counts as not) AND its timing came from sampled
pull-request runs (_crit_has_developer_timing: event_scope is not the
all-events fallback, which would make every number a push timing) AND the job
is on the merge-gating path (_opt80_merge_gating_jobs: its check is required,
or a required job in the same workflow transitively needs: it; an unread,
partial or empty required set is UNKNOWN and claims no merge wait, because a
workflow's slowest job can be a non-required benchmark), a stalled run sits on that
run's merge wait, so the finding also stamps a SEPARATE tail axis,
checkout_stall.tail_axis = {sampled_runs n, slow_runs m (at or above the tail threshold), logs_read L (the fetched log_probed_job_ids, at most the probe cap), tail_runs k (log-proven), counted_runs m - (L - k), one_in_n round(n / counted_runs), typical_checkout_p50_s, tail_checkout_p50_s (median of the proven runs), tail_loss_s, on_critical_path, merge_gating}. A read log with
no stall is left out of N; an unread slow run stays in, because the probe cap
bounds cost and is not evidence of a smooth fetch (counting only k let the cap
set N: 10 slow of 30 with 4 logs read said one in 8, not one in 3). The
verifier requires every proof to be one of log_probed_job_ids, those ids to
be distinct slow runs, and their count to equal logs_fetched. And
blocking_path.py renders it as "one run in N spends about X s longer on
checkout, and that run's log shows the fetch stalling": inside the
pole's section beside its merge-wait figure (an <!-- opt80-tail:<id> -->
marked line), or in a short off-pole block when the job is not a drilled pole.
tail_loss_s is the extra checkout time on a stalled run. How much of it
reaches the merge wait depends on what else gates that run, which this line does
not model, so it is stated as the checkout loss, not a merge-wait saving, for
"one run in N". It is the median proven run's loss, not a ceiling (a proven run
can lose more), so it reads "about", never "up to"; the self-check fails a
sentence still worded "up to". It is never summed into
wall_clock_p50_s, the headline, a pole's buy line, the Tier-2 section, the
runner-minutes or the certificate. verify_report.py's
check_opt80_tail_lines re-derives N and X from per_run_checkout_s and the
proofs' own quoted lines, fails a block on a finding off the critical path or
outside the detector's own gate (_opt80_tail_axis_eligible: the workflow is in
the stamped declared_pr_workflows, its per_workflow_timing event_scope is a
developer event (pull_request, pull_request_target, merge_group; the
verifier fails a missing event_scope, while the collector treats a missing one
as developer-timed), the job is its long_pole_job, and the stamped
tail_axis.merge_gating re-derives from required_checks,
required_checks_complete and workflow_job_graph), a
block with no marked line (or a line with no block), and any "one run in N"
sentence whose numbers no block re-derives to. A SLOWEST job with no tail line says
why in one of four sentences, keyed by the stamped tail_axis_withheld_reason:
the workflow is not shown to run on pull requests (push-only or unknown
trigger), it was timed on all events with no pull-request sample, branch
protection could not be read, or the merge does not wait on the job. None of
them claims a merge wait; check_opt80_tail_withheld_reasons pairs each
sentence to its reason, re-derives the reason from the same stamped facts, and
fails any merge-wait claim on a finding with no tail axis. A job that is not the
slowest carries no merge-wait sentence at all.
The self-check also no longer wrongly fails a slowest job whose check is named
<workflow> / <job>. Its certificate carries its own proof token,
checkout_tail_excess, and verify_report.py's
_opt80_checkout_stall_rederived arm recomputes the distribution, the tail
threshold, which runs were tail runs, each pause's seconds from the quoted
lines' own timestamps, and the credited minutes — bounding every count against
the sampled occurrences. Every withhold increments
findings_doc["opt80_withheld_by_gate"], the same visible-zero discipline OPT77
carries.
OPT82 (lint builds the whole type graph) is the first hygiene pattern that
deliberately carries NO number, and its flow is a straight line through four
modules. scan.py reads the ESLint configuration of the checkout into a
top-level type_aware_lint block of the findings document: a config FACT, not a
finding, and a reader that crashes yields a block with error set rather than an
exception, so the consumer withholds instead of the scan dying. Per config the
block records whether type-aware parsing is on, off or unresolved, the
evidence lines (path:line: text), the presets seen, the type-aware rules
enumerated (from references/type-aware-lint-rules.tsv plus custom rules whose
source calls getParserServices or getTypeChecker, reached by following the
config's relative imports, bounded) and any rule setting that could not be
resolved. collect_runs._detect_opt82_type_aware_lint then runs per workflow with
no gh calls at all, from that static block and the timings already sampled:
a YAML job runs ESLint (directly, or through a package.json script it can
trace), the job costs at least _OPT82_MIN_LINT_P50_S (60s) or is the measured
long pole, type-aware parsing is on in the config that applies to the job's
directory, and at least one type-aware rule can be named. On a slow lint job,
anything the read could not see is held back, never read as on or off: an
unresolvable value (an environment variable, a ternary, a parserOptions taken
from a variable), a lint step handed to turbo / nx / make / a node script or an
action, a ${{ }} in the step, an unreadable package.json, a shareable config
package or an unfollowed local import behind an "off" read, a config walk that
stopped before the lint directory (truncated with nothing recorded at or below
it), and a nested flat config whose effect depends on an ESLint major (10 looks
configs up per file) that is 10+ or unknown. The scan block records what these
need: per config shared_configs, unresolved_imports, sets_off and root
(for the legacy upward cascade), plus truncated,
package_scripts_unreadable and eslint_versions. Local base configs the
config imports or extends by relative path are followed and read as part of it.
Config selection picks ESLint's mode first (a flat config at or above the lint
directory means every .eslintrc* is ignored unless ESLINT_USE_FLAT_CONFIG=false).
A custom rule is named only when the plugin's rule mapping and the config's
plugins binding prove which typed source it is; settings whose final value
depends on config order (on and unscoped off, an unscoped disableTypeChecked)
are unresolved, never named.
The finding it emits has wall_clock_p50_s = 0, no runner-minute saving, sizing
basis uncredited and no _SIZING entry; _RM_DOOR_OVERRIDES["OPT82"] marks it
not derivable. The reason is that nothing sampled says how much of lint is the
type graph, so a number would be a guess. What it does carry is a ceiling, the
lint step's own measured p50 (or the lint job's p50 when the step was not
separately measured; ceiling_basis records which, and every surface names
the one it used), which blocking_path.py renders in the "Also
noticed" section as an upper bound and never as a forecast, under the summary
metric "uncredited, benchmark first". Being numberless by design, it is exempt
from the valueless-pole exclusion (_on_pole_job), so a lint job that is also a
drilled pole still gets its card, and from _ALSO_NOTICED_CAP, so it never
falls into the "+N more" tail where its rules and ledger would not reach the
reader. A drilled pole whose job carries a non-advisory OPT82 finding whose
lint step is the pole's dominant step (_opt82_pole_for,
_opt82_lint_is_dominant; OPT82 itself fires whatever share of the job lint
takes, so without that gate a slow test job with a short lint step would lose
its gap analysis) counts it as that pole's catalog match at any size, unlike
OPT79, which covers a pole only from 30s: OPT82 is uncredited by design and its
claim is the named cause, so the pole never reads as a coverage gap, takes no
gap-fill or gap capture, and its waterfall and prompt point at the card in Also
noticed; the verifier's OPT82 check fails a covered pole that still renders
the coverage-gap wording. Cards are keyed per lint job (_group_by_pattern_ranked keys OPT82 by
pattern + workflow + jobs, like OPT73/OPT77/OPT79), because each job has its own
configs, rules, ceiling and benchmark directory. The card has its own prompt, not the
generic bill or off-path saving line: it carries the SIZING clause, tells the
agent to run the two-command benchmark FIRST, lists each enumerated rule with the
rewrite QUESTION (never the answer), gives the fix order (rewrite, split into a
fast syntax-only pass and a type-aware pass limited to exactly those rules,
native type-aware linter only if the remaining rules are supported) and requires
the ledger: every rule maps to a pass or to a REPLACED BY row naming what the
replacement no longer checks, for a human to approve, and a test proves no rule
is left without a row. Where the type-aware pass may run comes from the
collector's merge_group_workflows stamp (the sampled workflows declaring
merge_group, stamped at the call site because the detector sees one workflow):
with a merge queue the PR pass may lint an explicit changed-file list while the
full pass gates the queue; without one (or no stamp) the full type-aware pass
stays a required whole-tree PR check. ESLint's --cache is never offered for the
type-aware pass (it does not track cross-file type dependencies). The prompt
never tells the agent to disable rules. verify_report.py closes the loop with a
check that fails an OPT82 finding with a positive wall_clock_p50_s or a truthy
runner-minute saving, with no enumerated rule, stamped from a failed ESLint
read or from a truncated walk that never reached its lint directory, with no card of its own (matched
by workflow + job on the card's **Where:** line) or whose card omits one of its
enumerated rules or claims a saving, or whose rendered card lacks the
ledger sentence or says, in any wording, to switch the rules off (disable, turn
off, drop / remove, set to off, disableTypeChecked, move to a non-blocking
job). That rail reads the card's instruction text only: the measured evidence
and the command lines (the CI's own lint command, the benchmark's labelled
timing-only run) are data and are skipped. Its pole double-frame check
(check_pole_not_reframed_as_hygiene) mirrors the engine's OPT82 exemption from
the valueless-pole exclusion. The detector's call site in collect() is guarded: an
exception inside it skips OPT82 for that workflow and is disclosed through
detectors_skipped, never a crashed data pass.
Every exit is counted in findings_doc["opt82_withheld_by_gate"]. Three gates are
verdicts (the lint job never ran in the sample, it is below the cost threshold,
type-aware parsing is off) and count only there; one is workflow-level
(workflow_yaml_unparsed, no job to name) and counts only there too. Seventeen
are could-not-tell exits
and are also listed in findings_doc["opt82_withheld_candidates"], which feeds the
shared held-back registry row type-aware lint: held back (noun "candidate lint
job(s)", shape "job"), so a lint job this pattern could not decide is named in
the Data sources table rather than reading as "measured, nothing found".
The workflow-level workflow_yaml_unparsed is never listed; that skip is
disclosed through detectors_skipped.
OPT79 (a cache that costs more than it saves) is a measured lever: a wall-clock
lever on the long pole, and a Tier-2 bill lever below the cluster floor
(alongside OPT64, OPT65, OPT77 and OPT80 as Tier-2 bill levers). It reads job LOGS during collection
without --with-logs; the cache-evidence and push-probe logs are already read
under --with-logs, so what is new is that this probe needs no flag.
For a job whose workflow file declares exactly one cache-restore step
(actions/cache, actions/cache/restore, or an owner/setup-* action with
its cache input set; caches on by default — setup-go v4+ unless cache: false, setup-uv v5+ via enable-cache, and self-caching actions such as
Swatinem/rust-cache or setup-gradle (_OPT79_OTHER_CACHE_USES_RE) — count
toward the "exactly one") that is saved by its own post phase, names a known
package store, and is followed by a recognised install of the same ecosystem
with no unrecognised run: step in between, it classifies the sampled occurrences
HIT or MISS from the verbatim cache line — the _CACHE_HIT_RE / _CACHE_MISS_RE
the rest of the cache family reads, plus _OPT79_EXTRA_MISS_RE /
_OPT79_EXTRA_HIT_RE for the wordings those two cannot match: the setup-*
family's own miss line (<package manager> cache is not found) and
astral-sh/setup-uv's pair. Both extras are kept OPT79-local so the eight other
cache patterns reading the shared matcher are unaffected.
actions/setup-node v5+ with no (or an empty) cache: input caches AUTOMATICALLY when the
repo-root package.json names the package manager, so whether such a job has
one cache or two is a fact about that file, not the YAML.
_opt79_resolve_package_json reads it ONCE per repo, and only when
_opt79_package_json_needed finds a job in a sampled workflow whose cache count depends on it,
from the verified --root checkout first and otherwise with one
contents/package.json call — the same sources, in the same order, as
_fetch_workflow_docs — and the parsed object is passed to BOTH the plan and the
detector, so they share one answer. _opt79_setup_node_auto_cache mirrors
setup-node's getNameFromPackageManagerField per major: v5 reads the top-level
packageManager for npm / yarn / pnpm (^(?:\^)?(npm|yarn|pnpm)@); v6+ read
devEngines.packageManager then packageManager for npm only
(^(\^)?npm(@.*)?$); a SHA or branch ref (_opt79_ref_major accepts only a
version tag, so a SHA that starts with digits is not a major) is decided only
when both rules agree. package-manager-cache is on only when empty or true, as in setup-node.
On → a real cache (priced like any other, stamped setup_node_auto_cache so the
recipe names package-manager-cache: false); off → not a cache; unknown (file
missing / 404 / fetch failed / invalid JSON / not an object; no checkout before
the setup step, or a checkout into a path: or of another repository:, which
_opt79_checkout_state tracks; a SHA whose rules disagree) → fail closed as
setup_action_cache_default_depends_on_repository_files. The read is stamped
in data_sources.setup_node_package_json as {needed, source, readable}.
That scan is SCOPED to the restore step's own ##[group]Run <step> … block,
which ends ONLY at the next ##[group]Run or ##[group]Post header; the
action's inner groups and every ##[endgroup] are skipped (actions/cache
prints its hit/miss line after closing its own group). It has to be: Turborepo prints cache miss, executing <task>, Gradle prints
Build cache miss for task … and buildx prints its own, all matched by
_CACHE_MISS_RE and all in the TEST step's output. Read unscoped, every genuine
cache HIT in a JavaScript monorepo classified as a two-cache job and was
discarded — the lever was deadest exactly where node_modules is largest. Each
per-run row stamps log_line_group, and the verifier refuses a row whose group
is not the restore step. A log with no such group withholds the occurrence.
Inside that group, a log showing both a miss and a hit line is excluded, never
guessed, in either order (run_log_shows_both_a_hit_and_a_miss_line). A
restore-keys fallback prints no miss line — the @actions/cache toolkit prints
Cache hit for restore-key: <key> before the action's Cache restored from key:
— so _opt79_classify_log returns the partial_hit verdict on that toolkit
line, when the restored key (read from whichever line carries it) differs from
the echoed primary key:, or when the post step saved a new cache; it is not a
clean hit and is excluded (run_log_shows_a_partial_restore_keys_hit). Partial
and two-verdict runs stay in the hit share's denominator
(hits / (hits + misses + ambiguous)), so the minutes are priced on the share
of runs that took the exact-hit path. Only successful job runs are
classified (occurrence_did_not_succeed otherwise: a failed or cancelled run
also skips the post save); a skipped occurrence is occurrence_was_skipped, and
a block step whose timestamps are absent, unparseable or reversed (end before
start) withholds its occurrence as
step_timestamps_unparseable_in_this_occurrence, and one not reported as
skipped whose timestamps parse and run forward to no in-window time
(_step_span: a started_at in 1970 or earlier, the year-1 placeholder
included, or a start more than 1s before its job or after it ended) withholds it
as step_has_no_in_window_time. An occurrence whose log was
never fetched is counted as unread rather than folded into a population;
an occurrence past the per-job 8-log cap is tallied
beyond_the_per_job_log_probe_cap, not as unread. When
unread occurrences leave either population short the withhold says
population_truncated_by_unread_logs rather than blaming a thin sample; runs
that were read and then set aside (no restore group, no cache line, two
verdicts, a partial restore, another runner, a block that did not measure) do
the same as population_truncated_by_excluded_runs, or as
population_truncated_by_unmeasurable_step_times when the block-step time gates
(no in-window time, unparseable or reversed timestamps, a step that never
completed, an install that measured 0s) set aside more runs than the rest
together. Every finding and uncredited row stamps the runs it set aside per gate
(excluded_runs_by_gate, only when one was), so a population thinned on one
side is auditable.
The install step is recognised from what it RUNS, never from its display name —
so the near-universal name: Install dependencies spelling is found, and a step
NAMED npm ci that runs npm run build is not priced as the install. A run:
block whose every line is not an install (pip install -e . then pytest -q)
withholds rather than charging the test suite to both sides. The display name is
still what the step's duration is looked up by.
Among the shape gates that answer from the workflow YAML alone (so those jobs
cost no log fetch): a SEPARATE actions/cache/save step withholds
(cache_is_saved_by_a_separate_step — its save is not the restore's post phase,
so the block cannot measure it); an unrecognised run: step between the cache
and the install withholds (first_step_after_cache_is_not_a_recognised_install
— it may be the real install); a cache naming no known package store withholds
(cache_path_names_no_known_package_store); an install whose ecosystem does not
match the cache withholds (install_package_manager_does_not_match_cache); and
more than one cache, counting the on-by-default and self-caching ones, withholds.
Three things keep it honest. The step set comes from the YAML and a step the run
RENDERED stamped with no measurable length (start and end in the same second), or
a skipped step with parseable timestamps, counts as 0s, so GitHub's one-second
granularity cannot
change which steps are being compared between runs. "Rendered sub-second" and
"not there at all" are kept apart, because conflating them fails OPEN on the term
that matters most: on actions/cache the save runs on a MISS, so a post step
that never completed, or a constructed Post <name> that matched nothing GitHub
rendered, would drop the biggest miss-side term and MANUFACTURE the excess. The
first withholds the occurrence, the second withholds the job, and
actions/cache/restore — which has no post phase — stamps post_step: null
rather than inventing a label (that reading holds only because a separate
actions/cache/save step withholds earlier, so no unmeasured save can exist). And the
RUNNER-MINUTE credit gate requires either a still-positive floor margin after
rounding to 0.1s (below_cluster_floor) or a lead of at least 1s over the
workflow's long pole (below_long_pole), so shrinking the job cannot make the
workflow take longer;
wall_clock_p50_s=0 is the conservative lower bound on that finding (below the
floor the saving may still shorten the gate through a needs: chain, which this
audit does not claim; at or above the floor a job chained to the long pole of a
pull-request workflow is not credited at all, while on a push-only workflow it
is). Below the workflow's cluster floor that proof is
below_cluster_floor — unlike OPT77, the token is not historical for OPT79, it
is the proof; at or above the floor but below the long pole it is
below_long_pole, measured against the long pole itself. The floor does NOT gate _opt79_candidates (which is
also what _opt79_log_plan selects from): gating there meant a slow job's logs
were never fetched and its cache never classified: not a suppressed measurement,
an absent one, and one no reader could distinguish from a repository with no such
cache. The candidate merely records below_cluster_floor; the floor decides HOW
the measurement is credited, in _detect_opt79_net_negative_cache. The detector
measures both populations as usual and then splits three ways. (Counted by
cache_net_negative.kind there are three tags, and four emitting arms: the two
runner-minute certificates share opt79_net_negative_cache; the credited pole
finding is opt79_pole_net_negative_cache; the uncredited row is
opt79_uncredited_pole_cache, with its sub-reasons below.)
- strictly shorter than the workflow's long pole → the credited runner-minute
finding described above (
opt79_net_negative_cache, Tier-2,wall_clock_p50_s=0), on any workflow, with one of two certificates:below_cluster_floor(marginfloor_p50 − job_p50) when the job is strictly below the floor with a floor margin still positive after rounding, elsebelow_long_pole(marginlong_pole_p50 − job_p50from the 0.1s-stamped p50s, at least 1s: within 1s of the slowest job counts as tied, the same cutoff the slowest-job arm uses;refnaming the long pole and both p50s). The second arm is the owner decision of 2026-10-06: shrinking a job shorter than the long pole cannot lengthen the gate, because the long pole is longer than it — the same move OPT77's whole-workflow arm makes against its slowest member. The comparison is against the ACTUAL long pole's p50, never the second-slowest job's; - the workflow's long pole, on a workflow that runs on pull requests → a credited
WALL-CLOCK finding (
opt79_pole_net_negative_cache), described below; - anything else → a row on
findings_doc[_OPT79_UNCREDITED_DOC_KEY], whichblocking_path's_opt79_uncredited_blockrenders as one numberless line beside_dropped_unprovable_banner.
The long-pole finding is where the merge wait lives. The floor is the SECOND-ranked
job's p50, so "not strictly below it" spans everything from second place upwards,
and only the workflow's long pole — on a workflow that can gate a PR — carries
the merge-wait claim this audit makes (a job in a needs: chain can also carry
merge wait, and a long pole can be off the merge-gating spine). That job's raw wall-clock is the measured excess
waste_s, pre-capped by CAP 1 (§5) through bound_within_workflow:
min(waste_s, long_pole_p50 − floor_p50), i.e. credit the pole only as far as
the next-tallest job's duration. The cap compares job DURATIONS and does not
follow needs: chains, so it is conservative only when the next job runs
alongside the pole or directly after it (more of the excess may then come off
the wait); it does not model a longer needs: chain of shorter jobs, where
another path can be the slowest. The finding's prose is
worded for both shapes (it names the next job's duration and the gap, never a
time the next job "finishes at" or a claim that it sets the merge wait), and
verify_report.py restates that sentence from the stamps. The capped
figure and its inputs are stamped under cache_net_negative.pole_sizing
(waste_s, long_pole_job, long_pole_p50_s, next_tallest_job,
floor_p50_s, headroom_s, raw_wall_clock_s and capped_by_next_tallest_job) and then goes through the SAME
cross-cutting cascade every wall-clock-positive finding gets in collect() (the
developer-facing gate, the measured critical-path floor, the cross-workflow
floor), with any further shrink recorded in wall_clock_derivation. The finding
shares the _opt79_stamp measurement block with the other two outcomes, has
runner_min_saving null and no tier2_neutrality, and stamps
on_critical_path and workflow_gates_pull_requests both true. A long pole
TIED with the next-tallest job (under 1s of headroom) is not credited: shrinking it
moves no merge wait, so it falls through to the uncredited line. blocking_path
renders the finding inside that job's long-pole section (_opt79_pole_for /
_opt79_pole_block), opened by an <!-- opt79-pole:<id> --> marker, as
💾 Measured cache cost - OPT79 · <title> (<id>) - up to **Ns** off the merge wait on the P% of sampled runs where the cache hit (H of R runs read)
(the header carries the hit-rate clause because the figure is per cache hit),
with risk: LOW, the guardrail (re-key or narrow
before removing; valid for this runner class only; never narrow what the install
installs) and a rollout line. It needs that placement because its capped figure
is often below the 30s long-pole floor, which would otherwise drop it from the
"Also noticed" appendix as a valueless pole-job row; when its figure clears
the 30s long-pole floor it also counts as a catalog match for the pole
(_opt79_pole_covers), so that pole gets no gap-fill or gap capture. Under 30s
the block still renders and the pole's prompt only adds "ALSO MEASURED ON THIS
JOB". The appendix
never takes a pole-cache finding: one whose job is not a drilled pole (the
slowest job of another pull-request workflow, or any in a static-only report)
renders the same marked block in _opt79_off_pole_block, a short section
with its own ## heading, anchor and Contents entry before "Also noticed"
(never under the checkout stall tails heading before it; the verifier fails a
marker placed under any heading but a long pole or its own), so every pole-cache finding renders exactly once with
its merge-wait seconds and the appendix never summarizes it as "no bill
saving". Unless a leaf, structural or data-driven match fired first, and only
when the figure clears the 30s long-pole floor, the pole's waterfall and agent
prompt name OPT79 and point at its
block (opt79_present / opt79=) instead of the data-driven "see Also
noticed" pointer or the coverage-gap wording, and when the generic cascade
lowered the figure the block prints each wall_clock_derivation step and its
reason. Runner-minutes are deliberately NOT stated: the
runner-minute section requires proof that the job is shorter than the
workflow's slowest job (below_cluster_floor or below_long_pole), which the
slowest job cannot have. The neutrality certificate is only required to promote a
finding into the runner-minute section; a finding that saves merge-wait time does
not need one, because the wall-clock bound cascade already handles "shorten only
as far as the next job". verify_report.py re-derives min(waste, headroom) from
the stamped block and fails any mismatch, any such finding carrying
tier2_neutrality or a non-null runner_min_saving, and any uncredited row whose
on_critical_path is missing or not false.
There is ONE tie constant, 1.0s (_OPT79_POLE_MIN_HEADROOM_S), shared by both
arms: the pole arm declines a pole whose headroom to the next-tallest job is
under 1.0s, and a job counts as tied with the long pole (a co-pole) when its
lead over the pole is under 1.0s (each p50 rounded to 0.1s first).
The uncredited line therefore has five ways in: (1) the long pole of a
workflow that runs on no pull request, or a job tied with it (runner-minutes
only, unpriced here; tallied under long_pole_of_a_workflow_no_pull_request_runs);
(2) a long pole of a pull-request workflow tied with the next-tallest job
(under 1s of headroom; a job TIED with the long pole is a co-pole and is
stamped the same way), pole_tied_with_next_job; (3) a pole the cross-checks
zeroed to a figure that rounds to 0s (pole_merge_wait_zeroed_by_cross_check);
(4) a pole off the merge-gating spine (pole_workflow_off_merge_gating_spine);
and (5) a job of a pull-request workflow at or above the floor that is
needs:-chained to the long pole (job_in_a_needs_chain_with_the_long_pole;
the chain's sum is the merge wait, so shrinking it may shorten that wait, and
on a push-only workflow such a job IS credited). A chain the workflow's job
graph cannot resolve is held back (needs_chain_with_the_long_pole_unresolved),
never credited as unchained. The zeroed and off-spine poles are first built as
pole findings and then demoted to uncredited rows in collect() by
_opt79_demote_uncredited_poles, after the cascade and after
_stamp_off_spine_findings. Each of cases 2-5 carries uncredited_reason (plus
uncredited_reason_detail and the cascade's own steps as
uncredited_derivation for the zeroed case), and verify_report.py
re-derives each reason rather than taking it on trust: the zeroing from
uncredited_derivation, the off-spine demotion against the dropped checks,
and the chain from workflow_job_graph (failing closed when the graph cannot
resolve the job). The row stamps long_pole_job, long_pole_p50_s,
job_p50_s, floor_p50_s, workflow_gates_pull_requests,
declares_pull_request and on_critical_path, which is always false there
(the job is not the credited pole), so the renderer has no merge-wait branch.
For a workflow no pull request runs it says the workflow "does not run on pull
requests", unless the YAML declares a pull-request trigger and none was
sampled, when it says "none of its N sampled runs was a pull request"; for a
declined pole it states the stamped reason ("X is this workflow's slowest
job, but ..."; a co-pole reads "{job} is tied with {lp_job}, this
workflow's slowest job (Ns), so neither job alone sets the merge wait"); for
a chained job it says "X is not this workflow's slowest job". A
schedule-only workflow is never told it has a merge wait at all. The retired
"at or above the second-slowest job, so this audit cannot prove that shrinking
it leaves the merge gate unchanged" wording is gone with the case it
described, and verify_report.py's converse rule fails an uncredited row with
no reason whose job is at least 1s shorter than its long pole: that job is the
below_long_pole finding.
Uncredited rows are not findings — no runner_min_saving, no
wall_clock_p50_s, no certificate, no Tier-2 row, no contribution to any total.
They are built by the SAME _opt79_stamp builder as the credited blocks (one
measurement, three outputs, one contract), so they carry per_run and every
multiplier, and verify_report.py's check_opt79_uncredited_rows_rederived
re-derives them row by row with only the credited-minutes and neutrality branches
skipped. That check is top-level rather than part of the Tier-2 pass, because a
repository whose only OPT79 result is an uncredited row has no Tier-2 stamps for
that pass to run on.
Its gh cost is a capped log probe plus at most one contents/package.json read
per repo (only when a setup-node v5+ automatic cache decides a job's cache count
and the checkout cannot serve the file). The log probe is run during
collection WITHOUT --with-logs,
(_OPT79_LOG_PROBE_MAX = 8 occurrences per candidate job,
_OPT79_MAX_CANDIDATE_JOBS = 2 jobs per workflow, candidates ranked by job p50 —
a PROXY for what a cache can cost, not a measurement of it — and
_OPT79_REPO_LOG_BUDGET = 24 across the whole repo; the first two are per
workflow, so without the third a thirty-workflow monorepo would multiply them
into hundreds of fetches), planned across every workflow up front by
_opt79_log_plan and fanned out in one _prefetch_text wave before the detector
loop. _opt79_trim_repo_probe_plan applies the repo-wide ceiling by SORTING the
whole plan on candidate job p50 before slicing, so the budget is spent on the
costliest candidates in the repository rather than on whichever workflow file was
walked first; every occurrence it cuts is counted under
beyond_the_repo_wide_log_budget, and the cut jobs' remaining occurrences count
under occurrence_has_no_captured_log rather than looking like a thin hit/miss
population. _opt79_candidates is the ONE selector shared by the plan and the
detector, so the two can never disagree about which jobs are ELIGIBLE (which of
those actually got logs is the budget's business, and is counted).
verify_report.py's _opt79_net_negative_cache_rederived arm recomputes every
per-run block from its three parts, re-reads each row's quoted line against its
stamped verdict AND against the log group it was read in, and re-derives both
medians, the waste, the floor, the hit share, the effective volume, the credited
minutes and the margin. Its restatements of the engine are of two kinds, and the
tests say which: the stamp-key tuple, the four numeric thresholds and the shared
cache tail fraction are asserted IDENTICAL to the engine's; the hit/miss regexes
are independent re-readings and are only spot-checked line by line.
Every gate that withheld is counted into findings_doc["opt79_withheld_by_gate"]
and logged at DEBUG (a credit-eligible job on a workflow with no measured volume
is no_monthly_volume, counted after it is measured). A candidate whose logs were
probed and which then exits on anything but a verdict (_OPT79_VERDICT_GATES:
measured healthy, or hitting too rarely) is also appended to
findings_doc["opt79_withheld_candidates"] as {workflow_file, job, gate}, and
so is a job whose YAML showed a cache and which fell out BEFORE any log was read
(_OPT79_EARLY_HELD_BACK_GATES: two caches, unreadable package.json, a cache
that does not serve the install, an unrecognised first step, a separate save step,
an unpriceable runner, the per-workflow probe budget). A job with no cache at all
is not a candidate. blocking_path renders the list through the SHARED
held-back row builder described under OPT77 above, as the cache hit/miss verdicts Data sources row ("N candidate cache(s) held back (<jobs>): <reason>.").
The reason is a module-level dict, gate to plain-English phrase
(_OPT79_HELD_BACK_REASONS, registered in _WITHHELD_PHRASES_BY_KEY, with a
standalone copy in verify_report.py), covering
_OPT79_HELD_BACK_GATES; a test enumerates the gates from the source so a new one
cannot ship without a phrase. verify_report.py re-derives the whole sentence
(count, job list, reason) — the same re-derivation it runs for OPT77 and OPT80 —
and FAILS on a gate with no phrase, so a held-back cache
is not indistinguishable from no cache and a code is never printed. The probe's own cost is stamped ONCE, in
data_sources.cache_probe_logs as {planned, probed, returned, budget} — the
row the report renders and the verifier re-derives; when planned exceeds
probed the rendered cell says so. And when probes were planned but NOTHING came
back, OPT79 is recorded in data_sources.detectors_skipped for the affected
workflows, so an unevaluated detector is named rather than reading as a detector
that found nothing.
OPT57 now has a measured timeout-default-burn upgrade. A missing
timeout-minutes key is only the structural gate: collect_runs.py emits a
finding only when a failed/timed-out sampled job without an explicit timeout
burned at least 95% of GitHub's 360 minute default and ended no more than 15
minutes past it (a longer failed span was waiting, not runner time), and the same workflow job has
at least three explicitly successful timed samples that support a recommended
timeout-minutes above p99. Candidate workflows come from the scanned workflow
graph, so this detector can run even when a workflow has no prior static finding.
The recommendation is rounded up from
max(p99 + 10m, p99 * 1.5, 15m) and withheld when that approaches the default,
so long legitimate jobs do not become false savings; matrix jobs are withheld
until a per-variant p99 safety proof exists. The credited amount is the observed
failed-run seconds above the p99-backed timeout, scaled by the matching
event-scoped all-status workflow volume, with wall_clock_p50_s=0,
realization=none, structured timeout_default_burn evidence, and a
detector-specific post_completion_waste certificate re-derived by
verify_report.py. OPT57 uses exact non-matrix runtime job matching and its
job-scoped timeout samples participate in Tier-2 de-overlap against whole-run
eliminators without collapsing different timeout jobs in the same workflow run.
The cost spine is runner-minutes only. The pricing layer (scripts/billing.py,
per-SKU rate derivation, references/runner-rates.json, and the SKU-arbitrage
ceiling detector) was retired 2026-07-20 — the report states runner-minutes and
leaves the per-minute rate to the reader (the maintainers' pre-public
development archive preserves the pricing infra for later re-introduction).
The v2 cost-spine table is intentionally invariant-first. It must not be
rendered from ad hoc markdown-side aggregation. The implementation must first
produce a machine-readable findings JSON source block (working name:
runner_minute_spine) with stable row identity (workflow_file, job_name, runner_label, event_scope, status_filter, attempt_filter, volume_filter). Each row must stamp its sample window, sampled workflow-run
count, sampled job occurrence count, sampled positive-duration occurrence count
(the occurrences GitHub actually bills — zero-span jobs, started_at == completed_at, bill 0), occurrence fraction or effective monthly job volume, 30d
workflow volume, mean sampled compute seconds per occurrence, mean sampled
billable-equivalent minutes per occurrence (ceil(duration/60)), raw compute
runner-min/mo, billable-equivalent min/mo, and share of the all-row derived
total. Because a
bucket can mix a short real run with zero-span occurrences, its MEAN
billable-equivalent minutes can legitimately fall below 1.0 while its MEAN
compute seconds stays positive; the per-occurrence 1-minute floor is therefore
re-derived from the stamped positive-duration count (round(mean_billed × occurrences) ≥ positive-duration occurrences), never from the aggregate mean
alone. Raw compute minutes are diagnostic; the share-of-total re-derives from the
billable-equivalent basis.
The block must also carry repo visibility, or point to the findings root value.
verify_report.py must fail closed when a rendered
cost-spine row has no matching JSON row, when row identity or sample-population
semantics are ambiguous, or when raw minutes, billable minutes,
or percentage of total do not
re-derive exactly within the existing rounding tolerance. Render may sort or
truncate rows, but it must disclose hidden rows and keep the percentage
denominator over every row in findings JSON. Any renderer emitting a cost-spine
surface must stamp the explicit <!-- ci-speedup:runner-minute-spine -->
marker, and verification must fail when that marker appears without a valid
runner_minute_spine block. Reviewers must also reject visible cost-spine
headings, tables, or totals that omit the marker because the verifier cannot
bind unmarked markdown to the source block. The first cost-spine implementation
PR must be data + verifier fixtures before any visible table, matching the
Tier-2 stamp-first pattern from PR-1/PR-3.
collect_runs.py stamps runner_minute_spine, grouping
sampled successful latest-attempt jobs into all-events rows and sampled
prior-attempt jobs into separate attempt_filter: prior,
status_filter: all-status rows. Every row stamps volume_filter: all-status
so the sampled job status/attempt population and 30d workflow volume population
are explicit. Still-triaged workflows are fetched into a cost-spine-only sample,
leaving critical-path triage unchanged while allowing sub-floor workflows to
appear in the cost-spine source block. The collector seeds the explicit
workflow-coverage set from paths returned by GitHub's workflow list plus
repo-rooted .github/workflows/*.yml|*.yaml paths, so source-file findings such
as OPT19 tests/conftest.py keep finding provenance without becoming
workflow-volume coverage gaps while API-missing workflow files still fail closed
as unknown-volume workflows. The block stamps
coverage_scope: sampled_workflows_in_play_with_job_data plus
workflow_coverage counts/omissions/failures; complete_repo_coverage is true
only when every positive-volume workflow in play has rows, every in-play
workflow has known volume, and no cost-spine source-row job fetch failed.
When complete_repo_coverage is true, the producer stamps render_ready: true
and blocking_path.py renders the marked
<!-- ci-speedup:runner-minute-spine --> table at the top of the Runner-minute
reductions section. The table is derived only from runner_minute_spine,
includes the all-row Total row, sorts by billable minutes, caps visible rows, and
discloses hidden rows next to the table. When
the spine renders without promoted Tier-2 findings, the section heading is
Runner-minute cost spine rather than a reductions claim. verify_report.py
validates the source block when present, fails any marked table without a valid
source block, rejects unmarked cost-spine headings/tables/totals, parses the
marked markdown table when render_ready: true, re-derives every rendered row
and the final blank-metadata Total row from the source block, requires the
rendered rows to match the sorted visible source rows,
requires and cross-checks cost-spine fetch failures against data_sources, and
rejects complete coverage that is not render-ready (or render-ready coverage
that is not complete). The Tier-2 verifier also re-derives each visible R-row's
Source block line from runner_minute_spine, failing promoted savings rows
that lack render-ready matching source rows, whose rendered row
count/minutes drifts from the source block, whose marker IDs are
duplicated, or whose credited savings exceed the matched source-row totals
(billable totals for OPT65).
A one-line workflow ▸ jobs ▸ steps hierarchy glossary
(the report's only definition of the three terms) renders once, under the
## 🗺️ Long pole map heading, and only when that section renders (§12.0).
Exception — pre-start WALL-CLOCK wait. Queue time (OPT43, the _WAIT_PATTERNS
set) is NOT hygiene: it is developer wall-clock wait before a job starts, which the
critical-path spine (job start → finish) doesn't capture, and it carries no
runner-minute saving. So _queue_wait_block renders it in its own "⏳ Pre-start
wait (queue time)" section ABOVE the hygiene appendix (with a Contents pointer),
ranked by P90, excluded from _also_noticed_block. For a needs:-gated job the
metric is measured from when its last needed upstream job completed (upstream run time
is never queue), and the embedded agent prompt (_hygiene_prompt(wait=True)) frames it
as wall-clock wait, not a bill cut. See collect_runs
_detect_opt43_queue_time (wc_p50 holds the floor-capped savable wait for
this family; the raw P90 is in wall_clock_uncapped_p50_s, present only when a
bound actually capped the P90 — absent when nothing capped, where wc_p50 is
itself the raw P90).
The sizing rules live in two reference docs - read them before touching
_size_finding or the detectors:
references/wall-clock-methodology.md- critical-path / long-pole / cluster-floor model, the serial-gate wall-clock-negative case, and the non-additive stacking rule.
references/savings-methodology.md- two-axis sizing, monthly-volume computation, the cache-finding log-evidence guardrail, and the bimodal-tail / serial-gate guards.
5. The wall-clock lever model - a cascade of physical bounds
This is the model that keeps the report honest about developer wait. The mental model in one line:
developer wall-clock wait
= max over all workflows triggered on the PR of (that workflow's critical path)
a finding is a real wall-clock lever
⟺ it shortens the long pole of the GATING workflow,
and only down to the next floorA workflow's critical path is not the sum of its job-seconds. With fan-out
parallelism, wall-clock ≈ entry-gate + max(parallel jobs) + joiner + scheduling overhead. So shortening a job only helps if that job is the long
pole, and only until it drops to the cluster floor (the second-tallest job).
Across workflows, shortening one workflow only helps if it is the slowest
workflow running on the PR.
The two measurement sources (and why the model needs both)
The merge-wait side of the model is fed by a deliberate two-source hybrid:
- Per-commit check-runs (
commits/{sha}/check-runson sampled PRs,_fetch_check_runs) define what gates a PR: the full concurrent set across every workflow, including fileless checks (CodeQL default setup, third-party apps) no YAML scan or job listing can see. This is why the crown and the per-PRpopulationsderive from check-runs, never from any single workflow run's duration — the merge wait is a max over concurrent checks, not one workflow's runtime. - Sampled job timings (
actions/runs/{id}/jobs) define how long the work measurably takes. A check-run clock spans creation → completion, so it absorbs queue time and re-run inflation;_pole_capsbuilds per-check caps from the sampled job p50s (job_p50_all/job_bimodal_all) and_segment_pr_populationsapplies the SAME caps when buildingpopulations, so the engine's argmax and every verifier re-derivation see identically de-inflated values. Job data also feeds the step drill-downs and the runner-minute spine.
Each source screens the other's failure mode: jobs alone miss fileless checks and per-PR gating frequency; check-runs alone are inflatable clocks. A check with no sampled job to cap it has nothing grounding its span in measurable Actions compute — such fileless spans measure how long the gate sat open across the PR's lifetime and are excluded from the crowning basis entirely (disclosed separately; see §12, and issue #42 on the disclosure vocabulary for external-primary-CI repos).
The cascade (raw estimate → bounds → effective)
CAP 1 is applied during per-finding sizing (it's model-specific - sharding
intentionally skips it). The cross-cutting bounds run afterward in
wall_clock.size_wall_clock, which enforces the invariants and records the
derivation.
raw per-pattern wall-clock estimate
(default_s, or a measured per-run figure)
│
▼
┌─── CAP 1 - within-workflow critical path (in _size_finding / detectors) ─┐
│ own_job_p50 ≤ floor_p50 → 0 (below the cluster floor: bill only) │
│ else → min(wc, long_pole_p50 - floor_p50) (shorten only to next job) │
└──────────────────────────────────┬─────────────────────────────────────────┘
▼ size_wall_clock(raw, ctx) runs CASCADE:
┌─── developer-facing gate (bound_developer_facing) ───────────────────────┐
│ workflow's sampled events all non-PR (push/schedule/workflow_run) │
│ → 0 (post-merge/scheduled time, not developer PR wait) │
└──────────────────────────────────┬─────────────────────────────────────────┘
▼
┌─── MEASURED critical-path floor (bound_measured_critical_path) ──────────┐
│ from gh check-runs of a representative PR - INCLUDES fileless checks │
│ (CodeQL default setup, app checks) the YAML scan can't see. │
│ NO sampled job for this workflow (config-file / dormant finding) → 0 │
│ (a modeled saving on a workflow that isn't even a PR check) │
│ slowest OTHER concurrent check ≥ this workflow's slowest check → 0 │
│ else → min(wc, own_max_check - slowest_other_check) │
└──────────────────────────────────┬─────────────────────────────────────────┘
▼
┌─── cross-workflow floor (bound_cross_workflow) ──── FALLBACK ────────────┐
│ only when check-runs unavailable: slowest sampled sibling workflow │
└──────────────────────────────────┬─────────────────────────────────────────┘
▼
final wall_clock_p50_s + wall_clock_derivation on the finding
(runner_min_saving is NEVER touched by any bound - the bill saving is
real even when the wall-clock saving floors to 0)The measured critical-path floor is the most important screen: a finding
only keeps wall-clock credit if its job is at/near the slowest check that
actually runs on a representative PR (gh commits/{sha}/check-runs). This is
the only source that sees checks with NO workflow file - CodeQL default setup,
third-party app checks - which are frequently the true long pole. Without it,
a #3 job (e.g. mastra's Docs E2E at 334s) was crowned the top wall-clock win
while CodeQL (1359s) and changed-tests (788s) gated the merge unseen. The same
floor also zeroes a finding whose workflow has no sampled job on the PR at
all - a config-file finding like turbo.json (OPT58) carries only a modeled
per-run estimate (e.g. 30s) with no measured presence on the critical path, so
crediting it wall-clock would rank a hygiene tweak above the real long pole. It
keeps its runner-minute / cache-correctness value; it just stops claiming
developer wait it was never observed to remove.
The report makes this concrete in the Long poles section at the top
(_long_poles_block), the report's diagnosis. For each gating check it shows: how
often it is the ACTUAL pole across sampled PRs (from pr_critical_path.populations
— so a check that's huge but rare, like mastra's changed-tests ~750s on only
3/16 PRs, isn't crowned over Lint ~264s on 8/16), the per-step breakdown of its
job (pr_critical_path.poles, persisted by collect_runs via _decompose_job_steps
for EVERY top pole, not just structural-routed ones), and the dominant
(root-cause) step marked. This is pure diagnosis: the block prescribes no fix
and points the reader to the Findings section, where each gating check is a
finding with its own agent prompt. Render-time only (no new gh calls), purely
descriptive of the current blocking path.
The Findings section then lists the top-N detected inefficiencies by measured
impact (Δ wall-clock on the gating checks first), each as a root-cause card with
its evidence and a ready-to-paste agent prompt — no prescribed fix, no bang-for-
buck fix ranking. Findings past the top-N cap are named in a one-line pointer and
kept in the findings JSON. verify_report cross-checks the report
against the findings JSON.
Run frequency is a core ranking input (separate from the cascade), applied
RELATIVELY - no absolute cutoff. A per-run wall-clock saving on a workflow that
runs on few PRs removes little developer wait overall, so collect_runs records
each workflow's run-share (30d volume / busiest PR workflow) and the wall-clock
ranking weights each finding by it: Δwc × run-share. A saving on a workflow
developers hit on most PRs therefore outranks an equal saving on a rarely-run
one, and a docs-only paths: gate sinks proportionally - without any hard
demotion threshold. The top-N cap stays the only materiality gate (the same
relative-to-repo principle as wall-clock sizing: no absolute second/minute or
frequency floor). Each finding surfaces its share as a "Run frequency" row.
size_wall_clock enforces two invariants on every bound: monotonic-down (a
bound may only lower the saving - raises otherwise) and no-silent-shrink
(any reduction must carry a reason). It returns the derivation chain (bound,
before, after, reason), which collect_runs stamps on the finding and the report
renders as a per-finding audit trail. Adding a new bound = a new Bound appended
to CASCADE + its test; it then flows through every finding.
What is measured vs derived vs assumed
- Measured (directly from sampled gh runs/jobs/logs):
- per-job
started_at/completed_at→ per-job durations → p50/p95 (_critical_path,_percentile); - per-step durations (from the job JSON, no extra API calls), all read through
one rule (
_step_span): askippedstep, a step whosestarted_atfalls in 1970 or earlier (the API's year-1 placeholder included), one with reversed or unparseable timestamps, or one that starts more than 1s before its job has no duration; a step's end is clamped to its job'scompleted_at(a step left with no time by that clamp has no duration either), and a naive stamp is read as UTC. The 1s slack rests on a real-data census of 3,208 steps across 176 sampled job runs in curl, playwright, flask and mastra: 0 steps started before their job, 0 ended after it, and all 136 year-0001 starts were on skipped steps, so in that sample the slack never applied beyond whole-second rounding. Every dropped step and every end cut back by more than 1s is logged at DEBUG (job, step, reason; never a response body) and counted on the job's decomposition and its pole (skipped_steps,unmeasured_steps,partially_measured_steps,trimmed_steps; the decomposition always stampsskipped_stepsand the others only when non-zero, and the pole stamp carries each only when non-zero; a job whose every step was dropped stampsstep_decomposition_reason: no_step_measured_in_sampleinstead of a step list, and its pole keeps its pointer lines and log drill), and the pole's step list carries one line saying how many skipped or untimeable declared steps it leaves out or times from fewer runs, whichverify_reportre-derives. The drawn per-run timeline keeps an in-window skipped step in its place at 0s; the dominant step's cross-run sample counts a skipped run as 0s and keeps the drilled run with no value when its own step has no usable time. Each job's longest sampled run is stamped (per_workflow_timing[wf].job_max) as the boundverify_reportholds any one step to; - trigger events that actually fired each workflow (
run.event, collected intoevents_by_wf); - cache hit/miss and install/build log lines (
--with-logsfor the cache evidence and push-probe logs; OPT79's own bounded probe of up to 24 job logs is read during collection without it).
- per-job
- Derived (computed from the measurements):
long_pole_p50(max job p50) andfloor_p50(second-tallest) per workflow;- the concurrency sets (
_concurrent_workflows: which workflows share the chosen PR/developer-wait event); - both caps (CAP 1 long-pole headroom, CAP 2 cross-workflow headroom).
- Assumed, and flagged in the finding's prose:
- sharding splits a suite's work in half (N=2 conservative floor) for OPT24/parallel-rebalance sizing;
- concurrency ≈ a shared trigger event (
_concurrent_workflowspicks one primary event -pull_request>merge_group>push- to avoid counting apush-only sibling as concurrent on the PR path); - the ephemeral-runner cache caveat (OPT3/8/9): a "warm cache" wall-clock saving is unproven on ephemeral runners until a warm-vs-cold delta is shown.
Where the cascade lives
The cross-cutting bound cascade lives in wall_clock.py - a stdlib-only
leaf module (no config/collect_runs import, so the bounds are unit-testable
via import wall_clock without the report/config name collision). It owns
_resolve_job_p50, _wf_basename, bound_within_workflow (CAP 1, applied
during sizing), bound_developer_facing + bound_cross_workflow + _concurrent_ workflows (the cross-cutting bounds), the Bound contract, WallClockContext/
WallClockResult, the CASCADE list, and size_wall_clock. collect_runs
owns the sampling and the _critical_path aggregation that PRODUCES the crit
dict the bounds consume, and calls size_wall_clock once per wall-clock-
positive finding. The report renders the cross-workflow context as a "Concurrent
workflows on the same trigger" table plus the per-finding derivation line.
Remaining bounds are tracked in §10.
5.1 The measured sizing door — one derivation path for every runner-minute saving
The wall-clock cascade above bounds the merge-wait axis. The runner-minute
(bill) axis has a symmetric contract, enforced in ONE place: the measured
sizing door (collect_runs._reground_runner_minute_savings, run once after the
cost spine is final). It exists because each finding pattern historically carried
its own sizing path, so models kept pricing from modeled or single-sample bases
where measured data exists — three instance-fixes in a week (OPT45 #33, OPT73 #43,
chain_win_s #45). The door makes that structurally impossible:
The contract. Every finding that credits a positive runner_min_saving flows
through the door, which either DERIVES the saving from, or CLAMPS it to, the
MEASURED cost-spine billable of its affected jobs (completed_billable_equiv_min_per_month,
completed job runs only since 2026-10-12, or billable_equiv_min_per_month where
_measured_compute_basis_for names every sampled run; summed via the shared _measured_billable_index / _measured_billable_for_jobs
join — at least as strict as verify_report's _base join, so a door-derived
figure can only ever match a SUBSET of the rows the verifier bounds against). A
finding whose affected jobs miss the join UNSIZES honestly (runner_min_saving = None, basis unmeasured_no_spine_match) rather than render an unbounded modeled
figure. Every sized finding stamps runner_min_basis.
EXACT job identity in the join (issue #52). The join keys on
(workflow_file, base-job) where a base-job is the full job name modulo its
matrix-leg parenthetical — build-image / build-image (linux/amd64) and
(linux/arm64) are two legs of ONE job (build-image / build-image), while
build-image-streaming / build-image is a DIFFERENT job. The index already SUMS a
base's matrix legs into one figure, so both the door and the guard reduce a
finding's affected jobs to their DISTINCT base identities before summing: a
finding that lists several legs of one job adds that job's compute ONCE, never
once per leg (the mastodon build-push-pr double-count that added the two
build-image legs' 16,642.4 twice → 33,284.8, escaping the clamp). The door and the
guard apply the SAME dedupe principle (L3), each through its own base normalization
— the door's _whole_run_cancel_base_key is at least as strict as the guard's
_base/_cmp_name (it does no scope-stripping) — so they tighten together and the
door still bounds to a subset of what the guard bounds.
Per-pattern semantics, one basis + join. A cancel-rate model multiplies
differently than a step-decomposition credit, so per-pattern semantics stay —
but the MEASURED BASIS and the JOIN live in one place. The policy is a total
function (_rm_door_policy, over _RM_DOOR_OVERRIDES + the _SIZING model
family):
| policy | patterns | rule | basis stamp |
|---|---|---|---|
| derive | OPT45 | hit_rate × Σ(measured billable) |
measured_spine_billable |
| clamp | OPT73 | min(modeled, Σ(measured billable)) |
measured_spine_clamped (or measured_spine_billable when already within) |
| not_spine_derivable (the EXPLICIT whitelist) | the measured run-elimination detectors (OPT46/47/64/65 — basis is the eliminated-runs slice, not per-job billable); the measured setup-prefix detector (OPT77 — basis is the per-job leading setup prefix); the measured cache-block detector (OPT79 — basis is one job's restore + install + post-save timings split by the run log's own cache hit/miss line); the measured checkout tail-excess detector (OPT80 — basis is one step's mean-minus-p50 across the sample); the modeled-static patterns (direct / runner-min-only, disclosed as modeled in the report's sized-of-total ratio); the other structural step-decomposition levers (OPT70/71/72/74/75, per-job step basis) |
retained, with the reason recorded in runner_min_door_note; a figure the verifier does not re-derive (no sizing_basis: measured, no Tier-2 certificate) is capped at Σ(measured billable) when EVERY affected job resolves to its own workflow's rows and the estimate exceeds it (cap stated in size_note; a matched sum of 0 clamps to 0.0, "no bill saving") |
not_spine_derivable (measured_spine_clamped when capped) |
The whitelist is visible, not a silent bypass: a reasoned entry per family,
and a modeled or step-decomposition estimate stands below the physical cap: it
never exceeds the measured compute of its affected jobs when every one of them
resolves to its own workflow's spine rows (literal key, or through
workflow_job_graph's name: override, the verifier's own _identities rule).
The cap never uses a sum that might understate the jobs' compute, because it LOWERS
a figure: a foreign namesake from the cross-workflow fallback, or only some of the
affected jobs, leaves the estimate as is and the guard reports the gap. It also
never touches a figure the verifier re-derives from stamped evidence
(sizing_basis: measured, or a Tier-2 certificate: OPT57/65/77/79/80 and the
_SIZING measured detectors), since lowering one would fail that re-derivation.
curl's OPT16 (2026-10-10, a flat ~10 s × 1463 runs = 243.8 min/mo on
http3-linux.yml's linux) is NOT a cap case: its job has a templated name, so it
resolves to no rows of its own and is a coverage gap; the cap never fires there. A job declared under a
${{ }}-templated name: that misses its own workflow's rows never takes the
cross-workflow same-name fallback, in the door or the guard: its rows carry the
rendered name, so a namesake in another file is a different job (an honest
coverage gap instead). A rm-crediting pattern with NO declared policy stamps the loud
UNCLASSIFIED_door_policy sentinel, and check_saving_carries_measured_basis
(verify_report) FAILs on it — so a new pattern cannot ship its own unmeasured
sizing path. Two more invariants complete the class cut:
check_saving_within_measured_compute (a saving never exceeds the compute it cuts;
PR #30; both it and the door pick each finding's basis through ONE rule,
_measured_compute_basis_for, mirrored in the verifier and pinned by a coupling
test: completed job runs (completed_billable_equiv_min_per_month; timed-out,
cancelled and failed-at-or-past-6-hour runs excluded, stamped per row by reason and summed in
excluded_unfinished, named by the cost-spine footnote) for modeled and derived
savings, every sampled run for _FAILURE_WASTE_PATTERNS (OPT35/46/48/57/64) and for
measured or Tier-2-certified savings; on a job whose every run was excluded a modeled,
derived or clamped saving renders unsized with that reason, and the verifier refuses a
sized one; a pre-split row that may include excluded runs (all-status,
or a continue-on-error job) bounding a completed-basis saving FAILs, every refusal
collected, 2026-10-12) and check_cluster_lever_ceiling_escapes_sibling (§5's cluster-floor lever
must not be capped by its own matrix sibling — issue #44). The door SKIPs (like the
compute guard) when no render-ready spine exists, so a legacy pre-door artifact
never trips it.
5.2 Presence-weighting the cluster crown (issue #56)
The cascade above sizes a lever's ceiling by magnitude; §5.2 gates whether an OPT73 cluster-floor
lever may claim that ceiling as wall-clock at all. A cluster's fix only saves typical-PR merge
wait if the cluster is on the typical merge-gating critical path — i.e. its workflow gates a
majority of sampled PRs. So _detect_shared_substep presence-weights the cluster the same way the
spine ranks poles (§11): a minority-present anchor leg can't lead the Evidence, and a minority-gate
workflow demotes its wall-clock to bill-only (option b), keeping the runner-minute saving.
A majority workflow with one minority leg keeps its wall-clock and re-anchors (option a). The full
rule, its two-track (a)/(b) decision, and the converse guard live in
§12.1b;
noted here because it is a wall-clock-crediting gate, alongside §5.1's sizing door.
5.3 The presence denominator is PR identity, not head-sha (merge-queue dedup, issue #58)
Every presence signal in §5.2 / §11 / §12.1b — check_present_n_pr, the per-check
present_on, _gate_counts, _workflow_gates_minority, _leg_presence_eligible — is a
count over sampled PRs. Those "PRs" are the head-SHAs collect_runs walks
(pr_sha_ts → _select_repr_shas), one population row per SHA. That identification of
one head-sha = one PR breaks under a merge queue: a merge_group run executes on a
GitHub-generated temporary branch gh-readonly-queue/<base>/pr-<N>-<sha>, a DISTINCT
head-sha from the PR's own head. A repo that runs its heavy suite in the queue therefore
minted a second population row per PR carrying only the queue's checks, so the heavy
suite read as present on a minority of the (now inflated) denominator and §5.2/§12.1b
demoted the real merge gate to bill-only while a lighter pull_request check was crowned —
confidently wrong on exactly the large-OSS merge-queue shape.
The denominator rule, applied at the data layer before stamping (_group_dev_shas_by_pr,
between pr_sha_ts accumulation and _select_repr_shas):
- One population row per PR IDENTITY, not per head-sha. A PR's row UNIONs the
check-runs across every head-sha it ran on — its
pull_requesthead and itsgh-readonly-queuecommit — so a gate that runs only in the queue is present on that PR. - Queue → PR linkage. The PR number is recovered from the queue branch
(
_merge_group_pr_number) and matched to thepull_requestrun'spull_requests[0].number; runs sharing a PR number collapse onto one("pr", N)group. - Non-derivable fallback (the stated rule for unlinkable queue data). A
merge_grouprun whose branch is off the naming scheme collapses onto a SINGLE orphan class row — N such runs add one to the denominator, not N, so they cannot dilute PR presence. Their timing is still measured (the row carries the heavy suite); the wrong answer would be to silently drop queue data, so orphans are kept, just never multiplied. - No-queue repos are untouched. With no
merge_grouprun in the sample the grouping is the identity map (one member per row) — byte-for-byte the pre-#58 per-sha behaviour.
Because the correction lands in per_sha_checks before anything is stamped, populations,
check_present_n_pr, _gate_counts, and chain_facts all inherit it, and every
verify_report mirror — which re-derives presence/floors from the stamped, already-deduped
populations — needs no change. (Under a config-era straddle this PR-identity sample is scoped
ONE more time before those spine-facing consumers read it: _era_scope_pr_spine_sample (§2.1a,
issue #80) drops the dropped-era PR rows so chain_facts / populations / presence describe the
kept era only — a scoped COPY, so the full deduped sample still seeds the thin-flip decision.) (A verifier guard for the inflation signature is not
addable: the stamped artifact carries no run event / queue-branch provenance to re-derive it
from, precisely because the fix removes the signature before stamping.) Red-proofed by
tests/test_mergequeue_presence_dedup.py.
5.4 OPT81 — a runner-class gap measured from the repo's own runs, and the last-resort runner advisory
_critical_path is unchanged: it keeps each job's DOMINANT runner population
(the p50 the long pole is ranked on) and nothing else. A1 builds its own
success-only split per label and decides the comparison (gates, medians, gap)
from it. The credit reads both: it needs the slower label to be the one the job
runs on most in that split AND to be _critical_path's dominant label for the job
(counted over every conclusion and every label, unclassifiable ones included),
because that label's population is what the pole's p50 describes.
A1 (measured). _detect_opt81_measured_runner_gap runs in the per-workflow
detector loop beside OPT77/79/80. It groups each job's successful occurrences by
runner label and classifies each label through two named tables:
_OPT81_RUNNER_CLASSES gives the class (standard GitHub-hosted, GitHub-hosted
slim, larger GitHub-hosted size, StarSling), the operating system and the size
tier (-N-cores, macOS large / xlarge, a StarSling -N suffix), and
_OPT81_RUNNER_ARCH gives the processor architecture (-arm is arm64; macOS
-xlarge and plain macos-14+ are Apple silicon; macOS -large, -intel and
macos-13 and older are Intel; everything else x64). Anything else, including a
generic self-hosted label, is unclassifiable and dropped from the comparison
with a count. Two labels are compared only when: each has at least
_OPT81_MIN_SAMPLES_PER_LABEL (8) samples; they share an operating system AND a
processor architecture (otherwise the gap would measure the platform:
different_operating_system / different_architecture, verdicts); they differ in
class or size tier (two image versions of one class are same_runner_class, a
verdict); the two populations either co-occur in the same runs (a runner matrix:
at least half of the smaller population's runs also ran the other label) or
overlap in time (each label's first run is no later than the other's last run;
two disjoint periods are a runs-on switch, where code changed in between could
explain the gap: not_interleaved, held back and listed); every compared run
executed the same step list (a 16-hex digest of the executed step names, skipped
steps excluded); and the median gap reaches max(30s, 25% of the slower median).
The credit is the measured gap only when the job is its workflow's long pole, the
two labels are not a runner matrix (a matrix runs both legs in every run, so it
is stated, never credited), and the job ran on the slower label strictly more
often than on the faster one (an exact tie is not dominant), and the slower label
is the pole's own population: _critical_path's dominant label for the job
(stamped as faster_runner.pole_runner_label). When an unclassifiable label or
failed runs make another label dominant, the gap is stated and the credit held,
counted as slow_label_is_not_the_poles_population. It is pre-capped
with bound_within_workflow (CAP 1) inside the detector and then flows through
the generic cascade like any credited saving. runner_min_saving is always
None: a different runner class bills differently and the skill carries no rate
table (the 2026-07-20 pricing punt; OPT66 stays retired). The A1 downsize
shape (faster_runner.kind == "opt81_downsize", #429) is not priced either: two
numeric core counts of one class and one image, five interleaved samples each,
medians within 5%. It reads "same speed on a smaller runner size: about the same
runner-minutes, billed at the smaller size's lower rate", credits no wall-clock
and no runner-minutes, and states runner_cost_effect as unsized ("runner-cost
saving: unsized (ci-speedup does not price runners; the smaller size bills at a
lower per-minute rate)"), with the same reason in runner_min_unsized_reason so
the group heading never reads "no bill saving". verify_report fails a downsize
that carries a price ratio, a rate source or a currency figure. Pinned, frozen,
compatibility and release routes are a verdict
gate (downsize_pinned_or_compatibility_route). Every number is stamped
with the per-run rows it came from (faster_runner.rows: run id, label, run
time, duration, step-list digest), and
verify_report.check_opt81_runner_comparison_rederived re-derives both medians,
both counts, the class / operating-system / architecture / size decision, the
matrix-or-overlap decision, the step-list equality, the gap, the floor and the
credit rule from those rows, and re-reads the pole's population from
per_workflow_timing[wf].job_runner. The detector costs no gh call.
A2 (advisory). _detect_opt81_runner_size_advisory runs once, AFTER the
structural track, because its last gate is about the levers the rest of the
pipeline produced. It fires only on a merge-gating pole (pr_critical_path.poles)
of a pull-request workflow, whose dominant step category is compute (build /
test) and not a wait or a byte move, on a standard GitHub-hosted label (checked
after the shape gates; any other label, self-hosted included, is a verdict and
never a held-back row), whose job did not run on two or more runner labels in the
sample (A1 reports on those), and when no cheaper lever addresses the pole: no
OPT70-74 or OPT78 on it (the uncredited OPT75, OPT83 and OPT84 are the three
exceptions: present, they render first and do not hold it back), no finding
(OPT75 included) crediting at least half its p50 (A2's own rule, not the structural router's suppression), no OPT24, no OPT79
of 30s or more (credited or uncredited).
The facts the gate examined are stamped as cheaper_levers_checked and rendered.
It is advisory: true, carries no wall_clock_p50_s and no runner_min_saving,
has no _SIZING key, and is stamped risk / guardrail / rollout; the
verifier fails it if it ever carries a number or if gate (d) does not hold on the
findings doc. The log-level leaves are only known at render time, so the
renderer applies the last sub-gate: a pole whose own log matched a leaf renders
the advisory as a one-line "held back" note instead of a recipe.
Rendering and the conflict, stated plainly. The publisher of this skill sells
CI runners. Every OPT81 finding renders exactly once as an anchored card
(<a id="opt81-<id>">): at its pole, after the pole's own prompt and any OPT75
decomposition, when its job is a rendered pole; otherwise in a "Runner class
comparisons" section. It is excluded from the Also-noticed appendix, the Tier-2
section, the headline and every total. Each card and each prompt carries the
same one-line disclosure (_OPT81_DISCLOSURE, three equal copies pinned by a
test). The advisory names exactly two options, a larger GitHub-hosted size or
StarSling runners, and says plainly that the StarSling benchmark can only run
after the StarSling GitHub app is installed; a string-level test fails if any
OPT81 text contains a name from a deny-list of common runner vendors and hosted CI systems or
a domain other than github.com / starsling.dev.
6. Admission gate & advisory routing
A finding is emitted ONLY when all three of SKILL.md's admission criteria
hold (specific root cause, positive instance evidence, an applicable
tool-producible remedy). The implementation:
- Config-change-only remedy.
OPT13(build step in jobs that don't need it) andOPT15(cross-workflow build redundancy) need judgment that has produced confident-but-wrong findings, so they are never auto-emitted. (The retiredreport.pysurfaced them plusOPT18as a manual-review checklist appendix; the spine renderer dropped that appendix — they stay catalogued for human application and are not rendered.) Omit rather than fake. - Reliability / non-CI-config remedy → advisory. A "finding" whose only
remedy lives in a domain ci-speedup can't edit (e.g. "go fix your flaky
test") is a signal, not a ranked optimization.
scan.py's_ADVISORY_PATTERNS = {"OPT19"}(test-source sleeps - the fix is test code, not CI config) stampsadvisory: trueat emit time;collect_runs.pydoes the same forOPT48(high failure rate).blocking_path.pyexcludes advisory findings from the "Also noticed" appendix and the headline - they are not rendered in the report (they stay in the findings JSON). - Unprovable cache → dropped. With
--with-logs, a cache-family finding whose logs show no cache line AND no install/build activity AND no measurable cost is unprovable (absence is not evidence)._attach_cache_log_ evidencemarks it_dropandcollect()moves it todropped_unprovable. When logs show the cache hitting on every sampled run, the finding is kept but its wall-clock is zeroed and re-flagged as config hygiene (the evidence refutes a wall-clock claim).
scan.py never emits an OPT-id absent from the catalog, and reports every
catalog pattern lacking a registered detector in
catalog_patterns_without_detector rather than faking coverage.
7. Evidence, provenance & verification
Measured evidence. Data-driven findings carry a
measured_evidencetable whose cells link the actual GitHub job logs (html_urlper sampled job) - a timing claim is proven by run timings, not by a workflowfile:line. The report renders this table instead of a code permalink. Cache findings quote the verbatim cache hit/miss line from the job log.Provenance.
scan.pyrecordsskill_commit_sha,commit_sha, andrepo(run.pyforwards them, deriving each from git HEAD when not passed, so a run never records a NULL sha).blocking_path.pyrenders the metadata header asci-speedup skill commit <sha>, appending-dirtywhen the orchestrator passedskill_tree_dirty(the HEAD sha alone can lag uncommitted working-tree edits). The catalog is linked at that exact commit (content-addressable permalinks).Installed copies have no git HEAD — the lockfile is the fallback (issue #2). The skills CLI installs
skills/ci-speedup/as a recursive copy with no.git, so the git derivation above returns None. Rather than record a NULL sha (which blanks the footer and FAILScheck_skill_commit_provenance, historically forcing the agent to re-run the whole gh data pass just to pass a guessed--skill-commit-sha),run.pyreads the installer's.skill-lock.json(a sibling of the installed skill dirs; a lockfile inside the skill root is accepted defensively) and stamps a distinctinstalled:<hash12>provenance form from the entry'sskillFolderHash— or the terminalinstalled:unversionedwhen no lockfile/entry exists. Never NULL, never a failed run, never a guessed remote sha; an explicit--skill-commit-shastill wins. The lockfile records a content hash, not a commit, soblocking_path.pyrenders it as a plainskill build <installed:…>identity string with no fabricated commit/catalog URL, andcheck_skill_commit_provenanceaccepts theinstalled:forms for live/installed runs (verified without--skill-repo) while still rejecting them for committed worked examples (verified with--skill-repo), which must carry a real, resolvable git sha.Provenance is squash-proof for committed examples. The problem, stated first: a report is stamped with the commit that rendered it, and a later squash-merge discards that commit, so a pure ancestry gate turns
mainred for a legitimate, unchanged report (it did, once).The original resolution was to stop enforcing ancestry on the worked examples -
test_committed_reports.pypassed no--skill-repo, socheck_skill_commit_provenancereportedskipped. That kept only the weaker guarantee that a well-formed, non-null skill commit is recorded. It was a deliberate, documented tradeoff, not an oversight - but it meant two committed reports (psf/requests,mastra) carried skill commits unresolvable onmainand nothing said so.Superseded 2026-07-08 (OD10). A squash-merge discards commits but preserves trees.
blocking_path.pynow also stamps the render-time git tree ofscripts/(_skill_scripts_tree_sha), rendered as(skill commit <sha>, scripts tree <tree>).check_skill_commit_provenancepasses when the recorded commit is an ancestor or the recorded scripts tree equals HEAD's exactly, and refuses a-dirtytree (rendered from uncommitted code, whose sha names the committed tree rather than the code that ran).scripts/and not the whole skill dir, because the skill dir can contain committed reports (the archive'sreports/corpus) - stamping the whole dir would make committing a report change the very tree that report claims to have been produced under.So the gate is now enforced on the committed examples, against the real checkout, and it no longer false-reds on a squash. The cost, accepted explicitly: an exact tree match means any change to
scripts/makes every committed report stale until re-rendered (zero gh calls; it re-renders from the committedfindings.json). That is what makes the "ship the refreshed worked examples with the change" rule a red test rather than a convention. Net: any merge method - squash or merge-commit - is safe.What
verify_report.pyenforces (one PASS/FAIL command, standalone - no imports of the skill scripts): the primary section is present (>=1## Long pole N:, or the fileless note); the title +Bottom linename the wall-clock (merge-wait) axis; every#pole-N/#pre-start-wait/#runner-minute-reductions/#also-noticedreference resolves to an anchor; the RCA hands off via per-finding agent prompts and never prescribes a fix or dead-ends; the TOC's "Also noticed" count equals the appendix rows + hidden pointer; the data basis is disclosed and any coverage gap names its file(s);Date (UTC)matches the filename date; the report is ASCII-hyphen only (no typographic dash); no ci-secure security-domain leakage; (with--skill-repo) the skill commit is HEAD or an ancestor of it, or the renderedscripts treetoken equals HEAD'sscripts/tree (the squash-proof path, OD10); every rendered finding pattern exists in the findings JSON; every data-driven finding carries ameasured_signal; Tier-2 rows re-derive from the stamped measured basis, neutrality certificates, de-overlapped totals, and billing rates; and the report drills one fully-formed long pole per independent gating check (>=2 when >=2 gate), each ending in a hand-off prompt — the sole exception being an aggregation gate (§12.6b), which carries no drill and no prompt by design and is held to the mirror-image invariant instead (it must name its slowest upstream member and prescribe nothing).
8. Reproducibility & determinism
Detection, ranking, and the measured numbers are deterministic by construction
(no LLM in that loop; the phase-4a gap-fill is the lone, labelled exception and
touches none of them). The remaining
source of drift was live gh re-sampling: a regen would sample whatever runs
exist now, producing a different finding set. The run-sampling pin closes
this: collect_runs.py --created-before <ISO> samples only runs created at or
before the timestamp.
_window_30d(created_before)- unpinned, the 30-day window ends "now" (created>=now-30d, original behavior); pinned, it ends at the pin (created=now-30d..pin) so the window doesn't drift forward. "Now" is_unpinned_now(), resolved ONCE per process.datetime.now()has second resolution and a real collection runs for minutes, so re-reading the clock per call made the unpinned window slide mid-run: two workflows' volume probes issued a second apart were counted over two different 30-day windows. One window per run._sample_runs(...)appends&created=<=<pin>so the sampled N successful runs are reproducible;_monthly_volumeand_detect_opt48_failure_rateuse the same pinned window for theirtotal_countqueries.- The pin is recorded as
data_sources.sampled_runs_created_before(null = unpinned). Pass the prior audit's scan time to a regen and it reproduces the same finding set instead of re-sampling live history.
gh usage is frugal: one workflow-list, one total-count per workflow, one
job-list per sampled run (default 8), one log per hottest cache job under
--with-logs, up to 24 job logs for the OPT79 net-negative-cache probe (one
prefetch wave, no --with-logs needed), and - only for a job that has already cleared every cheap OPT80
gate - up to four job logs for that job's tail runs (_OPT80_LOG_PROBE_MAX),
newest-first. No per-step API calls (step timings come from the job JSON).
How those calls are issued (one shared pool, bounded prefetch waves, a token-wide
rate governor) is §2.2; it changes the wall-clock of the pass, never its contents.
8.1 What the data pass does NOT re-fetch
Four things the pass could ask GitHub for twice and gets locally instead.
The equivalence is CONDITIONAL, and the condition is checked at runtime, not
assumed. Each derivation below can decide the value from a payload already in
hand for most inputs — and for the inputs where it cannot, it says so and pays
for the fetch. That distinction is the whole design: a derivation that returns a
confident answer from a payload that could not support one does not save a call,
it manufactures a wrong number. Two of these carry a "cannot decide -> UNKNOWN,
go ask REST" branch for exactly that reason, and unknown != empty is the rule
they enforce (§7).
On the corpora we measure, the conditions hold nearly always: the offline
full-pipeline replay drops 41 gh calls to 35, every sampled value byte-identical,
and the call count is pinned as a golden integer (_GOLDEN_GH_QUERY_COUNT) so a
future change to the budget has to be a deliberate edit.
One run-list page per workflow.
_all_status_runs(all conclusions) is a superset of thestatus=successsample, so_success_runs_from_all_statusderives the sample from it and the run-elimination detectors (OPT35/46/47/57/64) reuse the same page. Cannot decide when: the page is FULL (truncated at_COST_RUNLIST_MAX) and still holds fewer than--max-runssuccesses — it cannot see far enough back. Only then does the explicitstatus=successquery fire. A short page is the workflow's whole visible history, so its successes are all the successes there are; falling back there would re-fetch the identical runs and make a monorepo full of small/rarely-run workflows pay two run-list calls where it used to pay one — turning the reduction into a regression on the repos with the most workflows. If the fallback query itself fails, the derived sample is KEPT (_sample_runs(...) or runs): a short real sample beats no sample. The page is also cached fetch-once — but only on success. A failed fetch is never cached:_all_status_runsreturnsNone(not[]), because[]is a real answer ("this workflow has no runs") that would report the entire run-elimination family CLEAN over a literal "0 of 0 runs" evidence line on the strength of one transient timeout.One job-list per attempt-run. REST's
filter=latestis a server-side filter on a field thefilter=allpayload already carries (job.run_attempt), so_latest_attempt_jobsderives it. This holds on partial re-runs too, which is the non-obvious part: GitHub materializes the FULL job graph into every attempt, so a job that "Re-run failed jobs" did not re-execute still appears under the new attempt — new job id, newrun_attempt, original timestamps. There is no carried-over job left behind atrun_attempt: 1for the derivation to drop. (Pinned against recordedfilter=all+filter=latestpayloads for a real 3-attempt partial re-run:tests/fixtures/gh_recorded/.) Cannot decide when: the payload is TRUNCATED. The jobs endpoint is fetched unpaginated andfilter=allreturns jobs oldest-attempt-first, so a100-job run's page 1 can be entirely prior-attempt jobs — every one of them carrying a
run_attempt, so a missing-basis guard sails right past it. Left unguarded the derivation returns[],_dominant_prior_failing_jobfinds no latest-attempt keys, and OPT64 can never fire on a big-matrix repo — a guaranteed false-negative class on exactly the repos where re-run waste costs most. So a truncated payload is UNKNOWN, and_attempt_job_samplesre-fetchesfilter=latestfor that run. Cost is bounded by the number of truncated attempt-run payloads (zero on the better-auth corpus).Truncation is a fact about the PAYLOAD, and it disables the attempt-scoped derivations on BOTH sides.
_JobsPayload(alistsubclass) carries a.truncatedflag set by the fetcher from the endpoint's owntotal_count(len(jobs) < total_count), falling back to the page-size heuristic when a caller hands over a bare list._prior_attempt_jobschecks it first, before any basis — the explicitrun_attemptpath included:- a truncated
filter=allpage yields a prior-attempt set that is silently SHORT (recorded: dbt-core run 29121623799 — 100 of attempt 1's 114 jobs), so OPT64 would size re-run waste against a partial set and the "unique dominant failing job" contest could crown a job that only wins because the true one was cut off — a wrong root cause, asserted confidently; - a truncated
filter=latestpage yields shortlatest_keys, so a prior failing job that IS in the latest attempt looks absent and the finding is withheld.
Deriving truncation from
total_countrather than from length is what keeps this from becoming a permanent OPT64 kill-switch: when the jobs fetchers are paginated, a complete 150-job payload reportstruncated=Falseand every guard keyed on it becomes a no-op, rather than declaring the complete payload UNKNOWN.Deriving
filter=latestalso fixed a double-count: the two-fetch path ran the fetch-failure accounting once per fetch, charging one unreachable run to the cost-spine coverage gap twice.- a truncated
Workflow YAML comes off disk.
_fetch_workflow_docsreads<--root>/<wf_path>(the checkoutscan.pyalready walked), falling back toGET /contents/{path}when the local read yields no usable doc — missing, empty, conflict-markered, or invalid YAML alike. (All four are the same fact, "we learned nothing here". Falling back only on a missing file would DROP a present-but-broken workflow, silently no-opping everywf_doc-gated detector for it — OPT35's shard specs, OPT57's timeout specs, OPT24's shard recognizer,_declared_pr_workflows— and an absent finding reads as clean.) This is a tradeoff, not a free win, and the two sources genuinely disagree whenever the checkout and the default branch do./contents/serves the default branch's HEAD, which is not necessarily the commit the report stamps as audited. The checkout's working tree is what the report stamps — so parsing it makes the stamp true — but it can also be dirty, or a feature branch that produced none of the sampled runs (a workflow that gained apull_requesttrigger last week has PR runs in the sample but a push-onlyon:block in an old checkout, and_declared_pr_workflowswould drop a real PR gate).run.pypasses--rooton every run, so this is the default path, not an opt-in. We prefer the checkout — parse what you claim to have audited — and disclose the skew rather than hide it, on three axes:- Right repo? The local read is used only after the checkout is verified to
be a clone of
--repo(_root_is_clone_of, via the origin remote). A mismatched--rootwould otherwise size one repo's workflows against another repo's timings, silently. Unverifiable (no remote, not a checkout) falls back to the API. - Right commit line?
_root_branch_skewcompares HEAD against the repo'sdefault_branch— a feature branch, a detached HEAD (unless it is exactly the default tip, asgit worktree add --detachleaves it), or amainthat is behind/ahead oforigin/main. This is the half-dirtycannot see: a clean checkout on a feature branch has no uncommitted change, so the stamped sha is perfectly true while the detectors parse YAML that produced none of the sampled runs. Stamped asdata_sources.workflow_yaml_skewand rendered as a NAMED warning in the provenance block. We disclose rather than silently switch to the API, because quietly parsing the default branch while stamping the checkout's sha breaks parse what you stamp in the other direction. - Which source, per workflow?
data_sources.workflow_yaml_sourcerecords how many workflows were read from the checkout vs fetched from the API, and the Data-sources footer renders it. (Uncommitted edits are separately stamped: the audited commit renders<sha>-dirtywhen.github/workflowsis dirty,_workflows_are_dirty.)
- Right repo? The local read is used only after the checkout is verified to
be a clone of
A run's jobs are fetched once.
_JobFetchMemokeys the per-run job fetch by (fetch flavour, repo, run id), so the event-scoped detector probes (OPT36's schedule sample, the OPT35/OPT57 failure samples) read runs the main pass already fetched instead of re-fetching them. Only successful fetches are memoized - a failure is re-tried, never inherited - and hits are deep-copied, so two call sites never alias the same job dicts. The miss path takes a per-key lock, so two threads racing the same run id make ONE call (a plain check-then-act de-duplicates only sequential accesses — fine for today's per-workflow pools, wrong the moment the job pools are flattened cross-workflow).
A SKIPPED detector is disclosed, by name
The unknown != empty rule (§7) says a detector whose basis run list could not be
fetched must be SKIPPED, never sized against a laundered [] — which would render
"0 superseded runs of 0 runs sampled" as CLEAN off one transient gh timeout.
Skipping alone is only half of it. A finding that never appears reads exactly like a finding that looked and found nothing. Left at a stderr warning, the false negative just relocates: from reported clean off 0 runs to silently not evaluated, which is harder to catch because it looks fixed. The generic gh-error footnote cannot carry the disclosure either — it says a few runs are absent and the P50s are marginally thinner, which is a different failure and simply false here (no P50 is affected).
So the skip is DATA, and the reader sees it:
collect_runsrecordsdata_sources.detectors_skipped—[{workflow, detectors, reason}]— at each skip site (the all-status page for OPT35/46/47/64, OPT57's scoped page, OPT36's schedule page). Only detectors that WOULD have been dispatched for that workflow are named (OPT35/OPT57 are additionally gated on the workflow's own YAML, which is available regardless).blocking_pathrenders a NAMED provenance bullet per affected workflow: "ci.yml: OPT46/OPT47/OPT64 did not run … their absence from this report is UNKNOWN, not clean."verify_reportenforces it: a non-emptydetectors_skippedwhose workflows and detector ids are not named in the rendered report is a FAIL.
gh_error_count counts distinct unavailable resources, not attempts at them:
the run-list page is deliberately re-tried by the detector loop after the shallow
loop's fetch failed, and billing one dead page twice inflates the coverage banner
against the only question its reader is asking.
9. Testing strategy
Tests live under tests/ and are picked up by the repo-root pyproject.toml's
testpaths; CI runs pytest -v.
Detector unit tests (
test_scan_detectors.py, ~40 cases) - positive and negative (suppression) cases for the static detectors: OPT1/2/5/9/14/16/ 18/21/27/28/29/31/33/36/39/62/63/76, including load-bearing carve-outs (OPT28 git-history jobs and local composite actions, OPT76 declared-payload reads and transitively-resolved local composite actions, OPT33 aggregator/change-detection suppression, and the shared activation-fidelity gate — apull_request: types:-gated or job-if:label/activity-gated job is not "every PR", so OPT33/39/40 suppress it:_pr_trigger_runs_every_pr/_job_runs_on_every_pr).Sizing & bounds unit tests (
test_collect_runs_sizing.py, ~25 cases) - the cascade and the sizing models: cross-workflow concurrency (_concurrent_ workflows, both caps, the non-gate zeroing and no-op cases), the--created-beforewindow math, serial-gate negativity, below-floor demotion, matrix-name → display-name resolution, job-scoped sizing (OPT33/40) vs the long pole, qualitative degradation, and the ephemeral-cache caveat.Measured-evidence & render tests (
test_measured_evidence.py) - OPT24 long-pole context, OPT25 heterogeneous-split vs sharded-rebalance framing, verbatim cache miss/hit extraction, the unprovable-cache drop, OPT48 advisory shape, and that the report renders the evidence table (not a YAML fence). Several assert against committed worked-example reports underreports/(a corpus maintained in the pre-public development archive — this repository ships its public worked examples underexamples/instead, and these guards skip loudly when the corpus is absent).Committed-report guard (
test_committed_reports.py) - the committedreports/<repo>/(archive corpus; skips when absent) are documentation (illustrative worked examples), NOT the test input. The guard renders each committedfindings.jsonfresh with the current renderer and runs the invariants (no typographic dash, no coverage-gap dead-end, and the fullverify_report.pycheck set) against that fresh render, so a real renderer↔verifier drift SURFACES instead of hiding behind an old.md. A pre-existing per-check drift is tolerated only via_KNOWN_DRIFT(with a self-expiring tripwire), never a whole-repo skip; a non-empty-render + min-checks-fired floor blocks a vacuous pass. The committedfindings.jsondata shape is separately gated bytest_measured_evidence.py.Amended 2026-07-08 (OD10). This bullet used to add that the examples are "free to lag the renderer" and that "a renderer change can't force a report regen". Both are withdrawn. Unenforced in either direction, that licence let
langfuseandmastradrift silently from the renderer for weeks. A committed report must now equal a fresh render and carry ascripts treetoken resolving in this checkout, so a change toscripts/does force a re-render - one command, zero gh calls, since it renders from the committedfindings.json. Rendering also writes<report>.md.claims.json; that sidecar is part of the committed artifact (it switchesverify_reportto manifest-first comparison and binds each claim's rendered sentence to the report), so it is committed too and guarded for tracked-ness, not merely presence.Blocking-path report tests (
test_blocking_path.py, §12) - the five leaf detectors + the unrecognized-log fallthrough; the step timeline Gantt (offset bars, dominant-step marking, tiny-step collapse, P50 fallback, the check-vs-job reconciliation); and the cross-run check (bracket-vs-median labelling by n, the IQR verdict that reads a tight-cluster-plus-outlier as stable, n<2 suppression). The bundle capture —_persist_pole_logsnearest-P50 selection,_step_timeline, and_magnitude_sample's probe→escalate + categorical-skip — is covered intest_structural_findings.py.Artifact guard -
verify_report.pyis the one-command report-invariant checker (see §7), runnable on any landed report and wired into the e2e flow.
Em-dash sanitizer
The rendered RCA reports must use a plain ASCII hyphen. Em-dashes leak in
from many sources (hardcoded prose, detector notes, catalog TL;DRs), so
blocking_path.py's render() strips them once at the
render boundary via _strip_emdashes (replacing the glyph with -, preserving
spacing) rather than chasing every source. The committed-report guard
(test_committed_reports.py) renders each committed findings.json fresh and
fails CI on any typographic dash in that render (catching a renderer regression,
not a frozen snapshot), and verify_report.py re-checks no-typographic-dashes
per landed report.
Fence sanitizer (repo-text can't break out of a fence)
Repo-controlled free text — GitHub check/job/step names and verbatim
captured job-log / workflow-YAML evidence lines — is dropped into the rendered
report, both inside ```text fences and inline code spans / headings. Left
raw, a run of >=3 backticks would close a fence early: the rest of the report
renders as broken Markdown on GitHub, and verify_report.py desyncs (it splits
the same text with re.findall(r"```text\n(.*?)```"), so the identical stray
fence fools the safety net). _fence_safe(s) neutralizes it — defuse any >=3
backtick run to an equal-length apostrophe run, collapse embedded newlines/CRs to
a space (a name/step/log line is ONE line), drop dangerous control chars — and is
**byte-identical on clean single-line input**. It is folded into _clean_label
(the canonical NAME normalizer every check/job/step name flows through) so the
pole heading, the ```text waterfall labels, and the agent-prompt fences are
all neutralized at one chokepoint rather than per-sink; the heading also wraps
the check as an inline code span (_safe_span) so a */_/backtick can't render
as formatting. Verbatim evidence lines (which bypass _clean_label) are
fence-safed at each emission site, prompt bodies per-line (_fence_body), and
prose/table cells via _flatten_cell. verify_report._strip_scope mirrors the
_clean_label transform (kept verbatim-coupled by test_s1a) so the name
comparators stay aligned, and a defense-in-depth check_fences_balanced FAILS if
a CommonMark fence walk ever ends still inside a fence.
10. Status of the planned bounds
The cascade extraction (wall_clock.py, the Bound contract, the monotonic-
down + no-silent-shrink invariants, and size_wall_clock wired into production
with a rendered derivation) is done (§5). Disposition of the four physical
bounds the gaps list originally named - three implemented, one deliberately not:
- ✅ Developer-facing gate (
bound_developer_facing) - implemented. A workflow whose sampled triggers are ALL non-developer-facing (push-to-main, schedule,workflow_runafter merge) is not on the PR critical path, so its wall-clock saving is zeroed (runner-minutes untouched). Uses the measuredrun.eventset; removes the over-claim of crediting post-merge/scheduled time as developer PR wait. - ✅ Per-event critical paths - implemented.
_critical_pathis computed from the developer-facing event's runs (_developer_event:pull_request, elsepull_request_target(the fork-PR / triage variant a PR also waits on), elsemerge_group) when the workflow fired on one, not blended across triggers - a workflow's PR runs can have a different long pole than its push-to-main runs (e.g. better-authci.yml: 493s on PR vs 439s blended).pull_request_targetis a developer-wait event for this scoping: a workflow onpush+pull_request_targetwith no plainpull_requestonce found NO developer event and fell back toall-events, blending its post-merge push runs into the PR wait (roboflow/supervisionpr-conflict-labeler: a 120s push-mode retry that gates zero merges became a false "~2m on ~30% of PRs" bimodal gate). The crit'sevent_scopeis recorded and surfaced in the concurrent-workflows table;verify_report'scheck_speed_poles_completere-derives it and FAILS any PR-critical-path pole still scopedall-events, except the narrowtiming_source=pr_check_runscase where the pole p50 comes from sampled PR check-runs and no workflow-job step drill is present. Detector evidence p50 stays over all sampled runs (more robust); only the sizing critical path is PR-scoped, and the table notes the scope so the two are never confused. - ✅ Matrix post-shard residual floor - already handled in the OPT25 detector's sizing (the saving is floored at the next-slowest leg). Lives in the detector, not the loop cascade, because it's pattern-specific.
- ❌ Required-status-check gating - deliberately NOT built. Two blockers,
both confirmed: (1) data - the required-checks list lives behind branch
protection, which returns 404 without admin on the repo; ci-speedup's
normal case is auditing repos you don't own, so this would be "unknown" on
almost every real audit. (2) semantics - only required checks block the
merge, but developers wait on non-required checks in practice, so discounting
a non-required workflow's saving is genuinely ambiguous (merge-gate vs.
attention). The developer-facing gate already captures the defensible,
always-available core ("is this on the developer's critical path") via
run.event; required-check gating would add a usually-blank, contentious cap on top. Left out on purpose rather than shipped hollow.
Joint scenarios for two concurrent checks — observations only
scripts/joint_scenario.py holds a data contract and a pure calculator for
sizing two findings on two different concurrent checks together. It exists
because the per-finding stamps cannot answer that question: with checks at 300s
and 299s, cutting 100s off either alone moves the gate by 1s or 0s while doing
both moves it by 100s, and wall_clock_p50_s is an already-capped merge-wait
saving rather than a post-fix duration, so subtracting the stamps yields 1s.
The collector stamps matched observations only when every check in the whole
confirmed gating set maps to one concrete job execution, every execution has
attempt and runner identity, the checks have no needs: relationship or shared
upstream, and their measured intervals overlap within the scheduling tolerance.
Unknown required sets, matrix jobs and incomplete observations produce no stamp.
The report does not consume the contract yet. The artifact also deliberately
carries an empty effects list because no current finding supplies all four
remaining requirements named by MISSING_PRODUCER_EVIDENCE: per-observation
local reductions, affected-work identity, stable matrix-leg identity, and a
local-runtime-only scheduling/coverage/job-set certificate. Aggregate savings
stamps are not adapted into effects, so there is no supported joint claim.
The methodology is references/wall-clock-methodology.md §8; the tests are
tests/test_joint_sizing.py, which pin the numbers and every rejection code.
The refusal vocabulary is closed: REJECTION_CODES exports it, a rejection
built with a code outside it raises, and the suite asserts the exported set is
exactly the set of codes the module can actually produce.
The next step that would change this status is an effect producer, not more
calculator. Until one qualifies, the measured observations remain data-only.
11. The structural / critical-path track
The hygiene/data-driven catalog (OPT1–OPT69, OPT76, OPT77, OPT79, OPT80, OPT81, OPT82, OPT86, OPT88–OPT91, OPT93, OPT94, OPT96–OPT98 and OPT100–OPT103) is mostly declarative -
static findings are locally-checkable YAML defects, while measured Tier-2 rows
come from run history. Its blind spot: on real repos the merge is
gated by a check that is working as intended and simply slow, with no
catalog match. The old catalog-spine report used to dead-end there ("inherent
cost, outside this catalog"). The structural track (catalog category 14, OPT70–OPT75 plus OPT78, OPT83, OPT84, OPT85 and OPT87,
class: structural) is a second finding class that is not catalog-bound —
it is routed from the measured critical path instead of matched against YAML.
OPT70–OPT75 are routed by the deterministic router described below; OPT78 is
routed by the drill-time vitest-isolate-pool leaf detector (§12.3) and is
reported by scan.py as having no critical-path router. OPT85 is routed by
the drill-time per-invocation-startup leaf. OPT74 and OPT87 have no router
at all: scan.py lists them and nothing emits them. OPT83 is routed by
its own per-pole pass (_detect_opt83_parallel_steps) over each pole's step
decomposition, then checked for independence against the workflow YAML.
OPT84 is routed by its own per-pole pass (_detect_opt84_tool_swap) over the
same decomposition, the workflow YAML and the repo's package.json scripts.
Where it lives
scan.py— unchanged.class: structuralentries are excluded fromstatic_entries, so they never enter the per-file dispatch and never appear incatalog_patterns_without_detector(same treatment asdata-driven).collect_runs.pyowns detection (_detect_structural_candidates, after the hygiene sizing loop so an already-shortened check isn't double-surfaced):_decompose_job_stepsbreaks the long-pole job into per-step p50s and classifies each via_step_category(checkout / install / build / test / scan / package / setup) to find the dominant step + its share + the redundant-work ratio (setup+build ÷ payload). It first reconciles the sampled instances to a single workflow version (_current_version_steps): the sampled window can span a workflow MIGRATION, and aggregating per-step p50s by name across versions would inject phantom steps absent from the audited commit (e.g. a since-removedRun actions-rs/cargo@v1) and crown a non-existent dominant lever. Only step names present in the MOST RECENTLY-started instance (the version the audited tip runs) are aggregated._fetch_required_checksreads…/rulesets+…/branches/{b}/protection(viaGhClient.json(..., allow_missing=True), so an admin-only 404 is "unknown", not a coverage error). Required-status isrequired/not-required/unknown— a 404 is never asserted non-required.- The router picks the best-fit pattern per top critical-path check:
not-required → OPT71 (de-trigger); dominant build/test with high
redundant ratio → OPT72 (prefer the safe cache path); dominant
build/test otherwise → OPT70 (scope — HIGH risk); else → OPT75
(decompose, route by dominant category).
_detect_shared_substepadds OPT73 when a step category recurs across ≥2 cluster jobs.
wall_clock.pyadds two structural sizing helpers (leaf, unit-tested):credit_detrigger(de-triggering a non-required pole drops the wait to the next concurrent check) andcredit_shared_substep(a shared step lowers the whole cluster floor — the one lever that beats the long_pole−floor cap; wall-clock = per-job step time, runner-min = × the jobs). Both feed their RAW estimate into the samesize_wall_clockcascade as every other finding, so structural savings are population-weighted and floor-capped — never a single-PR best case.blocking_path.pyrenders the measured critical path as the report's spine — so a structural lever (the slowest gating check, OPT70–OPT75) is the headline, drilled gate → step-timeline → root cause, rather than a row in a ranked table that a trivial hygiene tweak could top. The structural signals (dominant step/category, redundancy ratio, required-status, shared substep) annotate the pole they came from; the catalog OPT70–OPT75 findings are therefore excluded from the off-path "Also noticed" appendix (_also_noticed_block, which is hygiene OPT1–OPT69/OPT76/OPT77/OPT79/OPT80/OPT82 only; OPT81 renders at its pole or in its own section) since the pole already represents them. Like every pole, a structural lever carries an agent prompt rather than a prescribed fix; for a HIGH-risk lever (e.g. OPT70 scope-to- changed) the prompt's failure-mode/guard section tells the agent to state the failure mode + full-build/full-suite fallback + parallel-run rollout before shipping. The fuller failure-mode/guardrail/rollout profile lives in the catalog body + the findings JSON.- Matrix sibling legs (
_collapsed_sibling_structural).by_matrixrenders only the slowest leg of a matrix as the pole; because_structural_for_poledoes NOT fold a distinct sibling leg AND the appendix excludes per-pole structural levers, a faster leg's own lever would otherwise render nowhere (a silent drop). So the representative pole SURFACES each collapsed leg's lever, carrying that leg's own measured numbers. Two forms (issue #53): a sibling whose lever is identical to the pole's own — same routed pattern id + dominant-step BASE name (the+ N more <category> stepaggregation suffix normalized away by_dominant_step_base) + dominant category (_struct_identity) — folds into ONE compact per-leg measurement line (_collapsed_sibling_line, boilerplate rendered once by the pole's own block); a sibling carrying a different lever keeps its full annotation block (_sibling_structural_annotation). Either way every leg's name + measured p50/share still render, so the anti-drop rule holds — only the duplicated guardrail/rollout/failure-mode boilerplate is dropped. This is presentation-only: the stamped findings are untouched, soverify_report's structural checks (which read findings.json) are unaffected.
- Matrix sibling legs (
Required-scoped spine (the merge-blocking pole)
The report answers "why is the merge slow?", so the spine — and therefore the
headline pole, which is just the slowest thing on it — must be the merge-blocking
checks. The spine is built in collect() from pr_check_p50; immediately after the
non-PR-gating drop (_dropped_non_pr) and before pr_checks_tuple is sorted,
_scope_spine_to_required restricts it to the checks reachable from the required set by
needs:-reachability over the workflow job graph.
The graph itself is repo-agnostic: scan.py's _build_workflow_job_graph emits
workflow_job_graph = {wf_file: {job_id: {name, needs:[job_id], reusable, matrix, timeout_minutes, continue_on_error}}} straight from the YAML (matrix ${{...}}
placeholders left in name for the consumer to regex-match; reusable flags a job with a
uses: reusable-workflow call; matrix flags a strategy.matrix job, whose check-run
names carry an appended (leg…); timeout_minutes records whether the job declares
timeout-minutes; continue_on_error records a LITERAL job-level continue-on-error: true, read through a loader that resolves booleans GitHub's way rather than PyYAML's, so
continue-on-error: yes — a string to GitHub — is not reported as advisory). collect_runs
consumes it:
_required_reachable_checks(candidates, req_names, job_graph, crit_by_wf)— returns the candidates that are required or a job the required work transitivelyneeds:. Each required check maps to a job node (_check_to_job_node). A nested<caller> / <child>name resolves to the editable leaf job while retaining its local caller path; that invocation's outer caller node becomes the anchor. A plain check anchors its file via_map_check_to_jobthen matches the jobnametemplate (exact, then matrix-regex). The reachable set is the anchors plus their downwardneeds:closure (everything an anchor depends on). A nested candidate inherits that reachability only through its caller-prefixed path. A bare check from a workflow's separatepull_requestrun has no such path and keeps its own reachability, even when the same leaf job also runs under a required caller. A candidate is kept iff required, or its node is reachable through the applicable identity, or it's file-backed-but-unpinnable (kept, never silently dropped); a fileless/external non-required check is dropped.- A match-anything matrix
name:binds nothing (no-timing binders). A job whosename:is ENTIRELY a matrix placeholder (${{ matrix.target }}) expands to^.+?$— it matches every check-run name, so it carries no discriminating signal. The no-sampled-timing binders (_check_to_job_node_scanned/_check_to_workflow_file_static, the scanned-graph fallback used when a workflow was triaged out of job-fetching) refuse such a degenerate template (_name_template_is_degenerate): otherwise it grabs a managed/external check-run that appears in no workflow YAML (a NetlifyRedirect rules/ CLA / app check) and mis-anchors it to a workflow file, fabricating a file-backed long pole with a wrong-file agent prompt (live:tokio-rs/tokio). A degenerate-named job's own legs still resolve via the sampled-timing anchor (_map_check_to_job), which a foreign external check — having no sampled job — never reaches; so the refusal costs nothing for real legs and keeps a genuinely fileless check fileless.verify_report._external_check_misbound_offendersre-derives this fromworkflow_job_graphand fails any no-sampled-job pole bound to a workflow that produces no matching job. - Cross-workflow same-name ambiguity bails, never guesses (
_map_check_to_job). A monorepo can declare the SAME job name (Build,test) in several package workflows, so the sampled-timing binder's match tier lands in more than one file. There is no evidence to pick one — the check-runs endpoint carries no workflow path, and a check-run's own started→completed span is queue-inflated so it can't select the "closest" job p50. The binder used to keep the SLOWEST match, mis-attributing the pole'sworkflow_file/ steps / fix to a different workflow (issue #59). It now returns None on genuine cross-workflow ambiguity — the same refuse-don't-guess stance as_check_to_job_node_scanned— and routes to the honest unmapped path. Therequire_developer_timingfilter is applied FIRST as disambiguating evidence: when only one candidate workflow is PR/merge-timed, the ambiguity is resolved and the binder pins that file. Same-name jobs within one workflow (matrix legs) are not a file-attribution ambiguity and still resolve to the slowest, so every unambiguous repo is byte-identical. Downstream stays honest:_is_pr_gate_checkdecides PR-gating from the FULL matching-workflow set (_workflows_matching_check), never the single slowest pick, and the structural disclosure distinguishes an ambiguous file-backed check (_check_producing_workflows> 1 → "produced by a same-named job in more than one workflow; give them distinct names to drill") from a genuinely fileless bot/app check — a real in-repo check is never called "third-party". The bail is a FILE-attribution bail only: an ambiguous check that is the real merge gate must still crown the headline. Its crowning MAGNITUDE is unambiguous (a PR waits on the slowest same-named job), so the spine grounds it on that job p50 (_check_grounded_job_p50, never the queue-inflated check-run span) and stamps itworkflow_jobs, keeping it in the crowning basis;_partition_fileless_checkstherefore never mis-drops a file-backed ambiguous gate into the fileless / PR-lifetime-latency bucket. Only its per-file step drill/fix stays withheld. - Reach, not file co-residence.
needs:-reachability handles both real shapes a name-prefix or file matcher gets wrong: a bare aggregator (better-auth'sci/e2e@~3s, whichneeds:the real slow jobs that share no name prefix — kept), and an independent sibling (a slow non-required job sharing a required workflow file that no required jobneeds:— dropped, where file-membership would wrongly keep it). - Gated hard. The filter only fires when
required_checksis non-None,complete, non-empty, and the required suite was satisfiable on the sample. A partial read leaves an absent check UNKNOWN (not not-required) — dropping it could discard a check an unread ruleset gates; an unsatisfiable suite means the required names aren't in the sample at all (the PR-floor fallback renders), so filtering would empty the spine. It also never empties the spine on an over-tight match (returns the input unchanged). With no usable graph it degrades to literal required membership (the conservative subset). - Satisfiability is computed over OBSERVABLE required checks. A required check that
never appears as a check-run anywhere in the sampled window is a status-only/external
gate (a
Devin ReviewGitHub-App commit status, an enterprise status check) that can't be sampled — so the per-PR suite test (req_names ⊆ checks) excludes it and re-selects on the observed required subset (_select_repr_shas), recording the excluded names inrequired_checks_unobservable. Without this, one unobservable required status empties the suite and tips an otherwise-normal repo into the PR-floor fallback (live:triggerdotdev/trigger.dev, whose requiredAll PR Checksaggregator gates every PR but whose requiredDevin Reviewstatus is never a check-run). A fully-external suite (nothing observable) still falls back to the PR-floor. - A status the repo posts itself is not an outside gate (issue #119). scan.py records
workflow_status_publishers: each workflow whose YAML, local action, or repo-local script posts a commit status (arepos/…/statuses/<sha>call,createCommitStatus, a known status action), with the literal contexts those sources post (contexts). Only when a context is built at run time does it keep the sources' string literals (names) as a looser fallback; no raw source text is stored. Local-action entrypoints resolve from the action's folder, and scripts are followed transitively (bounded). A required context equal to one of those (_repo_published_required) is an in-repo aggregate that may wait on any job with noneeds:edge. A name that merely appears in a publisher (its title, a comment, the checks an aggregator waits on) does not count: that would un-scope a correctly scoped gate. The sampler never calls the suite unsatisfiable on its account (required_checks_repo_published),_required_suite_all_externalreturns False, and when it names no job (_required_reachability_unresolved) the scoper stays inert and the structural stamp saysunknown, nevernot-required. The pole is anunresolvedbest guess; which jobs the publisher waits on is not traced. - No silent drop. Excluded checks are recorded on
cp["dropped_non_required_checks"](sibling todropped_non_pr_checks) AND surfaced as a report footnote ("Narrowed to merge-blocking checks: …"); a dropped check loses its structural pole/findings but the exclusion is visible in both the JSON and the rendered report. - Stamp agreement. The
required_statusstamp in_detect_structural_candidatesuses the same reachability (_required_reachable_checksover the scoped checks), so a kept leg (e.g. a shardedTestjob feeding a requiredMerge Test Reportsrollup) is stampedrequired, never offered an OPT71 de-trigger that would tell the user to drop a job that gates the merge.
Filtering pr_check_p50 before the tuple propagates consistently to
critical_path_check, checks, poles, the drilled matrices, populations, structural
routing, and the renderer — no renderer change needed. Live-confirmed on mastra-ai/mastra:
without it, the non-required changed-tests (its own changed-test-gate.yml) headlined the
report even though it gates no merge; an early file-membership matcher then mis-kept the
non-required Validate build outputs (a lint.yml sibling that required Lint doesn't
needs:), which needs:-reachability correctly drops.
Rare-pole demotion — by actual-critical-path FREQUENCY, not presence (when no required set
scopes the spine). The required filter is inert on a repo whose required set is
unreadable/None (no admin) — there the spine is the raw PR-floor, and a raw-p50 sort lets a
check that a typical PR never actually waits on headline. So the pr_checks_tuple sort is
two-tier by POLE FREQUENCY: a check that is the ACTUAL critical path (the slowest job a PR
ran) on fewer than _POLE_RECUR_FLOOR (2) sampled PRs is a one-path outlier and ranks below
the genuine recurring gates, each tier by p50 desc. Pole frequency is
_pole_frequencies(per_sha_checks, set(pr_check_p50), caps) — the per-PR argmax over the SAME
job-p50-capped spans populations uses (_pole_caps), so an inflatable check-run span (queue
/ re-run time) can't crown a light check and the headline can never disagree with the
populations verify_report re-derives from; co-slowest ties credit every tied check. Each
check carries its count as pole_n (and present_on for disclosure). A required check is
exempt (gates by definition); inert below _RARE_PRESENCE_MIN_PR (6) PRs.
Why frequency and NOT presence (the expo/expo class this replaced): the old present > half
cutoff demoted every check present on ≤ 50% of PRs. On a path-partitioned monorepo each heavy
suite runs on a MINORITY of PRs (a PR touches only part of the tree), so that cutoff demoted
the entire heavy-job set at once and crowned a lightweight always-present check-packages
that — by the findings' own populations — was the actual slowest job on 0 of 20 sampled
PRs, burying the genuine 20-minute native/CLI suites. Ranking by how often a check is the real
critical path surfaces the recurring gates a developer actually waits on. Because it reorders
the same pr_checks_tuple, the headline, poles, structural routing (_is_typical_check uses
the same pole_n >= _POLE_RECUR_FLOOR), and the run.py data-pass summary all agree.
verify_report.check_headline_pole_actually_gates re-derives the per-PR pole from
populations and FAILs a headline that gates < floor PRs while a genuine gate was passed over
— the class-detector that makes this a class fix (it independently flagged a second victim,
Photoroom, across the dogfood corpus). The renderer applies the matching split and labels the
demoted pole by why it was demoted — reading pole_n, "Rarely the merge gate — the actual
slowest check a PR waits on, on only N/npop sampled PRs" — falling back to the legacy
present_on presence wording ("opt-in / rare — ran on only N/M sampled PRs") only for
pre-pole_n findings. Live-confirmed on expo/expo
(headline check-packages 2.9m/pole-on-0 → the real ios-test-e2e 26m + playwright-windows
23m) and paradedb/paradedb (a label-gated benchmark ~153m on 1/20 PRs demoted below the real
~23m test gate).
Demoted-pole framing depends on whether the spine is required-scoped. The rank demotion
above is pole-frequency-based and applies on both a PR-floor spine and a required-scoped one —
but the label must not. On a PR-floor / recency spine a demoted minority check really can be a
non-required opt-in job, so the renderer keeps the "opt-in / throughput, not merge-wait"
framing. On a required-scoped spine the data layer has filtered to required-reachable
checks, so a demoted minority leg is a required gate that's merely path-conditional (it
gates the PRs whose paths trigger it, where it IS the merge pole) — the renderer reframes all
three label sites (headline, "Also slower" note, per-pole role) as "required ·
path-conditional — ran on N/M PRs". The flag it keys off is spine_required_scoped, set
only when _scope_spine_to_required ACTUALLY narrowed the spine — not the sampling-level
required_suite_scoped, which is True whenever a required set was satisfiable even on a
partial / anchorless read where the spine was left unscoped and still holds non-required
checks (relabeling there would re-introduce the very mislabel this guards against). Where a
required-scoped spine mixes a path-conditional required leg with an external review check, the
"Also slower" tail addresses each on its own terms. Live-confirmed on
triggerdotdev/trigger.dev: internal unit tests (the slowest leg, 6m51s, 9/20 PRs) is
path-filtered to internal-packages/** changes — required, not opt-in.
The risk axis (surfaced in the pole's prompt, never demoting the rank)
Structural changes can degrade correctness, not just performance, so every
structural finding carries risk + a mandatory guardrail + rollout +
failure_mode — built by construction from _STRUCTURAL_META (in
collect_runs.py), so the risk axis can never be silently dropped. Risk does
NOT demote the rank (that would bury the biggest speed win — the whole reason
the measured critical path is the spine); instead it is made loud where the pole
renders: the per-pole agent prompt's failure-mode/guard section spells out the
failure mode + full-build/full-suite fallback + parallel-run rollout, so a
HIGH-risk lever (e.g. OPT70 scope-to-changed) is never handed over as a safe
quick win. (The pole's 🔴/🟠/🟡 dot is a wall-clock duration tier, not the risk
rating.) The catalog body for each structural pattern is the long-form risk
profile; _STRUCTURAL_META is its structured form.
OPT81's advisory half rides at the END of a pole, not in this track. It is
not a structural finding (it carries no OPT70-75 id and joins no
_structural_for_pole), but it is routed off the same measured pole and only
after this track has run: OPT70-74 and OPT78 on the pole suppress it by being
present. The three uncredited exceptions are OPT75, OPT83 and OPT84: their
presence does not suppress it (all render first, OPT83 and OPT84 because each
needs a benchmark before any saving is known; neither is in
_OPT81_CHEAPER_STRUCTURAL), though an OPT75 credit of half the pole's median
or more does, like any credited finding. It renders after the pole's own prompt,
its OPT75 block, any OPT83 card and any OPT84 card. See §5.4.
Parallel steps: the YAML side is walked, the run-data side is not (yet)
GitHub Actions parallel steps (2026-06-25) let a step be a - parallel: group
(a list of ordinary child steps run concurrently, then an implicit wait), let a
step carry background: true, and add control steps (wait:, wait-all:,
cancel:) that run nothing.
- YAML side — fixed.
scripts/workflow_steps.pyis the one reader of a job'ssteps:list. It returns every leaf step in declaration order with each group's children spliced in (nested groups too), skips-but-counts control steps (keeping a control step'sname:), and tags each leafin_parallel_group/background. Every static reader goes through it:scan.py's_steps/_steps_with_if/job_walk(so every per-job detector, OPT28's checkout loop, OPT17's sleep loop,_job_needs_git_historyand_index_local_git_actions), and incollect_runs.pythe OPT77 setup fingerprint, OPT79's cache block and itspackage.jsonprobe, OPT80's checkout and retry readers, OPT82's lint-step read, the OPT24 shard check and OPT83's candidate-run and independence reads. - Malformed and invalid groups. A group is MALFORMED, its steps not read,
when its
parallel:value is not a list, the list contains itself through a YAML alias, it is nested more than 64 groups deep, the walk has already readWALK_MAX_NODES(10,000) leaves and groups (a list reused through aliases is read again at each use, so sibling groups aliasing one list double per level), or it holds an item that is not a step. Each records its kind. The file lands inscan_incompletewith the kind named, and every finding of a detector whose claim is an ABSENCE (OPT1, OPT2, OPT5, OPT14, OPT21, OPT28, OPT29, OPT31, OPT39, OPT76) is held back for that job, listed on its gap record asjob_has_an_unreadable_parallel_group; OPT79 and OPT80 return the same reason rather than "no steps" / "no checkout". A group on a step that also hasrun:/uses:is INVALID (GitHub rejects it): its command is a leaf and its children are read as the group's children (tagged in-group), so order-free verdicts such as the git-history check match a flat read. It is not a coverage gap; the Parallel steps row names it. - Order-aware readers use the tags. OPT2 fails closed: any cache declared
before the install counts as preceding it, background included; only a
sibling in the install's own group is excluded. OPT5 says two setup actions
in one group run side by side. OPT12's preamble ends at the first group, and
a job whose first step is in a group is listed under
parallel_steps.held_back(first_step_inside_a_parallel_group). OPT79 holds back a cache whose restore or install runs in a group or in the background, or that shares a group with its install (the overlap reason is named before any install-shape reason). - Provenance. When a repo uses the syntax (groups, control steps or
background steps),
scan.pystampsparallel_steps: the counts (groups,steps_in_groups,control_steps,background_steps),malformed_groups/_files/_jobs/_reasons,invalid_groups/_files/_jobs, the job listsjobs_with_groups,jobs_with_backgroundand (in a file holding either)sequential_jobs, andheld_back. Otherwise the key is absent and the scan output is byte-identical to before. The Data sources table renders a "Parallel steps" row from it (workflow_steps.parallel_steps_disclosurefor the cell,workflow_steps.parallel_steps_used_forfor its "Used for" text, which names groups,background: truesteps or both, from the same counts). - Run-data side — NOT changed.
_decompose_job_stepsstill sums per-step p50s into the job p50, and_step_timelinestill lays steps out end to end. With overlapping steps both are wrong (the sum over-counts, the timeline draws concurrent steps in sequence), but how the jobs API records a parallel child, a background step or the implicit group wait is not documented, so it needs a live probe before the timing model changes. Until then the guards are:- a credited lever priced from step medians (OPT24 sharding, the structural
OPT70/72/75 route) is HELD BACK when the step it is priced from runs in a
group or in the background, or the job cannot be pinned to its YAML in a
workflow that uses the syntax (
dominant_step_runs_inside_a_parallel_group, the "parallel steps: held back" row); the pole keeps the renderer's generic dominant-step hand-off; - the pole drill never says a job's steps run one after another when the stamp
names it as overlapping, and says they "may" overlap when the pole's
workflow holds an overlapping job it cannot match to one (
_pole_steps_overlap: job key, display name, matrix leg, a${{ }}-templated name, a reusable-workflow callee's/ <job>tail; never across workflow files; a pole with no known workflow file only ever gets the hedged wording); - a control step is never crowned the dominant step. A control step's duration
is time spent blocked on other steps. The bare names
wait,wait-all,canceland a name starting "Wait for all background steps" are an assumption about how run data labels control steps and the implicit group wait (no GitHub source states them); a control step the author NAMED is read from the repo's YAML and excluded by that name. Both the collector's crown and the renderer's fallbacks skip them; a work step that merely starts "Wait for" stays eligible.
- a credited lever priced from step medians (OPT24 sharding, the structural
OPT70/72/75 route) is HELD BACK when the step it is priced from runs in a
group or in the background, or the job cannot be pinned to its YAML in a
workflow that uses the syntax (
- Known limits. Apart from the order-aware readers above and OPT83 (which
asks whether steps could run at once), static detectors read a group's
children in declaration order. OPT77's setup fingerprint sums setup seconds that may
overlap in a group, so it can overstate. An unnamed
wait: servermay show up in run data under a generated name such as "Wait for server"; that is unprobed, so it can still be picked as dominant. A user step literally namedWait,Cancelorwait-allis excluded from dominant-step picking. Timing (step-sum p50, timeline) is unchanged and needs the live probe. - Group-level keys are an assumption. GitHub does not document
if:,continue-on-errorortimeout-minuteson aparallel:group. The walker treats a group-levelif:as inherited by its children — an ASSUMPTION, marked unverified — so an install in a gated group is not reported as unconditional, and OPT76 searches thatif:for a payload path. - Other engines. The leaf / group / control / malformed / invalid semantics
follow ci-score's
_walk_stepsfrom PR #120, and ci-secure's walker from PR #121 uses a depth cap of the same number in different units (it counts every YAML node with> 64; this walker countsparallel:nesting with>= 64). Neither is onmainyet, so the parity this section relies on is pinned only by the repo-root git-history parity test. - Independent check.
verify_reportre-derives the whole Parallel steps row on its own (its own copy of the malformed-kind phrases, pinned equal), fails a row with no stamp, and fails a pole whosefile ▸ checkheading the stamp names as overlapping when its drill says the steps run one after another; it imports nothing from the skill.
OPT83: independent steps on the long pole, from the step decomposition plus the workflow YAML
OPT83 is the structural pattern for a drilled pole whose dominant cost is
several independent compute steps run one after another. collect() runs
_detect_opt83_parallel_steps once per drilled pole in pr_critical_path.poles,
right after the structural router and before OPT81's A2 advisory.
Each call is guarded: an exception skips OPT83 for that pole only, and is
disclosed through detectors_skipped under its workflow with the pole's job
named. It makes no gh call of its own; it reads
the pole's stamped per-step p50 decomposition (steps, job_p50_s) and the
workflow file. A step is a candidate when its category is compute (build, test,
scan or package per _step_category), its name is not boilerplate and does not
contain the word sleep, wait, wait-all or cancel, and it measures at least 20s
(_OPT83_MIN_STEP_S) and at least 15% (_OPT83_MIN_SHARE) of the pole's
decomposed p50. Two or more candidates that sit directly next to each other in
the job's top-level steps: list form a run (steps are read through the shared
workflow_steps.job_walk); any other step, a parallel: group, or a wait /
wait-all / cancel control step between them breaks it, and such a pole is a
verdict, counted only. The top-level list is read only to see control steps
between candidates: walk.leaves drops them, and a control step is an order
barrier. Steps already inside a parallel: group or marked background: true
are not candidates. When several runs qualify, the largest-bound run is the only
one checked; if it fails independence the pole is held back, and the held-back
row names that run's steps.
Independence is read from the workflow YAML, never from timings, and it fails closed. The pole is held back when any of these holds:
- the step timings or the workflow file cannot be read, or a step record is unreadable (not an object, no name, no category, or one name twice);
- the pole's check is produced by more than one workflow file
(
ambiguous_workflows); a check no workflow produces stays a verdict; - the pole or a qualifying step is not matched to exactly one YAML job or step;
- a candidate, or any step before the last one, sets
background:with a${{ }}expression; - two candidates share a cache or artifact; a candidate is an action whose
effects are not read; a candidate publishes, deploys or uploads (measured
category
package, or such a word or command in its name or command); - one candidate reads another's step outputs, or a non-final candidate writes GITHUB_ENV, GITHUB_OUTPUT or GITHUB_PATH;
- a
${{ }}sits anywhere in a candidate but its name, or in an inheriteddefaults.run.working-directory(checked on the raw value, before it is normalized); - a candidate changes directory;
- a candidate that may write build output shares, or nests with, another
candidate's working directory, in either order; a working directory outside
the checkout overlaps every tree, and with a build present any step aimed
outside its own tree (
--prefix,--cwd,--dir,-C,--outDir, a../or absolute path) shares every tree; - a candidate installs dependencies (any line, or any
&&/;/||/|part of a line); - a candidate carries a step-level
if:or setscontinue-on-error(a step allowed to fail changes what a failing parallel child does to the job); - the steps share tool state:
services:containers, two Docker-driving steps, any Lighthouse-running candidate (an audit shares the runner's cores with its siblings, which skews its score), or two candidates in one tree running the same Cargo, Maven, Gradle, .NET, sbt, Swift, Mix or Xcode build, or both running a recognised coverage tool; - a Docker / Podman / Compose command or container action sits in any step up to the last candidate, or a background step runs before the first;
- the job holds a malformed
parallel:group (job_has_an_unreadable_parallel_group, the scan's own reason): that group's steps were never read, so independence cannot be shown, whatever the candidates' own run looks like.
OPT83 does NOT hold back OPT81's runner-size advisory on the same pole: it is
uncredited and needs a benchmark, so the reader sees both, the OPT83 card first.
Held-back poles are counted in opt83_withheld_by_gate, listed in
opt83_withheld_candidates, and shown in the "independent steps: held back"
Data sources row from the shared blocking_path._WITHHELD_ROWS registry, with plain
reasons from _OPT83_WITHHOLD_PHRASES. That label is distinct from the
"parallel steps: held back" row, which lists credited levers (OPT24, the
OPT70/72/75 route) priced from step times that overlap: one row per doc key, so
a reader never sees one label counting two different things. The verdict gates (pole not file-backed,
fewer than two qualifying steps, steps not adjacent, steps already parallel) are
counted only. The parallel-steps reader here is the shared workflow_steps.py
described above; no second YAML walker was added.
The finding is deliberately uncredited: no _SIZING key, wall_clock_p50_s and
runner_min_saving are None, sizing_basis is uncredited, and the runner-minute
door cannot derive it. It stamps ceiling_s (sum of the candidates' p50s minus
the largest), labelled an upper bound that assumes each step keeps its solo speed
on one runner. It never enters a total, Tier 2, the headline or Also noticed; the
card tells the reader to benchmark first. The renderer places one anchored card
(opt83-<id>) inside the pole's section after the pole's agent prompt (so after
any OPT75 block) and before any OPT81 card, an aggregation-gate pole included; a
finding whose pole is not rendered gets its own "Independent steps on other long
poles" section.
verify_report.check_opt83_parallel_steps re-checks each listed step against
the candidate rule (a verbatim copy), re-derives the upper bound, rejects any
other duration, and fails: a saving, a sizing basis other than uncredited, a
ceiling that is not sum minus max, a missing or duplicated card, a card without
the upper bound, its label, the no-weakening rail and the runner caveat, a card
carrying a duration in any time unit other than the candidate steps' p50s, the
pole's p50 and the upper bound, or any percentage, a step listing that differs
from the finding's, an orphan card, a card outside its pole's section when the
pole is rendered (or anywhere but the "other long poles" section when it is
not), a finding anchor named outside its own section, a rendered card whose
findings file cannot be read, and a held-back counter that disagrees, gate by
gate, with the held-back list. The phrase table is mirrored in verify_report
and pinned like its siblings.
OPT84: a slow step on the long pole runs a tool that has a native drop-in
OPT84 is the structural pattern for a drilled pole whose heavy step runs a
JavaScript dev tool with a native drop-in (Prettier → oxfmt, ESLint → oxlint,
tsc --noEmit → TypeScript 7). _detect_opt84_tool_swap runs once per drilled
pole beside OPT83, after OPT82 has run on the same workflow, and makes no gh
call of its own. It reads the pole's stamped step decomposition (steps,
job_p50_s), the workflow file, and the scan's type_aware_lint block, which
carries the package.json scripts per directory, the tool versions found in
each manifest, and the ESLint config facts. A step is a candidate when it
measures at least 20s (OPT83's absolute floor; unlike OPT83 there is no
share-of-job floor, _OPT84_MIN_SHARE = 0.0, so a 29s lint on a 318s gate
still gets a card that states its share), its category is scan or build, and it is a run: step. Its
command is split into leaf commands, following package-manager script calls
through package.json up to five levels deep (OPT82's resolver, shared) and
unwrapping runner calls; the leaves are matched against the v1 rows of the
catalog's swap table (Prettier, ESLint, tsc --noEmit; Babel and Webpack are
recipe only).
The table's install row (npm / pnpm / Yarn install → bun install) is the one
row with a share floor: an install-category step is a candidate, for that row
only, when it is also at least _OPT84_INSTALL_MIN_SHARE (0.15, OPT83's floor)
of the pole, and an install inside a scan / build step reports the row only
when that step clears the same floor. Its verdict comes from the scan's
package-manager reads (scan._tal_package_manager_read: lockfiles,
packageManager, pnpm-workspace.yaml, .npmrc, pnpm hook files, Yarn's
nodeLinker and Plug'n'Play loader) of the install's directory and every
directory above it: package_manager for the root, package_manager_dirs for
each other directory with a package.json. _opt84_install_layout merges them
nearest first (the nearest lockfile, packageManager and nodeLinker win, as
Yarn's own config does), and _opt84_install_fit decides: already on bun is
target_already_in_use; Yarn Plug'n'Play or a pnpm hook is the
install_swap_blocked verdict; a missing or unreadable read drops the install
row, and holds the step back (install_layout_unreadable) only when nothing
else on it reports. On an install step, an exit the read cannot follow is the
tool_not_in_table verdict unless the step's run: names an install
(_OPT84_INSTALL_HINT_RE). Otherwise the row is always a
dual-run whose reasons are the caveats to check, and the card adds the
side-by-side install recipe (_OPT84_INSTALL_RECIPE, pinned against the
verifier's copy).
Verdicts, counted only: a pole not backed by a workflow file
(pole_not_file_backed), no candidate (checked first; a pole whose check maps
to more than one workflow carries no measured steps today, so it reaches the
held-back path), no table tool on the step, the target already in use (the
same step runs oxfmt / oxlint, eslint-plugin-oxlint or a declared
oxlint / oxfmt sits on the step's manifest walk, or typescript 7+ is
declared), a tsc that emits, or a step whose only table tool is an ESLint run
OPT82 already reports (OPT82 is the finding; other table tools on that step
still qualify, and the card shows that ESLint row as "see OPT82 first").
Generic runners (make, just, pre-commit, concurrently, ng, gulp,
docker, …) and actions are a verdict only in a repo with no JavaScript
manifest, where a direct leaf such as npx prettier --check . still fires; in
a repo with one, a generic runner is held back, and so is any action outside
the known allow-list (checkout, setup-*, cache, CodeQL, Docker and similar).
An unreadable manifest keeps the held-back path.
Held back, failing closed: unreadable step timings or workflow file, a check
produced by more than one workflow, a pole or step not matched to exactly one
YAML job or step, a ${{ }} in the command, working directory, env: or
shell:, a directory change, a script the resolver cannot follow (a missing
or unfollowable npm run target, too deep, unreadable package.json, or an
unresolved delegator such as turbo run, nx or node ./script/x.js; a
task-runner expansion is trusted only when the manifest walk was complete), a
generic runner or unknown action in a JavaScript repo (above), a uses: step
whose uses: or name: says it may lint, format or type-check
(_opt84_action_may_swap), an ESLint config that could not be fully read, a
scan with no ESLint config read or whose config reader failed
(type_aware_lint_scan_missing, type_aware_config_reader_failed, the two
scan gates shared with OPT82), a package.json or Prettier config on the
Prettier step's walk that could not be read (the Prettier fit fails closed;
a Prettier config file the audit does not open, such as .prettierrc.js or
prettier.config.*, makes the fit "run both" instead), a typescript
version the audit cannot pin down (no package.json declares it, unreadable,
a >= range, or a walk that was truncated, never reached the step's
directory or hit an unreadable manifest), or a working directory outside the
checkout. Held-back poles are counted in
opt84_withheld_by_gate, listed in opt84_withheld_candidates, and shown in
the "tool swap: held back" Data sources row from the shared _WITHHELD_ROWS
registry, with plain reasons from _OPT84_WITHHOLD_PHRASES.
The finding is deliberately uncredited, like OPT83: no _SIZING key, no
wall-clock or runner-minute saving, sizing_basis is uncredited, and the
runner-minute door cannot derive it. Its only number is ceiling_s, the
candidate step's measured p50: the most the swap can remove, not a forecast.
The swap table's published speedup ranges are quoted as the catalog's, never as
the repo's. When several table tools share one step there is one card and one
ceiling, and the card says the split is unknown until measured; several
leaves of one tool in a step are one tool row. Risk is MEDIUM, and HIGH
whenever the card reports an ESLint row outright (drop-in or dual-run; a run
deferred to OPT82 does not count). The ESLint fit is dual-run whenever a config
carries custom local rules, the tailwindcss plugin, a framework plugin or any
other plugin oxlint does not carry natively, a react-hooks/exhaustive-deps
rule, type-aware parsing that is on or unresolved (unless OPT82 already reports
the step, which defers it), facts the scan did not read, or a shareable config
or config import it did not follow (fail closed), and whenever the command
passes --rulesdir, --plugin, --rule, --parser or --ext. The
Prettier fit is dual-run when a prettier-plugin-* is declared on the walk, a
Prettier config it reads lists plugins, or a Prettier config file it does
not open applies. The card renders once in the pole's section,
after any OPT83 card and before any OPT81 card, and never in a total, Tier 2,
the headline or Also noticed. It carries no prompt block: each tool row says
the target is the native replacement with its fit ("drop-in", "run both: …"
or "see OPT82 first"), and links the target's own docs (the oxfmt and oxlint
migration guides plus their overview pages, _OPT84_DOCS; the TypeScript 7
announcement, as that row has no migration guide). It carries the no-weakening rail and a
benchmark line that tells the reader to give the target the repo's migrated
config first (until then the target runs on its defaults) and then time both
tools on the same runner class. verify_report.check_opt84_tool_swap
re-derives, from the drilled pole's stamped steps, each card's step p50,
ceiling, job p50 and share, and checks the step qualifies; it checks the tool
rows (each table row named once, a legal fit, the table's docs links and
published range, an ESLint deferral only where OPT82 reports that step), the
severity against the engine's rule, the split flag, that no saving is carried,
and that exactly one card renders in the right place. It also pins the
held-back counts against the listed entries, like its siblings. It also
re-derives each tool row's fit from the config facts stamped on the finding
(plugins, custom rules, type-aware state, unopened Prettier config files) and
fails a fit those facts do not support. It does not re-resolve the command or
re-read the scripts.
12. The blocking-path report (blocking_path.py)
blocking_path.py is the skill's single, data-first renderer (§2) — answering
one question, "why is the merge slow?", as a drill-down from the gating check
down to the root cause. (It replaced an earlier catalog-spine renderer, report.py,
when the measured critical path became the report's spine.) Like the rest of the skill
(§2) it stops at the measured root cause and does not prescribe the fix: each
pole ends with a prompt that restates the root cause and points the user's
coding agent at the tool's official docs (_PROMPTS, keyed by fix_key) — no
recommendation, no code diff. It reads the same findings.json plus an optional
data bundle (logs + timelines + magnitude samples) that collect_runs captures
under --with-logs --data-dir. CLI: --in findings.json + per-pole
--log/--steps/--mag KEY=PATH (KEY = a substring of the workflow filename) +
--captured-at. With no bundle it still renders (level-1 + P50 step bars).
12.0 The Long pole map — the cascade up top (presentation-only)
Immediately after the Contents the report renders a ## 🗺️ Long pole map: the whole
blocker cascade in one text fence, so the reader sees the shape of the wait before the
per-pole drills. It is checks-first (the physically true model — GitHub gates on
CHECKS, which all race concurrently; a workflows-first grouping would hide a runner-up
blocker living in a different file):
- Level 1 — the flat race of merge-gating checks (
src[:9], the same typical-PR set the Contents draws — no new data source), each bar labelled{check} · {workflow file}(the file from the check's ownworkflow_file, else a same-named pole's; a fileless / external check renders with no suffix). The ◀┐ hangs from the row of the check the cascade descends into — the check of pole 1 (pole_wfs[0], "Long pole 1" below). - Level 2 — that check's stamped steps (
pole["steps"], p50-desc, top 6, no roll-up row — the map is orientation; the pole section keeps the full accounting). Each step's % is its share of the check's own p50 (the job wall level 1 just showed), via_emit_level'spct_denom. The ◀ lands on the dominant-category lead (_dom_lead_idx, agreeing with the crown), falling back to row 0. - Level 3 — the dominant step's internals, rendered only when the descent pole's
parsed-log leaf has a
deeperfirst level of ≥2 rows (the same_parse_log → _apply_cache_dist → _demote_offcategory_leafpipeline the per-pole loop runs — hoisted into_derive_pole_leafand shared via_pole_leavesso the map's L3 and the pole's L3 can never disagree). Onlydeeper[0]is drawn (itsblocker_notewhendeeperis a single level).
The whole cascade reuses the pole-drill _emit_level machinery (mark_idx + header_below
connector idiom). Degenerate levels collapse — never a one-bar level; level numbering
stays semantic (checks=1, steps=2, internals=3, matching the pole sections). If level 1 is
too thin (<2 rows) but level 2 qualifies, the fence opens at level 2; if the descent pole's
check isn't among the shown rows, pole_wfs is empty, or the pole has no usable steps, it
falls back to the level-1-only render (the pre-cascade #75 form); if nothing qualifies
(one check, one step), the section is skipped entirely (exactly where the if src: guard
skips it). The map is presentation-only: no claims-layer cs.add(...), no verify_report
check binds to any map line — the Contents list + the stamped fields stay the single source
of truth.
12.1 Three levels, three provenance scopes
| Level | What | Source | Aggregation |
|---|---|---|---|
| L1 — concurrent checks (jobs) | the checks a PR waits on, slowest first; top = the gate | pr_critical_path in findings.json |
P50 across sampled PRs |
| L2 — the gating job's steps | a sequential timeline of the steps | one representative run's real per-step start/end (--steps) |
one run (absolute) |
| L3+ — the dominant step's internals | per-file / per-suite / cache split → the root cause | that same run's raw log, parsed by the leaf detectors (--log) |
one run (absolute) |
The split is deliberate: L1 jobs run concurrently (bars are left-aligned —
they all start together; you wait for the slowest), whereas a job's steps run in
succession, so L2 is an offset Gantt (░ = elapsed-before-start, █ = the
step running) rather than left-aligned bars. Drawing steps left-aligned implied
they overlap, which is backwards — the timeline fixes that and the steps sum exactly
to the one run's wall time.
12.1a The render-layer single door — the headline consumes the stamped ceilings (issue #49)
The Bottom-line / headline lever is selected and sized from the stamped per-finding
wall-clock ceilings (findings[].wall_clock_p50_s), never re-derived. This is the
render-layer analogue of §5.1's sizing door: the data layer already ran the whole
cascade (§5) and stamped one authoritative ceiling per finding, so the renderer must
consume it rather than compute a second, divergent number.
The subtle case is the cluster-floor lever (OPT73). Its fix cuts a step shared
across concurrent matrix sibling legs, so it drops the WHOLE cluster in lockstep and
saves its full stamped wall_clock_p50_s of merge wait. The per-pole
_pole_addressable / chain-headroom arithmetic can't see that — it floors at the next
sibling leg, which descends with the fix — so before #49 it headlined a tiny
per-leg number (mastodon: ~36s over a stamped 627s; electron: ~5m37s over 2635s) while
the real lever sat in "Also noticed". The renderer now, at the Bottom-line site,
compares the win the branch cascade would show against the biggest credited cluster
ceiling (_headline_cluster_lever); when the ceiling wins, the Bottom line LEADS with
it (_cluster_headline_bottom_line, naming the shared step + cluster and pointing at
the OPT73 appendix entry). The contract:
- Keyed on the PERSISTED marker.
collect_runsstampscluster_floor_lever=Trueon every OPT73 cluster construction; the renderer selects only findings carrying that marker AND a positivewall_clock_p50_s(a credited developer-facing win, not a bill-only / off-path zero). A legacy artifact predating the stamp has no marker, so the branch never fires and the report renders byte-identically — the chain "up to X" sentence keeps its legitimate uses (a real chain with no dominating cluster lever). - Re-derives the SAME selection. The renderer never recomputes headroom; it reads the ceiling the data layer sized. The two layers cannot disagree because there is one number.
- Honest about concurrency. The "drops the whole cluster in lockstep" framing is
true only for a concurrent matrix cluster. A
needs:-chained cluster runs SEQUENTIALLY, socollect_runsalso persistscluster_legs_concurrent, and the Bottom line phrases a sequential cluster as "N sequential (needs:-chained) stages … the per-stage savings compound on the critical path" — never "concurrent … in lockstep" (the same mislabel §12.1a's appendix already avoids). Missing marker defaults to concurrent (the matrix-leg case). - Guarded both ways.
verify_report.check_headline_consumes_stamped_cluster_ceilingre-derives the max credited cluster ceiling from findings.json and FAILs when the rendered Bottom-line lever is strictly smaller (the burial). With no persisted marker it SKIPs LOUDLY (a legacy artifact it cannot verify) rather than reading clean — the L8 fail-open that let this defect survive PR #47.
12.1b The cluster crown is presence-weighted — a minority workflow can't headline (issue #56)
§12.1a stops a real cluster ceiling from being under-credited; this stops the converse — a
minority-present cluster over-crowning the typical-PR headline. The OPT73 detector originally
chose its anchor (the leg whose p50 sizes the win and leads the Evidence) by absolute slowest p50,
while the merge-wait spine is presence-weighted (§11): a check that is the actual pole on < _POLE_RECUR_FLOOR sampled PRs is a one-path outlier the ranking demotes. The two disagreed on
microsoft/playwright: OPT73 anchored on Test msedge-dev on macos-latest (the pole on ~0/20 PRs, in
NO Level-1 pole) and let its tests_secondary.yml cluster — a workflow that gates only 2/20 PRs —
crown the Bottom line's "biggest single measured win (~3m 15s)" and self-label "sits ON the merge-
gating critical path", above the honest ceiling (pole 1's own ~2m 37s on the 13/20 majority workflow).
The rule composes with the existing rare-demotion (§12.2 _ms_freq_demoted) and re-uses the spine's
own predicate (never a parallel notion — the L3/L5 mirror-the-engine discipline):
- Anchor (both cases).
collect_runs._detect_shared_substeporders the cluster legs presence-eligible first (_leg_presence_eligible, the inverse of the spine'sis_rare— the exact complement for any spine-ranked leg, with an unknown-to-the-map leg treated as eligible rather than rare, so we never exclude on partial info: required, or the actual pole on>= _POLE_RECUR_FLOORPRs, or unknown/small-sample), then by p50. Soaffected_jobs[0]— the Evidence lead and the appendix**Where:**lead — is never a minority-present leg. When every leg is eligible (the common matrix case, every majority cluster with no rare leg — mastodontest-ruby, the electronbuildchain) the order is byte-identical to before. - Crown / on-path label — follow the data. A majority workflow with one minority leg (option
a) keeps its wall-clock and simply re-anchors — its cluster genuinely gates the typical PR. A
minority workflow (option b,
_workflow_gates_minority: its summed gate count is<= halfthe sampled PRs — a workflow-level majority test on the same count_toc_blockrenders, coarser than the spine's per-check_typical_checksplit) has its wall-clock demoted to bill-only (wall_clock_p50_s=0,realization=none, tier 2) — exactly the existing all-legs-rare demotion path — so it never becomes a_is_credited_cluster_lever, never crowns the headline, and never carries the on-path label. That gate count is summed in the check-name domain via_map_check_to_job(_workflow_gate_freq), not by a raw job-name lookup, so a name-divergent majority workflow is not spuriously demoted. The runner-minute (bill) saving is untouched on either axis: all the legs still pay the shared step whenever the workflow runs. - Guarded (the converse of §12.1a).
verify_report.check_headline_lever_is_presence_eligiblere-derives the crowned lever's workflow gate frequency fromchecks[].pole_n(the sameworkflow_file-summed count_toc_blockrenders) and FAILs when a minority-gate workflow crowns the Bottom line while a majority ceiling exists. A required check in the workflow EXEMPTS it (gates by definition); no stampedpole_nor a below-floor sample is a LOUD narrow SKIP (never reads clean).
12.2 Representative-run selection — nearest-P50 (_persist_pole_logs)
The drilled run is the qualifying instance whose duration is closest to the job
P50 (job_p50_s, else the check p50_s). Qualifying = duration ≥ a floor of
half the median sampled duration (with an absolute _NOOP_FLOOR_S backstop), which
drops short-circuit / self-skipped no-ops (a gated job like changed-tests that
does nothing on most PRs) so they can't be picked. Nearest-P50 (not the slowest, not
the high-skewed qualifying median) makes the drill reconcile with the headline
instead of overstating it.
Everything the drill derives — the floor, the qualifying set, the representative
run, its step timeline, the cross-run sample and the log handed to the fixing
agent — comes from one pool, scoped to the headline runner before the floor is
computed. A job whose sampled runs span more than one runner label inside one
window has a P50 scoped to the label it runs on most (§ _critical_path), so an
unscoped pool mixes populations the headline never measured with the one it did.
The floor would then be set by the wrong machines: a slower non-headline population
drags the median up, and genuine headline runs are discarded as "no-ops". Scoping
first is what makes the drill's every figure describe the runs the headline
reports. The scope falls back to the unfiltered pool when no headline label is
recorded, or when filtering would empty the pool (losing the drill is worse than
losing the runner scope); both fallbacks are traced at DEBUG.
The relative floor is additionally clamped to the stamped typical time
(max(min(0.5 × median, stamped), _NOOP_FLOOR_S)). Read that bound exactly: it
stops the floor rising ABOVE the stamped typical time, so the run nearest the
headline P50 always stays eligible and the drill can never reconcile with
nothing. It does NOT bound the floor below the population's minimum — on the
unfiltered fallback, headline runs BELOW the stamped P50 can still be cut (a
mixed pool of 200/220/240/260/280 plus six at 2000 gives a floor of 240, dropping
the 200 and 220). Only scoping the pool first keeps the whole population. The
absolute backstop still wins over the clamp, so a job whose own typical time is
a self-skip cannot pull no-op instances back in.
The level-1 bar is the check-run gate time (p50_s); the timeline is the
job clock (job_p50_s). They differ — the job includes the runner
setup/teardown steps the check-run clock excludes — so the report states the gap
explicitly rather than leaving a confusing "8m41 job under a 7m05 headline".
12.2a The addressable-ceiling floor — pole-relative co-occurrence, not global presence
The headline win and each pole's "what a change here can buy" note size the addressable ceiling
as pole_p50 − binding_floor (_pole_addressable / _floor_note), where the binding floor is the
slowest concurrent check a typical gating PR also waits on — cutting the pole only helps down to
that floor. Selecting the floor by global presence (a sibling counts only if present on ≥ 0.8 ×
the most-present check anywhere) is wrong: it demotes a genuinely co-occurring 2nd-slowest sibling
that merely runs on slightly fewer PRs than some trivial universal check, so a faster check
becomes the floor and the ceiling overstates (~2× on Infisical). The floor is instead re-derived
relative to the pole's own gating PRs — the PRs where it is the per-PR slowest — from
pr_critical_path.populations: a sibling qualifies iff it co-occurs with the pole on a strict
majority of those gating PRs (_pole_cooccurrence stashes the per-pole counts in render();
_floor_qualifies applies the majority test inside _binding_floor / _floor_candidate). Absent
populations (a minimal/static doc) it falls back to the legacy global-presence test, unchanged.
tests/verify_report.py::check_pole_ceiling_within_cooccurrence is the class invariant that
re-derives this independently and asserts the rendered ceiling never exceeds it.
A modal-chain MEMBER pole is the exception to the pole_p50 − binding_floor arithmetic
(review V2 / OD-F2, #220): needs: serializes it with its chain partners, so a "concurrent floor"
that is really its predecessor/dependent would frame a serial stage as a competing sibling and
overstate the win (deepgram rendered ~28s against a 5.0s stamped headroom). render() stashes the
chain story on member poles (_chain_member, the same threading pattern as _cooccur); both
helpers then cap the member's win at the stamped chain headroom (chain_summary.chain_win_p50_s,
the same bound the N3 cascade applies to member findings) and the member's own span, and the note
renders the chain-stage story instead. The suppression guards still run first (a suppressed note
stays suppressed), non-members are untouched, and
tests/verify_report.py::check_headline_chain_matches_stamp cell 3d re-derives the bound from the
per-PR chain_facts (median of max(chain_s − runner_up_s, 0)) and fails any drilled member note
that exceeds it.
The floor candidates come from a floor_pool that is the full concurrent-check set, NOT the
typical-PR chart set src (Class A #6). The two are deliberately distinct: src is what the L1
waterfall draws (the checks a typical PR runs) and what "merge wait" is measured over, while
floor_pool is every concurrent check a pole could floor against. Conflating them (floor pool =
src = typical) hid a check that is heavy but runs on a MINORITY of PRs — it couldn't be in the
typical chart, so it also couldn't be a floor, so a pole it genuinely caps overstated its ceiling AND
the heavy check was disclosed nowhere on the spine (lightdash E2E: API (Vitest), 13m, on ~50% of
PRs). The co-occurrence filter (above) still gates which floor_pool members qualify per pole, so
widening the pool can't let a rare non-co-occurring check inflate the floor.
tests/verify_report.py::check_spine_heavy_check_disclosed asserts each drilled pole's re-derived
binding floor is disclosed on the spine (drilled / named as the floor / in the "Also slower"
footnote), never silently dropped — a matrix sibling-leg floor is out of scope (the pole represents
its own matrix).
The TOC/header "<workflow> gates N/M PRs" count is wf_gate — how often that workflow holds the
per-PR pole — and it is summed over all of the workflow's checks, not the representative poles
(Class A #2). Summing representatives only dropped a non-representative matrix sibling leg's
pole_count (razorpay/blade blade-validate undercounted 4/20 vs the true 5/20). Since exactly one
check is the per-PR slowest, summing a workflow's checks never double-counts a PR.
tests/verify_report.py::check_workflow_gate_frequency_matches re-derives each workflow's count from
populations and asserts the rendered N/M matches exactly.
When the slowest typical check is ALSO the frequency gate but non-universal (gate_is_slowest and
floor_lowered), the headline discloses BOTH its own conditional time and the lower
population-weighted typical wait (merge_dur = _population_typical_floor, the median of the per-PR
critical-path maxima). WHY the typical wait is lower depends on how often the check runs, and the two
causes are NOT interchangeable: at minority presence (present ≤ npop × _RARE_PRESENCE_FRAC) a
typical PR SKIPS the check, so its absence is what lowers the median — the presence-causal "ran on
only N/npop sampled PRs, so a typical PR finishes in {merge_dur}" framing is faithful; at
majority presence a typical PR RUNS the check, so presence CANNOT lower the median (nx
main-linux, present 19/20, whose ~46m conditional p50 — measured over a wider run-sample — cannot
be dropped to ~11m by 95% presence), and the renderer instead attributes the gap to
conditional-p50-overstatement (a duration/population skew). Blaming presence there is a non-sequitur
the populations contradict. tests/verify_report.py::check_headline_presence_causal_only_when_minority
re-derives present/npop for the named check from populations and FAILs any form-1 (name-first)
presence-causal headline whose check is majority-present. (Its regex intentionally matches only the
form-1 wording; the form-2 name-after branch renders its presence clause unconditionally and is a
known coverage gap, not a contradiction — its floor-reconciliation is guarded separately.)
The crowning basis excludes fileless/managed status checks (issue #12 — a contract change).
The merge-wait headline states what CI makes a typical PR wait. A fileless/managed status check
— a bot gate, a label gate, an external app check — produces no sampled workflow job, so its only
timing is its pr_check_runs span: the wall from when the check-run was created to when it
completed. For a label/bot gate that span is PR-lifetime status-gating latency (a label that sat
open for 8 days reads as an 8-day span), not CI wall-clock. _pole_caps builds its de-inflation caps
only from sampled job_p50 / job_bimodal, so a check with no sampled job is never capped, and its
raw span would flow uncapped into critical_path_s / chain_summary.makespan_p50_s and crown the
headline over file-backed poles tracing <1% of it (electron/electron: Backport Labels Added /
faraday/cage crowning ~8 days). The physical-bounds clamp below cannot catch this: with the fileless
span in the basis, both the crowned figure and the makespan carry the same inflated span, so the
clamp passes trivially — the cap has to happen at the crowning basis, not the bound.
The product rule: PR-lifetime latency of a fileless/managed status check is NEVER a valid basis
for the CI merge-wait headline. collect_runs._partition_fileless_checks (called just before the
present-first ranking) splits the spine's pr_check_p50 into the job-groundable basis and the
fileless set — a check is job-groundable iff it has a sampled developer-timed job
(workflow_jobs), the sampled-timing mapper resolves it, OR the scanned job graph maps it to a
workflow file (a triage-skipped but file-backed check the crown-recovery pass can still recover — real
CI compute, only its per-run drill was skipped; that notion is the same one the check→file binders and
the degenerate-matrix guard use, never re-derived). The fileless set is DROPPED from pr_check_p50
before ranking, so critical_path_check / critical_path_s / populations / chain_facts /
chain_summary.makespan_p50_s all re-derive from the job-groundable population (the exclusion is at
the data layer, so every crowning input moves together). It is not discarded: the excluded set is
stamped in pr_critical_path.fileless_status_checks (name, span_s, basis), and blocking_path
renders a labelled DISCLOSURE near the headline naming the slowest fileless check and its span,
explicitly framed as PR-lifetime status-gating latency (not CI compute). When every tracked check
is fileless (all_checks_fileless), the report says so rather than crowning a status-gating span.
tests/verify_report.py's check_headline_basis_excludes_fileless re-derives the contract — no
crowned slot (critical_path_check, checks[], poles[], modal chain) may name a stamped fileless
check, and the rendered disclosure must bind to the stamped list — so the engine and the verifier
mirror move together (L3/L4/L5).
The physical-bounds clamp on the headline wait (issue #25 family, #22/#24). The rendered
"X until all checks finish" figure is a WALL, and the model that produces it (chain_p50, a median of
per-PR summed needs: spans; or floor_p50, the population-weighted typical floor) can drift OFF the
measured wall in either direction. Two coherence rules constrain it, both re-derived by the verifier:
(1) merge_dur is CAPPED at the measured makespan p50 (chain_summary.makespan_p50_s, the median
per-PR span-capped max(end) − min(start)) — a re-run-inflated check-run clock in populations can
leave floor_p50 above the actual wall (nx main-linux: a 15m08 population floor vs an 11m00
makespan), and a wall can never exceed the measured span; (2) for a needs: CHAIN the total is
additionally FLOORED to the largest modal member's own p50 (checks[].p50_s) — a serial chain
cannot finish faster than its longest single stage, yet chain_p50 diluted by fast PRs can render a
"total" below a member it claims to sum (tokio compile → miri-test: a 17m18 sum below miri-test's own
18m36). The clamp caps at the wall first, then floors to the member (the member is a hard MEASURED
lower bound, so it wins when a member exceeds the wall). The chain claim carries the coherent
chain_wait_p50_s; a report already within [largest member, makespan] renders byte-identically.
Note the deliberate asymmetry between (1) and (2): because the member floor is applied AFTER the
makespan cap, a modal member whose own drill p50 out-measures the span-capped makespan (measured over
a wider PR sample than the chain-forming subset) will crown the headline ABOVE the measured makespan
— (2) treats the member as the harder measured bound and lets it win. This is intentional and stays
coherent (the rendered figure is still a measured member p50, and chain_wait_p50_s/makespan_p50_s/
largest_member_p50_s are all stamped so the reader can re-derive it), but it means bound (b) does
NOT enforce a strict makespan ceiling on a chain headline; its ceiling relaxes to
max(makespan, largest member p50) precisely so a correct member-floored report does not false-FAIL.
tests/verify_report.py's check_aggregate_total_ge_largest_member (bound a) and
check_headline_wait_within_makespan (bound b) re-derive both bounds from chain_facts + checks[]
independently; check_headline_chain_matches_stamp and check_headline_floor_presence_reconciled
mirror the clamp so the headline-family checks stay consistent with the rendered figure. A companion
guard check_saving_within_measured_compute (bound c) holds any finding's credited runner_min_saving
at or below its affected jobs' measured monthly billable compute in the runner_minute_spine. Bound (c) is now enforced at the SOURCE, for every rm-crediting pattern, by the measured sizing door
(§5.1):
collect_runs._reground_runner_minute_savings runs once the spine is final and routes each finding
through one policy. Whole-run-cancel patterns (OPT45 missing-concurrency, _SIZING cost_basis: affected_jobs) stay inside the bound BY DERIVATION rather than clamp:
_reground_whole_run_cancel_saving re-derives the saving as hit_rate × Σ(affected-job completed_billable_equiv_min_per_month from the spine) (completed job runs: timed-out,
cancelled and failed-at-or-past-6-hour runs excluded, 2026-10-12) — the same measured rows bound (c) reads, joined the
same way (matrix-stripped base, reusable-workflow fallback via the shared _measured_billable_index),
and at least as strictly — so hit_rate ≤ 1 puts it within bound by construction (superseding the
provisional sizing-time figure, which grounds the no-spine case in the affected jobs' summed p50, never
the workflow long pole priced at full volume). The cluster-floor lever (OPT73) is CLAMPED to the same
Σ(measured billable) — #43, nx 1919.7 → 1404.4. A finding whose affected jobs ALL miss the spine join
is unsized at the source (runner_min_saving = null, basis unmeasured_no_spine_match) rather than
left carrying its provisional figure — bound (c)'s loud coverage SKIP only fires when EVERY
runner-minute finding misses the spine, so an unmatched one in a mixed report would otherwise render
unbounded. The old long-pole × full-volume model over-credited a workflow whose cancelled jobs run on
only a fraction of its runs (#33). Every sized finding stamps runner_min_basis, and
check_saving_carries_measured_basis FAILs a saving that carries none.
A VALUELESS finding whose jobs are ALL drilled long poles is excluded from the off-path "Also noticed"
appendix (Class A #5): the pole headlines that job as the single biggest lever, so re-listing a
valueless co-located finding on it (e.g. OPT24 on the pole's own pytest-torch job) as ~0-wall-clock
"minor cleanup" contradicts the headline. _also_noticed_block drops a finding only when it has no
bill saving AND is not a credited wall-clock lever (_saves_wall_clock) AND every one of its
affected_jobs (matrix (variant) stripped via _job_base) is a drilled pole — NARROW on purpose, so
a credited OPT73 bill lever, a wall-clock lever, or a finding that also touches a non-pole job is kept
(never a silent drop of a real lever). tests/verify_report.py::check_pole_not_reframed_as_hygiene
asserts no valueless all-pole-job finding (criteria read from findings.json, not the lossy rendered
**Where:**) appears in the appendix.
The TOC "🧹 Also noticed" pointer must agree with the appendix on on/off-path (Class A #7). When a
shown finding is a credited wall-clock lever that sits ON the critical path, _also_noticed_block
returns any_wc_lever=True and _toc_block reframes the pointer rather than blanket-labelling the
section "off-path … below the critical path / ~0 wall-clock" (which would contradict the inline
on-path row — getlago). tests/verify_report.py::check_toc_also_noticed_label_honest asserts the
pointer never makes the blanket off-path claim while the appendix carries the on-path sentence.
Two DECOUPLED gates govern the appendix "sits ON the merge-gating critical path" note, and the
split is load-bearing. _saves_wall_clock is the credited-MAGNITUDE / catalog-COVERAGE gate: it
declines a finding below the ≥30s long-pole floor (rejects sub-floor rounding artifacts) or stamped
off_spine (a required-scoped drop — encord §6 Cause 2), and it is what _data_driven_for_pole /
_catalog_covers consult to decide a pole is catalog-covered (NOT a coverage gap → no phase-4a
gap-fill, no .ci-speedup-gaps/ capture, no phase-4c detector-draft). _frames_on_path
(= _saves_wall_clock AND not spine_rare) is the SEPARATE appendix on-path FRAMING gate. The
spine_rare demotion is applied ONLY in _frames_on_path, never in _saves_wall_clock — because
a job the footnote demotes as OPT-IN / RARE (present on a MINORITY of sampled PRs, so "a typical PR
doesn't wait on it") keeps its FULL multi-minute wall_clock_p50_s (the concurrency cascade bounds
only against concurrent checks, not against the presence demotion) and IS still genuinely
catalog-covered; declining it inside _saves_wall_clock would manufacture a FALSE coverage gap for
a rare job that IS the drilled pole (the paradedb Test pg_search shape — its log matched no
detector but an OPT24 fired on it). So the appendix keeps showing such a lever's real wall-clock
magnitude but reframes it "opt-in / rare — helps the minority of PRs that run it, not the typical
merge path", and the pole-waterfall's data-driven pointer is reworded to match
(_pole_waterfall(..., data_driven_on_path=…)). render() stamps spine_rare via
_stamp_spine_rare using the renderer's OWN opt-in test (not _typical_check(name) AND present on
a minority — the same populations/present_on basis the footnote uses). The stamp join is by check
NAME — exact, or unexpanded matrix base↔rendered leg — NOT by (workflow, job) identity, because
the footnote demotes a check NAME ("X … a typical PR doesn't wait on it") and the reader can't
tell which workflow's X a later "sits ON the critical path" note is about. So it deliberately does
NOT fold DISTINCT sibling matrix legs (_same_matrix — a finding on the exact RARE leg test (macos-latest) must not be blocked by its TYPICAL sibling test (windows-latest)) and does NOT
skip a _wf_conflict (a same-NAME finding in another workflow must be demoted too — the tauri
test (macos-latest) double-framing had both arms). The kept-guard (mirrors
_stamp_off_spine_findings) still stamps only when the job maps ONLY to opt-in/rare checks, so a
job also on a typical check — or an unexpanded-base finding covering a typical leg — keeps its
legitimate on-path framing; over-stamping under the NAME rule can only make the appendix MORE
conservative, never manufacture the opposite contradiction. A multi-leg cluster (OPT73) finding is
a distinct arm: it is genuinely on-path via its TYPICAL leg (so it is NOT spine_rare), but its
appendix **Where:** must LEAD with that on-path leg, not affected_jobs[0] — _also_noticed_block
picks the first non-spine_rare leg (_lead_job → _occ_loc(lead_job=…)), and _detect_shared_substep
now orders affected_jobs slowest-first (matching the evidence's "slowest cluster job") so the
displayed location is the on-path job even for older bundles' order. The presence-minority clause is load-bearing: a FREQUENCY-demoted
matrix leg present on EVERY PR but rarely the single slowest (requests build (3.10, windows-latest), pole_n 0 on 11/11 PRs) is NOT opt-in — its matrix IS on the critical path every
PR, so an OPT73/OPT24 across it legitimately cuts wall-clock, and _ms_freq_demoted gives it a
different ("throughput/cost, a heavier check gates ahead") footnote.
tests/verify_report.py::check_dropped_check_not_framed_on_path re-derives BOTH the hard-dropped
set (any framing context) and the opt-in/rare set (_rare_demoted_check_names, from
checks[].pole_n/presence) and asserts neither is framed on-path — the rare set scoped to the
appendix note only, since a demoted Long-pole HEADER carries its own opt-in body framing and is no
contradiction.
12.3 The leaf detectors and the load-bearing magnitude (_parse_log)
blocking_path._parse_log is the single source of truth for root-cause detection —
the regex-over-tool-output detectors below, each returning {fix_key, unit_label, deeper: [levels], evidence, magnitude} (fix_key names the root-cause family + selects the
agent prompt in _PROMPTS; it is not a prescribed fix):
fix_key |
Root cause | magnitude (the one load-bearing number) |
|---|---|---|
prisma-migrate-once |
db push --force-reset per test group |
DB-migration share of the slowest test file |
vitest-v8-coverage |
compile/instrument/load dominates measured summed worker time | compile+instrument share of test work |
vitest-isolate-pool |
per-file isolation re-pays the import cost (OPT78, HIGH risk). Gated on scan.py's test_runner_isolation config fact — whose verdict field must read isolation_on — so every _parse_log caller (render, _gap_poles, _persist_pole_logs / _magnitude_sample) must pass that block; a guard test pins that every call site does |
import share of the vitest run |
vitest-import-bound |
the same import-bound split when OPT78 is WITHHELD (opted out, config not fully read, an unresolvable isolate value, a crashed config reader, vm pool): names the split and the withheld reason, forbids flipping isolation. Its config sentence travels as the leaf's config_fact, rendered outside the untrusted-log block |
import share of the vitest run |
turbo-remote-cache |
remote caching off / 0 cached → every package rebuilt | packages rebuilt (cache-miss %) |
turbo-partial-cache |
caching ON but ≥40% rebuilt every run → unstable cache key (cache-key churn) | packages rebuilt despite caching (cache-miss %) |
install-lifecycle-build |
a root prepare/postinstall lifecycle script runs a build DURING <pm> install — "work runs during install", not a cache miss |
build wall run inside install (seconds) |
buildx-no-cache |
docker buildx imports no cache → every layer cold |
slowest cold layer's share of the cold-build wall (%) |
playwright-parallel |
specs run as sequential invocations | none — the finding is the sequencing, shown by the timeline itself |
within-job-duplicate-test |
two invocations print the same completed test signature under the same readable active runtime and package scope (OPT16) | later completion duration (seconds, upper bound) |
within-job-duplicate-build |
two invocations print the same package/output build signature under the same readable active runtime (OPT16) | later completion duration (seconds, upper bound) |
lighthouse-serial-audits |
Lighthouse CI audits every URL numberOfRuns times per collect pass, one after another (OPT103). Counts completed passes only, and fires at 20 or more audits with at least two per distinct URL |
audits beyond the first per URL, as a share of all audits (%) |
runtime-ts-transpilation |
a long-pole command directly invokes a runtime TypeScript loader and its profiler summary assigns at least 30s and 20% of the job to loader self time (OPT106) | profiled loader self time (seconds, ceiling) |
A detector with magnitude: None is categorical only: the cause is structural
(visible in the timeline / config), so there's no scalar to validate and no
single step is crowned for a drill.
OPT83 and OPT84 (category 14) are not leaf detectors: they are routed from a pole's step decomposition and its workflow YAML (§11), never from its log.
install-lifecycle-build fires only from a bare <pm> install step (no
--ignore-scripts, no explicit &&/; chain, and — for a run: | multi-line block — no
explicit build in the echoed command lines), with a real lifecycle-script marker in the
section and a timed in-install turbo build ≥ _LIFECYCLE_BUILD_FLOOR_S (30s). It is placed
BEFORE the turbo detectors so a plain later-step turbo build still classifies as
turbo-remote-cache/turbo-partial-cache. The install-section boundary + gating block come
from _install_build_section, the single source both this detector and _cache_state_of_log
read, so the leaf and the per-run cache state can never measure different builds.
The within-job-duplicate-test / within-job-duplicate-build pair occupies the
slot immediately before the Cargo-specific double-compile leaf. The generic pair
recognizes completed test and package-build signatures hidden behind wrappers,
including two invocations inside one step; the Cargo leaf remains later and more
specific to crate/profile/target context. The separate fix keys keep test and
build category reconciliation exact. All three cap their magnitude at the later
invocation's measured completion wall.
12.4 Cross-run check — adaptive sampling (_magnitude_sample)
The drill comes from one run. The categorical cause (istanbul on, cache cold) is stable run-to-run, but the magnitude ("migrations are half the file") is one sample. The cross-run check validates only the magnitude, and does so adaptively to keep log fetches (the expensive call) proportional to the doubt:
- Cheap probe — parse the drilled run + the fastest & slowest qualifying runs
(
k_probe=3), bracketing the spread for ~2 extra log fetches. - Escalate only if wide — if
range/median > _MAG_WIDE_REL(0.25), widen tok_max=8runs spread evenly across the qualifying set (by duration), so the in-between region — where a second cluster would hide — gets sampled. A tight probe stops here: stability is proven cheaply. - Robust verdict — the renderer (
_mag_line) reports the drilled value as a bracket at small n and a real median only at n≥5, and judges spread by IQR (not raw range) once widened, so a tight cluster plus one stray run reads "effectively stable (1 outlier)" rather than "varies run to run". That distinction — smooth wide spread vs. tight-plus-outlier — is the entire payoff of escalating.
_magnitude_sample lives in collect_runs but calls blocking_path._parse_log
(same-skill import) so the magnitude uses the same detectors as the drill — no
duplicate parsing logic. Cost: 2 extra fetches/pole when tight, ≤k_max−1 when wide.
Singleton sample ⇒ no section, and no citation of it. When only the drilled run
carries a value (<2 numeric samples — a repo whose other sampled runs skipped the
job, or a shallow probe), _mag_line returns [] and the "🔬 Cross-run check" section
does not render. The per-pole agent prompt + the LLM-analysis provenance line therefore
gate their "validated across runs in the cross-run check above" claim on whether the
section actually rendered (cross_run_rendered = bool(mag_block)), falling back to
"measured in the drilled run" — a prompt must never cite a section that isn't there.
verify_report.check_rca_hands_off_never_prescribes re-derives this per pole (a
cross-run-check locator phrase must co-occur with the section marker in the same pole).
12.4a Cache-distribution grounding — a cache pole's framing must match its measured hit rate
A cache-miss finding is born from ONE drilled run's log, and the drilled run is deliberately the
slow-mode representative — so its miss rate is NOT evidence the cache is a broad wall-clock lever.
For a cache leaf (turbo-*, buildx-no-cache, install-lifecycle-build, dependency-cache), _cache_distribution
stamps a per-pole cache_dist onto pr_critical_path.poles[]: the miss rate across the sampled
runs, split PR vs push, with fork PRs excluded from the upstream median (a fork PR gets no repo
secrets, so a secrets-gated remote cache is unreachable to it, and it cannot restore an upstream
feature branch's own scope — so it can run colder than upstream), plus a re-derivable
verdict (cold | churn | miss-tail | mostly-warm | insufficient, _cache_verdict). The miss metric is always the per-run
cache_state.miss_pct (read by _cache_state_of_log uniformly), NOT the leaf's magnitude (which is
seconds for install-lifecycle-build). dependency-cache uses a 50% partial-miss floor
(_DEPENDENCY_CACHE_FLOOR_PCT, mirrored in verify_report._VR_CACHE_FLOOR_BY_FIX_KEY) and the
OPT96 confirmation described in "Default-branch dependency-cache seeding (OPT96)" below.
Two properties make it honest:
- Every sampled run counts, not only the miss-heavy ones. The distribution reuses the logs
_magnitude_samplealready fetched (no new PR-bucket gh calls), but a run enters iff itscache_stateparsed — regardless of whether_parse_logre-fires the leaf on it. Without this a warm run (below the leaf's fire floor) would drop out and aturbo-partial-cachepole's distribution could only ever be ≥40%-miss → stuck at churn. The push probe is a targeted escalation: ≤_CACHE_PUSH_PROBE_MAX(3) default-branch log fetches, only when a cache leaf fired, disclosed indata_sources.cache_dist_probe. - The renderer obeys the verdict.
_apply_cache_diststrips the "BIGGEST LEVER" / "cache-key churn" framing on amostly-warm/miss-tailverdict and_cache_health_blockrenders the distribution + a mandatory cache-context caveat and an invisible<!-- ci-speedup:cache-context -->marker.insufficient(fewer than 2 upstream runs exposed a summary — no cross-run grounding) also strips the over-claim but discloses a single-run basis in the note instead of a distribution (no marker, since there is nothing measured to size against).cold/churnkeep the broad framing.
The class is enforced by verify_report.check_cache_claim_backed_by_distribution (a rendered cache
claim must carry a cache_dist) and check_cache_framing_matches_distribution (the stamped verdict
must equal the verdict RE-DERIVED from the pole's own raw values — a buggy engine can't stamp its
way past it — and a demoted pole must drop the churn/BIGGEST-LEVER framing: mostly-warm/miss-tail
must also carry the marker, while insufficient must drop the framing but is marker-exempt). Both
SKIP on findings that predate data_sources.cache_dist_probe.
12.4b Off-category leaf demotion — a crowned cause must be the pole's dominant measured work (issue #16)
A _parse_log leaf fires on a tool marker ANYWHERE in the joined job log — it does not check that
the tool's work is the pole's DOMINANT step. When it is not, the leaf hijacks the pole's MEASURED
CAUSE: the eslint-no-cache (scan) leaf fired on nrwl/nx's one combined Run Checks/Lint/Test/ Build step (an nx affected that lints + tests + builds — binned test/payload by §11's
classifier so the OPT72 redundant-work numerator isn't inflated) and crowned it as "the lint step",
pinning the pole's full ~5m08s ceiling on a lint-cache fix though lint is a MINORITY of the measured
time (sveltejs/svelte was the same shape on a type-check-dominated Lint pole).
The rule (mirroring the §12.4a "reframe a leaf off a re-derivable measured fact" precedent):
a leaf may crown the cause / claim the pole's ceiling only when its target step-category AGREES
with the pole's measured dominant_category — the crown collect_runs._decompose_job_steps computes
over the SAME _step_category taxonomy (§11) — AND, for a leaf that shares a coarse category with a
distinct sibling tool (eslint vs. type-check, both bin scan), the dominant STEP is one the leaf
actually addresses. blocking_path._offcategory_leaf maps each fix_key to its category
(_LEAF_STEP_CATEGORY) and, at the leaf-selection point (right after _apply_cache_dist),
_demote_offcategory_leaf splits a crowning leaf from a demoted one. A demoted leaf is not
dropped silently: the pole falls back to its generic dominant-step hand-off (which points at the
measured dominant step) and the leaf renders below the prompt as a labelled secondary observation
(_offcategory_note_block, tagged <!-- ci-speedup:offcategory-leaf -->). A demoted pole is also
NOT treated as a coverage gap (a detector DID match its log), so it never pulls an LLM gap-fill.
Ceiling design — demote, don't bound. Inside a single combined step the lint sub-share is
unmeasurable from step timing alone, so any partial ceiling would be invented; the honest ceiling is
the generic dominant-step wall, which IS measured. So a crowned leaf always has category == dominant_category, and the pole's dominant_p50_s it claims is fully backed by measured time in
that category.
lighthouse-serial-audits is the second leaf with a dominant-step token. A Lighthouse
step bins scan (the classifier reads lighthouse and lhci in the step name), or test
when its name also says so. It shares both bins with unrelated steps, so the leaf crowns a
pole only when the dominant step names lighthouse or lhci.
Every pole whose leaf survives demotion emits a per-pole <!-- ci-speedup:leaf-crown fix_key=… -->
machine marker. verify_report.check_detector_leaf_agrees_with_dominant_category re-derives the
crowned leaf's category from that marker's fix_key (_VR_LEAF_STEP_CATEGORY, pinned to the engine
map by a coupling test — never a trusted stamped category enum), reads the pole's dominant_category
ground truth from findings.json, and FAILs any pole whose crowned leaf category ≠ its dominant
category (classified AUTO_SEED / fabricated-or-unsupported-finding).
12.5 The data bundle (findings_doc["data_bundle"])
_persist_pole_logs captures, per pole, into --data-dir (all tied to the one
nearest-P50 run so L2/L3 stay coherent):
<job>-<id>.log— the drill log (file).<job>-<id>.steps.json— the step timeline (steps_file); no extra gh call, the jobs listing already carries each step'sstarted_at/completed_at.<job>-<id>.mag.json— the cross-run magnitude sample (mag_file; absent for a categorical finding).
Each manifest entry also records the cross-run duration sample (free — reuses
the sampled instances) and selected: "nearest-p50". The renderer matches a bundle
entry to a pole by workflow-filename substring (longest key wins).
12.6 What the reader is told to trust, and the hand-off
The report labels the two scopes so neither is over-read: the categorical root cause is stable across runs (shown once); the magnitudes are this run's, validated by the cross-run check. Level 1 is P50 across PRs; the timeline and the drill are one representative run, absolute for that run, not P50.
Then it hands off, it does not prescribe — the skill-wide contract (§2).
Each pole closes with _PROMPTS[fix_key]: a copy-paste prompt that restates the
measured root cause, links the tool's official docs, and asks the user's agent to
investigate the repo and apply a safe change (stating the failure mode). There is
deliberately no recommendation and no code diff in the report — the agent has
the repo checkout + the captured bundle and reasons the fix in context. (The
blocking-path-*.md worked examples — under reports/ in the pre-public
development archive; see examples/ in this repository — are the committed
documentation of this shape (verified by a fresh render each run, §7).)
12.6a The PR-floor on a push-only repo — never bury a measured pole as static-only
The merge-wait spine is normally anchored on the pull_request flow. A push-only
repo (no PR flow — the team merges straight to the default branch, so the PR sample is
0/N and poles is empty) used to dead-end to the static-only "no run history to
measure" report even when per_workflow_timing held a clear measured long pole — the
genuinely slowest measured job (live: webflow/js-webflow-api, test p50 51.5s vs
compile 25.5s) never appeared, and the report answered "why is the merge slow?" with
runner-minute hygiene only. Root cause: collect_runs._select_pr_floor_workflows (the
PR-floor synthesis, §12) was gated on _PR_VOLUME_EVENTS, so a push-only workflow
synthesized no floor. Fix: when NO PR-volume workflow ran, _select_pr_floor_workflows
falls back to _PUSH_VOLUME_EVENTS ({push}) — the push CI is the developer's merge
wait there — so a measured long pole still anchors a clearly-demoted PR-floor spine. The
fallback fires only when no PR-volume workflow qualifies, so a normal PR repo's
post-merge push (deploy) workflow never displaces its PR spine.
The honesty guard is a re-derivation invariant, not a renderer patch: each
per_workflow_timing entry now carries its observed events, and
verify_report.check_primary_section_present re-derives the merge-path eligibility
(mirroring _select_pr_floor_workflows — PR-volume first, push fallback, CI-clean) and
fails any static-only report that has a measured merge-path long pole. A cron/release-only
repo (timed but on no merge-path event) stays legitimately static-only; an older findings
JSON without the events stamp declines to fire rather than false-flag.
12.6b The aggregation gate — the degenerate chain SINK (issue #1)
Many repos make a single trivial job the one required status check: it runs no
work, needs: everything else in its workflow, and exists only so branch
protection has one name to require (vercel/next.js thank you, build — job
buildPassed, needs: [deploy-target, build, build-wasm, build-native], body
run: exit 1 behind an if:, P50 3s). Because it gates every PR it is
correctly crowned by pole frequency — but the generic pole render then handed the
reader a drill placeholder and a "capture timing, then optimize this step" agent
prompt over a 3-second no-op: correct data, inert advice, at the very first
thing a reader sees.
blocking_path._agg_gate_shape detects the shape at render time from artifact
data already stamped (workflow_job_graph + the check spine — no producer
change): (a) headline P50 <= _AGG_GATE_TRIVIAL_S (30s — below any real
hosted-runner job's checkout+setup floor); **(b)** the pole's job is a *terminal*
node whose transitive needs: closure (>= 2 jobs) covers every non-terminal job
in its workflow — uncovered siblings must themselves be terminal, which is
exactly the shape of an if:-conditional peer sink (publishRelease), so
"effectively all the other jobs" is expressed structurally rather than by reading
if: (which the scanned graph does not carry); (c) no captured/sampled step
above the trivial threshold (step data is a disqualifier only — a 3s gate is
never drilled, and the role line discloses when the shape rests on structure
alone); (d) at least one upstream member resolves to a measured check, so
the report can name where the wait actually is. Duration alone never matches: a
real 3s lint job with no needs: coverage keeps the ordinary rendering.
On a match the pole renders a pole_role_line claim saying what it is — an
aggregation gate whose wait IS its needs: upstream — names the slowest measured
upstream member with its P50, and closes with a pointer to the pole that drills
that member (an anchor when it renders, otherwise a named check plus what a re-run
would do — never a dead link). It renders no drill, no floor note, and no agent
prompt. A sink that is itself a modal-chain member (only ever the chain's
last stage — a sink is terminal) still renders as the aggregation gate, with its
chain stage stated in the same role line, so the verdict job ending a
suite → verdict chain never gets a drill of its own verdict step. A pole carrying
a matched log-detector leaf or a routed specific structural lever (OPT70/71/72)
keeps today's render — there the drill is not inert, and suppressing it would
silently drop a measured lever; the generic OPT75 ("speed up the dominant step")
does not count, because on a sink that step is the verdict itself.
Ranking, routing and the digest agree. The sink's slowest measured upstream
member inherits the sink's gate rank (a job a required verdict needs: is
required in effect), so the p50 tie-break puts the real time-holder at Long pole 1
and the 3s sink after it; before this a required verdict headed the report ahead of
the suite it waits on (starslingdev/skills: test, 3s, ahead of test (self-hosted), 4m32s). The structural router (collect_runs._is_aggregation_gate_check,
the same (a)–(d) test, exempting measured build work that routes to OPT70/OPT72)
routes no lever to a verdict sink, and summary.py lists sinks under
"Aggregation gates", never under "Addressable long poles".
Two coupled invariants hold the seam:
verify_report.check_speed_poles_complete exempts such a pole from the
"every pole ends in a prompt" rule, and
check_aggregation_gate_poles_never_prescribe fails any aggregation-framed
pole that carries a prompt, omits the upstream pointer, or whose shape does not
re-derive from findings.json (_agg_gate_pole_keys, mirroring the renderer) —
so the exemption can never launder a genuinely stunted pole.
12.7 The coverage-gap fallback — never dead-end on an unrecognised stack
The catalog can't have a detector for every build tool, test runner, or framework.
A drilled pole whose log _parse_log doesn't recognise used to render only the step
timeline + a "no known root-cause pattern — coverage gap" note and stop. That is a
product failure: ci-speedup exists to tell the user what's slow and how to fix
it, and an honest dead-end still delivers neither. The fix is two-layered, in this
order of preference:
Extend the deterministic catalog first. When a gap recurs for a common stack, add/loosen a
_parse_logdetector so the breakdown stays measured + auditable.turbo-partial-cache(§12.3) is the worked example: the original cold-cache detector only fired on a fully cold cache (0 cached / remote disabled), so a build with caching ON but most packages still rebuilding (e.g. 57/150 cached, 62% miss — the signature of an unstable cache key) fell through. The detector now also fires on a high partial miss rate (_PARTIAL_MISS_FLOOR_PCT = 40%), hedged because some misses are legitimately-changed packages (the constraint + cross-run check say so). Deterministic coverage is always preferable to the LLM fallback.LLM gap-fill for whatever the catalog still can't reach. It is impossible to catch every stack deterministically, so when
_parse_logreturnsNonethe agent running the skill (SKILL.md phase 4a) reads the captured job log and writes a root-cause analysis —{cause, breakdown, evidence, prompt}— passed back to the renderer viablocking_path.py --analysis KEY=PATH. It renders as a clearly labelled "🤖 LLM root-cause analysis" section (cause + breakdown + verbatim evidence) and a tailored prompt, in place of the dead-end; the waterfall's gap note redirects to it (analysis_present).The determinism boundary is strict. Detection, ranking, the critical-path spine, the step timeline, and the cross-run magnitude check stay fully deterministic and measured — the gap-fill is the only non-deterministic step and touches none of them. It supplies the cause narrative for a pole the catalog couldn't analyse, and to stay honest it must: (a) ground every claim in verbatim log lines it quotes in
evidence(no invented magnitudes — figures are computed from lines in the log); (b) be labelled LLM-derived ("a strong lead to verify, not a measured catalog detector"); (c) end in a prompt, never a prescribed diff. Validation: disabling a real detector (e.g. the vitest one) and letting the gap-fill read the same log reproduces the detector's conclusion + evidence — the fallback reaches catalog-quality findings from the log alone.verify_reportand the adversarial-review rubric (Scope 5) enforce the label + grounding and that no pole ships as a dead-end.A generic structural match is still a gap. OPT75 (the neutral "one step holds the time" catch-all, routed on the dominant step's category) and OPT71 (routed on required-status) name no cause inside the job (
_GENERIC_STRUCTURAL_PATTERNS). A pole whose only catalog matches are these, with a captured log, renders the gap note, is listed by_gap_poles, and takes--analysis; its generic card still renders beside the analysis. Every other structural pattern (OPT70/72/73/74/78) names a cause and covers its pole.verify_reportfails any pole that says "no known root-cause pattern matched" with no analysis block, and the renderer says that phrase only for a real gap (a captured log, no covering pattern, no analysis yet).
13. Cross-references
SKILL.md- the canonical contract (phases, admission gate, quality review).references/patterns/INDEX.md- the 109-pattern catalog, one file per pattern (frontmatter + body); the source of truth for detection and the report's TL;DR / pattern background. The single-filereferences/optimization-patterns.mdis compiled fromreferences/patterns/bymaintainers/ci-speedup/scripts/build_catalog.py.references/wall-clock-methodology.md- critical-path / long-pole / cluster-floor model and the non-additive rule.
references/savings-methodology.md- two-axis sizing and the cache-evidence guardrail.references/adversarial-review-rubric.md- the hostile-review contract (actionability, evidence-verifies-the-headline, ≥2-pass agreement).
maintainers/ci-speedup/MAINTAINERS.md+maintainers/ci-speedup/loops/gap-to-catalog-prompt.md(maintainer-only, not shipped in an installed skill — source checkout only) - the gap → catalog loop (auto-draft a detector when the LLM fallback fires; §12.7).
Default-branch dependency-cache seeding (OPT96)
The static pass correlates PR restore families with default-branch save points,
including local reusable callers and local composite actions. Seed events are a push
to the default branch, schedule, workflow_run and repository_dispatch, each
evaluated against its own conditions; merge_group never counts (temporary queue
ref) and a workflow_dispatch-only saver does not seed, though the evidence notes a
manual warmer. Push filters: tags-only is non-branch only when neither branches
nor branches-ignore is set, string filters are one pattern, null filters are safe.
Unknown conditions remain reachable. Key matching treats every ${{ }} expression
as a wildcard; a writer matches when either template can match the other
(restore-keys as prefixes). A match resting only on an all-expression template is
undecidable and is held back; an all-expression reader with no matching writer is flagged.
Uncertainty never reaches scan_incomplete. A reader the scan cannot decide is held
back (no finding) and recorded in the top-level opt96_unresolved list (workflow_file,
job, family, reason), rendered as a "cache seeding: held back" row in the Data sources
table; a detector error is caught and disclosed the same way. Hold-back scope: a writer
whose saves cannot be read on a seed event (remote reusable workflow, remote action not
on the non-caching allowlist of checkout / upload-artifact / download-artifact /
github-script, unreadable local composite) holds back only readers whose workflow or
caller chain references the same repository or local path, or whose literal key prefix
appears in that step's with: inputs. A missing local reusable file hides no writer.
The default branch comes from --default-branch, else origin/HEAD (evidence notes it may
be stale), else main/master; collect_runs compares the scan's opt96_default_branch
stamp with GitHub's default branch and, on mismatch, demotes every OPT96 finding to
unconfirmed and discloses it in opt96_unresolved. Readers inside a local reusable
workflow carry callers (caller workflow, job chain, callee display name) so they ground
on the caller's pole as "caller job / callee job".
Measured confirmation needs a median upstream PR miss rate of at least 50% and either
every sampled push log showing only misses with no save, the job absent from sampled
push runs, or no push runs at all. Only push jobs that completed (success/failure) count;
queued, in-progress, cancelled or display-name-mismatched push jobs are unknown and never
confirm. The no-push-runs shortcut applies only when the workflow's on: has no
push/schedule/workflow_run/repository_dispatch trigger and its run and job lists
downloaded; otherwise the reason is "push probe unavailable". An unconfirmed finding's
size_note names one of: no matching pole, pole log not fetched, another detector took
precedence, confirmation failed, push probe unavailable, default-branch mismatch.
The dependency-cache and OPT101 cache-path leaves are observation-only
(_OBSERVATION_ONLY_LEAVES): a restore line never closes a coverage gap or holds back the OPT81 advisory, and a default-branch
restore refutes the measured confirmation. Dependency-cache sampling and the push probe
run only for a pole whose job a static OPT96 finding names (_opt96_pole_matches: job id,
or its name: / matrix leg resolved through the workflow YAML).
The "Dependency cache: check this" log leaf renders only when the OPT96-gated sampling
ran on that pole (pole.cache_dist.fix_key == "dependency-cache"); a restore line alone
renders nothing. Cross-run magnitude samples accept a sibling run's number only when the
same detector (fix_key) fired on it (turbo's remote-off and partial leaves count as one
family, _MAG_LEAF_FAMILY), and for OPT96-named poles the pole magnitude stays
the dominant step's wall time with cache outcomes living only in cache_dist.
The dependency-cache leaf reads restore outcomes using the existing magnitude
sample and bounded push probe. Only a matching static finding can promote the
measured install hit/miss difference, multiplied by upstream PR miss share.
Multiple cache results and absent hit timings remain uncredited. The leaf itself
is a secondary observation, never the entire install-step ceiling. Credited rows
pass the within-workflow cap and the existing cross-workflow cascade; below-tail
or unconfirmed rows retain “check this” wording. No additional probe budget is
introduced.
Single-slot queue capacity (OPT97)
_detect_opt97_single_slot_queue groups each workflow's sampled jobs by job and
exact runner-label set. Readiness excludes upstream execution; an entry job
under a workflow-level global group is timed from the run's creation, because
GitHub creates such a run's jobs only when the group frees. A population that
meets the queue thresholds (15 samples, median 60 s, P90 300 s) is evaluated
only on a self-hosted label set or with a static hint (global group, one
repo-level runner, device step). On hosted runners the account's
concurrent-job limit raises queue and in-flight work together, so that queue
stays with OPT43.
In-flight counts never come from the newest-success sample, which sees a small
fraction of competing work. collect() hands the detector
_opt97_dense_competition: one bounded read of every run, any status, created
in the population's window (at most 3 run-list pages), then job lists only for
runs whose workflow could share the runners or group (at most 60; sampled runs
are reused and responses shared across populations). A failed or capped read is
noted "competition not measured" and never fires. Call counts, thin samples,
unresolved jobs and unmeasured populations reach the Data sources table
(opt97_competition_calls, opt97_notes; each row shows 5 entries, the rest
stay in findings JSON).
Measured OPT97 replaces the same workflow's same-job OPT43 (OPT44 is accepted but has no detector), uses the pre-start axis, and credits only established gating jobs: the median same-run completion change when the queue above the P10 of the other sampled jobs is removed. No baseline or incomplete completion timestamps give no credit; existing floors still apply. No utilization or billing claim is derived from correlation.
Merge validation (OPT98)
The required-check reader optionally returns strict/merge-queue metadata from
the same API responses. Strict is on when any source says so and off only when
classic protection was read (a non-admin's 404 there leaves it unknown). A
queue is present when a ruleset, a merge_group trigger or a cached
merge_group run shows one. A strict, queue-free candidate uses one REST search
for the prior 30 complete UTC days before the pin; total_count supplies volume.
At 30 merges a day, up to 30 of the newest merged PR timelines supply the
rebase-churn share (two-parent commits and head force-pushes). Run pages can't:
GitHub empties pull_requests[] once a PR merges.
_opt98_static_scope classifies each queue workflow as full, scoped (OPT70
affected-only signals or a build/remote cache), queue-only or unknown
(event-dependent conditions, commands, inputs or matrices; reusable workflows;
unread local actions). Full candidates need 15 successful first-attempt runs per
event and a queue/PR median ratio from 0.8 to 1.5. Both cases are uncredited and
render in a dedicated Repeated merge validation section so a drilled pole cannot
hide the observation. Unknown scopes, thin samples, slower queue runs, malformed
workflows, unavailable branch rules and missing timings are named in data
sources; merge_queue.status says measured, partial or not_measured.
Infrastructure faults (OPT100)
_opt100_workflow_pass runs once per workflow after the repeated-attempt
prefetch. It reuses the all-status and prior-attempt job pages, resolves
observed job names to exactly one merge-gating YAML job, and reads evidence
only for gating jobs with at least five failures: one log per failed job with
a failed step, or one check-run annotations call (the job id is its check-run
id) per failed job with no failed step, which is how a lost runner appears.
_opt100_job_class parses log timestamps with the shared OPT80 line parser
(seven fractional digits), keeps the first failed step's lines up to one second
past its whole-second end, and stops at that step's runner exit line. The
signature table in blocking_path.py then classifies lost runner, out of
memory (the runner's own exit 137 line), browser crash (crash wording only) and
network (errno or a 502/503 beside HTTP wording, in install, fetch or "Set up
job" steps). Retry compute counts only jobs that ran in the failed attempt and
again in the next. OPT100 is reported as not evaluated only when no finding
fired and unread job pages or logs could change the verdict; a fired finding
states its unread-log count. OPT64 then runs over the retries OPT100 did not
claim.
PR lifecycle context
pr_lifecycle.py adds a separate pr_lifecycle document block and renders it
near the report's top on every path. It does not alter CI findings or savings.
The collector makes at most four read-only GraphQL requests, 10 merged PRs per
page, ordered by most recent update (GraphQL has no merged-at ordering; update
order tracks merges, while creation order drops long-lived PRs merged recently).
Ten per page and GitHub Actions suites only (app 15368) keep a page inside
GitHub's per-query resource limits: installed apps open an empty suite on every
commit, and 25 PRs of full job history exceeded the limit on busy repos. A
query failure or malformed page makes the whole block not measured. A PR
whose own history is incomplete (force-pushed, a truncated nested connection,
required checks missing or not green on the merged head, a malformed timestamp)
is left out and counted by reason; the block shows "Measured N of M" and is
not measured when fewer than 10 PRs remain.
CI running is the union of all GitHub Actions job attempts, clipped to PR life;
skipped jobs, which GitHub stamps as ending just before they start, count as zero.
The merged head's statusCheckRollup (every app's latest checks plus commit
statuses) decides when the last required check passed.
Red waiting extends from failed-check completion to the next commit timestamp;
GitHub no longer supports Commit.pushedDate, so this is explicitly not push
latency.
Green waiting covers every review round. It is the union of each stretch where the PR's latest commit had all its required checks passed and no newer commit existed, running until the next commit timestamp or the merge. The merged head's stretch starts when its rollup shows the last required check passing. An earlier commit is read from the GitHub Actions jobs the same query already returns (no extra request): its stretch starts at the latest completion among the required names, and only when every required name is a passing job on that commit. An earlier commit whose required checks are absent, failing or unreadable adds no green time; it never makes the PR unmeasurable. Only the merged head's data is mandatory. A required check that only another app or a commit status reports keeps earlier rounds out of green, since those are read on the merged head alone. Running wins interval overlaps, followed by red then green; draft waiting and everything else form the remainder. Ready-for-review and convert-to-draft timeline events exclude draft idle time from red/green waiting. The final-head CI share is a subset, not a fifth bucket.
Required checks come from rulesets and classic branch protection. When they are
not fully readable (the usual case for a repository the caller cannot
administer), the block is an estimate rather than not measured. Every check
on each PR's merged head stands in for the required set: the rollup's latest
check runs and commit statuses, excluding skipped and neutral ones and ones that
did not complete by the merge (pending, never completed, or finishing after it).
The estimate needs the head's complete check list (the rollup's pageInfo); a
truncated or empty list leaves the PR out, counted by reason. An earlier commit
is green under the estimate only when those names and all of its own completed
jobs passed. The block's data records basis (required or
all-final-commit-checks), and the rendered block carries the label
"Estimated: required checks are not readable, ..." with the bias it implies:
an optional slow check delays the green moment, so green waiting may be
understated. Readable required checks keep the stricter rules (a required check
finishing after merge leaves the PR out).
The report gives per-PR median and nearest-rank p90 shares, and uses strict
<10% final-head CI / >25% green thresholds for its headline. Current policy,
update-order sampling and coverage are disclosed; no historical-policy claim is
made. verify_report.py re-checks the rendered block (median <= p90, final-head
CI within all CI, headline iff its own rendered shares cross the thresholds) and,
given the findings, requires the estimate label exactly when the recorded basis
is all-final-commit-checks (a measured block with no recorded basis fails).
GraphQL uses a dedicated one-attempt GhClient transport (no retries, so the four-call ceiling includes transport failures), existing governor/breaker, and hashed-query record/replay keys. See GitHub's commit schema and pull-request schema.
Cache effective path mismatch (OPT101)
The scan emits cache_effective_paths as optional context, never findings.
When a drilled log contains the missing-path warning (actions/cache logs it as
[warning]Path Validation Error: ...), collection checks
every sampled successful instance of the same job that reached the same known
actions/cache save post-step, without the duration filter used to pick the drill.
Timestamp-scoped warning evidence is required in every such attempt (the window
runs one second past the whole-second step end; a warning adjacent cache
post-steps could both claim withholds unless all of them warned); unknown logs
and mixed results withhold. Each log is read once, the drilled one is reused, and
it is released before the next. The pole carries cache_path_sample to the
renderer, which adds an uncredited observation and prompt beside the pole's own
leaf and secondary note; the pole stays a coverage gap when nothing else matched. It never
claims the pole ceiling or contributes to _SIZING. See the pattern for the
existing-wrong-path coverage limit and static-context exclusions.