ci-optimiser
Mission and rules
Make CI as fast as possible without letting it lie. Target PR feedback p50 <120 seconds and p95 <180 seconds, including dispatch, waiting, setup, execution and final required-check reporting. Keep observed timings separate from targets and runner billing.
Two rules outrank speed:
- Run the full required suite before anything reaches main, on the combined candidate in a merge queue. Enumerate the baseline suite, new tests, platform/database variants, build/type/coverage and migration obligations; existing branch-protection entries alone do not define completeness. Selection may narrow PR feedback only. Execute tests afresh on the queue; bypass test-result caches, retain dependency caches. Post-push or scheduled tests cannot substitute for this gate. If the provider cannot enforce it, report the blocker and do not merge.
- Never hide a failure. No check/test/download retries, broader timeouts, dropped tests, flaky-test quarantine, weakened assertions, ignored exit codes or changed coverage thresholds to obtain green. Diagnose the first failure, fix its cause and validate a new revision. Keep failed attempts visible. Repeat planned benchmark trials and shuffled-order experiments without replacing failed results.
Start with the strict performance contract. It controls the defaults in this skill and takes precedence over historical compatibility examples. Current source and migration details are in toolchain migration. Honor explicit user instructions and already granted authorization.
Scope and autonomy
Infer the requested repository and scope, then verify its actual code. Apply JS/TS choices only to JS/TS projects; record other stacks as not applicable with evidence. Keep a cheap full PR suite when selection would add overhead or risk. Do not claim vendor benchmark numbers as this repo's results.
| Mode | Work |
|---|---|
OBSERVE |
Read code, config, protection, logs and billing evidence; produce the baseline. An audit-only request stays read-only. |
EXPERIMENT |
Use isolated branches for authorized candidate and negative-control runs; preserve all outcomes. |
IMPLEMENT |
Complete authorized reversible work, validate it and open one small PR per intervention. |
MONITOR |
Inspect real CI for regression and proposed rollback; schedule recurring work only when requested. |
Reuse authorization in the conversation. Do all useful, authorized preparation before asking. Collect missing accounts, app installations, secrets, admin capabilities and incremental spending into one concrete list with expected monthly cost and assumptions. Do not buy capacity or connect billing without authorization. Never bypass protection to finish a task, and never run untrusted PR code with privileged credentials.
Audit before editing
- Read repository guidance, workflows and command implementations. Resolve required check identities, provider rules, main admission, deployments and trust boundaries. Read relevant history before changing a deliberate choice.
- Collect the last 150 runs, paginating runs, jobs and attempts. Retain failures, cancellations and incomplete work. If fewer exist, report the actual count and retention/access limits. Prefer the user's requested connector; otherwise use
collect_gh.py --limit 150 --include-incomplete --retries 0with reviewed read-only access. - For each job, report waiting, setup and running p50/p95, sample count, event/configuration era, runner class, outcomes and attempt. Expose teardown/unattributed time and overlapping intervals. Missing step timing is unknown, never zero. Separate dependency delay from runner waiting and first attempts from reruns.
- Measure the whole required path and total runner use separately. Name run-created latency as a proxy when push receipt is unavailable. Do not add percentiles or overlapping job savings. Rank the biggest measured bucket first; do not assume tests are the bottleneck.
- Inventory effective rules, formatted files, typecheck projects, tests, coverage, build outputs and dependency graph. Record exact baseline commit, tool versions and price date. Use the field guide and measurement rules.
Budgets and defaults
Treat all limits as strict: reaching the limit fails it. Enforce per-run stage/job budgets in CI; evaluate p50/p95 from a rolling comparable cohort. A timeout is not a passing measurement. Unknown evidence cannot certify a budget.
| Area | Budget / preferred change | Adoption evidence |
|---|---|---|
| Waiting | <10 s per job; migrate every job, including tiny aggregates, to Blacksmith once app access is ready | Same-commit queue and total time; isolation, concurrency, architecture and actual cost. Keep another runner only with a measured reason or named activation blocker. |
| Setup | <30 s; >95% exact PR dependency-cache hits, seeded by trusted main | Cache node_modules itself for applicable JS projects; exact compatibility keys, frozen cold path and measured restore benefit. Use full-history filter: blob:none. |
| Lint | <5 s; Oxlint | Enable applicable rules/plugins, compare effective rules, paths and diagnostics, and seed violations. Retain only specifically missing checks until equivalence is proven. |
| Typecheck | <5 s; bun check --no-script, TypeScript 7 fallback; tsrs locally |
Whole-project/dependent coverage, supported compiler options and declaration/build obligations. Never substitute an unrelated package script or partial lint coverage. |
| Format / install | Oxfmt; Bun install and scripts | Compare formatting inventory/plugins and frozen lockfile/lifecycle/native dependency behavior. Delete old tooling after coverage and behavior parity. |
| Tests | <60 s per shard; benchmark 8-core workers before multiplying shards | Duration-balanced complete inventory, CPU/memory-aware worker counts and independent database per shard; isolate concurrent worker data too. |
| App reuse | Vitest isolate: false for suites whose cleanup is proven |
Executable cleanup after each file, contract in AGENTS.md, recorded random seeds, leak probes and unchanged full-suite coverage. |
| Shape | Independent lint, types, build and tests in parallel | Preserve real dependencies and a stable fail-closed aggregate. Separate migrations and unrelated apps from the slowest job without moving them beyond the merge gate. |
| Selection | PR-only Nx/Turbo affected or a verified stack graph | Unknown inputs select FULL. Deliberately hide a relevant real change; the selection guard/full-suite oracle must go red. Queue always executes FULL. |
| Local | <9 s actual pre-push hook | Changed-file format/lint plus typecheck of affected projects/dependents; prove installation and nonzero propagation. CI stays authoritative. |
Validation architecture
Use four validation categories to map commands and guarantees, not just job names:
| Category | Enforcement boundary |
|---|---|
rules |
Format, lint, types and policy checks in authoritative CI; fast executable hooks locally. |
change |
Edited and affected tests/dependents on PRs, FULL for uncertainty or high-risk changes. |
integration |
Full suite on each combined queue candidate before main admission, regardless of PR selection. |
release |
Exact validated source and artifact with product-specific compatibility, provenance and rollout evidence; deployment authorization remains separate. |
Keep ci or the existing required identity stable. On GitHub, add merge_group: {types: [checks_requested]} and verify a provider rule requires the queue and the full aggregate. Workflow YAML alone does not enforce main admission. Use PR-specific concurrency; give queue/main runs unique groups so a later pending run cannot replace an earlier candidate.
Experiment and ship
- One change at a time. Run old/new on the same source and workload before replacing tools. Record diagnostics, missing rules/files, test IDs/count/coverage, build/declaration outputs, elapsed time and memory. Pin versions and include cold/warm paths.
- Prove red. Exercise real broken behavior and assert nonzero through the required gate. Cover child failure, cancellation, unexpected skip, missing shard/report, stale revision and malformed selection. An intentionally expected failure in a regression test is not permission to swallow failures in CI.
- Enforce budgets. Use a monotonic command guard such as
scripts/run_budget.pyfor work duration and provider job timestamps for waiting/setup. Keep the enforcement/reporting lane itself bounded. Do not widen limits or exclude slow observations to make the dashboard green. - Open one small PR per intervention. Include linked real before/after runs, exact commits, sample size, full inventory/coverage evidence, misses, cost assumptions and rollback. Label small samples preliminary; do not manufacture 150 observations or infer p95 from a single run.
- Validate on CI. Report total required-path time, total runner minutes and estimated/actual cost, before and after. Recompute the path after each change. Merge only through the configured full-suite queue when authorized; do not use admin bypass.
Use the audit template, PR template and agent brief. Missing account or provider access does not excuse unfinished local work; prepare the exact change, tests and costed activation steps first.
Integrity invariants
- Require every intended child to succeed; unexpected skipped, cancelled, timed-out or absent work fails. Keep workflow triggers broad enough that the aggregate actually runs.
- Preserve first-failure evidence and test inventory. Fix race conditions, nondeterminism, state leaks and network assumptions instead of retrying them.
- Pin actions and toolchains per repo policy. Use ephemeral, least-privilege runners. Trusted jobs never restore caches written by untrusted jobs.
- Bind evidence to exact repo, SHA, event, workflow, attempt, config and artifact. A previous green, cached test result, mutable branch name or changed artifact cannot certify the current candidate.
- Treat helpers as scoped evidence tools. Legacy evaluator support for rolling policies does not authorize rolling main under this contract. A report-generation exit code is not merge eligibility.
Resources
- Strict performance contract — budgets, migration order, queue/caching, enforcement, negative controls and cost.
- Toolchain migration — current supported commands, parity traps, cleanup and sources.
- Discovery matrix, optimization playbook, security.
- Measurement, examples, dated sources, recipes.
- Validation architecture, platform semantics, stack adapters.
- Evidence formats, field guide, measured adapters, feature matrix.
Self-test: python3 -m unittest discover -s tests -v from this skill's directory.