Index of every page on this site: llms.txt
All skills
wdm0006 avatar

/reproducing-ci-locally

@909f594
by Will McGinniswdm0006/python-skills116 stars
18
Run it yourself

Run the CI gate on your machine so it agrees with the runner — deriving the exact command, paths, markers, and env from the workflow file instead of the Makefile, unblocking gate steps that short-circuit and hide the next failure, pinning the linter version CI resolves, building the interpreter/toolchain environment the runner builds, and confirming the run is green instead of explaining a red job away. Use when a check passes locally but fails in CI (or the reverse), when a lint/format job goes red on an untouched file, when setting up a local dev loop for an unfamiliar repo, or before pushing a branch you expect to merge.

Use this Skill: https://skilld.dev/gh/wdm0006/python-skills/reproducing-ci-locally

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈164 tokens for metadata: the name and description. ≈3.2k when used: this file.

Description uses 8.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈656 characters, or 8.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Reproducing CI Locally

A local check is only useful if it runs the same thing the runner runs. Most "green locally, red in CI" failures are not bugs in the code — they are a difference between two commands: different paths, different test markers, different env, a different linter version, or a different interpreter.

The fix is mechanical: derive the local command from the workflow file, not from the Makefile, not from habit, not from what the last repo used.

Read the workflow before you run anything

The workflow is the contract. The Makefile is a convenience that drifts from it.

# What the gate actually is, in order
sed -n '/jobs:/,$p' .github/workflows/ci.yml

# Every command CI runs, across all workflows
grep -rn "run:" .github/workflows/

Copy out four things, verbatim:

  1. The commands and their order.
  2. The paths each command is scoped to (ruff check app tests scripts is not ruff check .).
  3. Test selection — marker expressions, -k filters, which suites are excluded.
  4. The env: block, and the runtime/toolchain versions in setup-* steps.

Each of those four is a distinct way to get a wrong answer locally.

Paths. If CI lints app tests scripts and you run ruff check ., you get findings from directories CI never looks at — a red that isn't a merge blocker and shouldn't be "fixed" in an unrelated PR. Run it the narrow way to reproduce the gate; run it the wide way only when you're deliberately auditing.

Markers. A suite-wide make test that excludes one marker is not the CI gate if CI excludes six. Live-credential integration tests deselected in CI will run locally, hit a fake key, and fail in a way that looks like a regression:

# Wrong: local shorthand — pulls in suites CI never runs
pytest -m "not browser"

# Right: the full expression, copied from the workflow
pytest -m "not browser and not slow and not load and not integration"

Env. Config objects instantiated at import time (a settings singleton at module scope, an engine built when the module loads) make collection fail without the workflow's variables — a wall of "Field required" errors that looks like a broken suite. Mirror the env: block, including the shape of values: if CI passes a Postgres URL and the module builds a pooled engine, a local SQLite URL raises on arguments that dialect rejects before a single test runs.

Keep those values in a gitignored .env.ci copied from the workflow's env: block, so the local command is the workflow command plus one set -a:

set -a; . ./.env.ci; set +a
pytest -m "not browser and not slow and not load and not integration"

A short-circuiting gate hides the next failure

Gate steps run in order and the job stops at the first red. So the CI log shows you one failure even when three are waiting:

- run: ruff check .          # fails here …
- run: ruff format --check . # … so this never runs, and you never see it

You fix the lint error, push, and get an immediate second red for formatting. Same shape everywhere: cargo fmt --all -- --check before cargo clippy --all-targets -- -D warnings before cargo test means a formatting failure tells you nothing about whether clippy or the tests pass.

Run every gate step locally, even after one fails. Don't &&-chain them while diagnosing — run them separately and collect the whole set:

ruff check app tests scripts;  echo "lint:   $?"
ruff format --check app tests; echo "format: $?"
pytest -m "not integration";   echo "tests:  $?"

The corollary: after a red job, never report "only X is broken." Everything downstream of X is unmeasured until you run it.

Pin what gates the build, and reproduce the version CI resolves

An unpinned gating tool means the gate changes without a commit. A range like ruff>=0.4.0 resolves to whatever shipped this morning, and a release that widens file coverage — a formatter that starts formatting code blocks inside Markdown, a linter that promotes a rule to default — turns every open PR red on files nobody touched.

Two habits:

  • Pin the linter, formatter, and toolchain in the manifest, and bump them in a dedicated PR where the reformat is the whole diff.
  • Reproduce with the version CI resolves, not the one you happen to have:
uvx ruff@0.16.4 format --check .      # exactly what the runner would install

# Node: CI does `npm ci` then `npx prettier --check web` — that's the LOCKFILE's
# prettier. A bare `npx prettier` fetches the latest and flags files CI is fine
# with. Read the pinned version, then ask for it.
grep -m1 -A2 '"node_modules/prettier"' package-lock.json
npx -y prettier@3.8.3 --check web

Formatting a file CI never complained about is not a fix — it's an unrelated diff caused by using a different tool than the gate.

Build the environment the runner builds

Package managers will happily invent an environment for you, and the one they invent is not CI's.

  • A fresh clone or worktree has no virtualenv. uv run <tool> silently creates a bare one without your dev extras, then fails with Failed to spawn: ruff — which reads like a missing dependency rather than a missing environment.
  • uv run re-syncs from the lockfile against your host interpreter. On a Python newer than CI's matrix, a pinned dependency with no wheel for that version gets built from source and fails on a compiler error that has nothing to do with your change.
  • Locked and unlocked installs resolve different versions. If CI runs pip install -e ".[dev]" (no lockfile) while a local uv run syncs from the lockfile, an unpinned dependency is old locally and new on the runner. A test asserting a version-sensitive value (a readability score, a formatted string) then passes under the lock and goes red in CI — or the reverse, if uv run silently swaps your hand-built environment for the locked one. Before calling a failure pre-existing or unrelated, note which install produced it (importlib.metadata.version("<pkg>"), not a stale __version__), and reproduce with the install CI uses.
  • Extras differ. If CI installs [dev,web] and make install installs [dev], the full suite errors at collection locally on an import CI has.

Build it explicitly, at CI's interpreter version, with CI's extras:

uv venv .venv --python 3.12 --seed
uv pip install --python "$PWD/.venv/bin/python" -e ".[dev,web]"
.venv/bin/python -m ruff check app tests scripts
.venv/bin/python -m pytest -m "not integration"

Driving the tools as .venv/bin/python -m <tool> sidesteps the re-sync entirely. If you prefer uv run, pass --no-sync. And prefix with env -u VIRTUAL_ENV when a shell profile exports one — otherwise the run is silently redirected into an unrelated environment and its results mean nothing.

The same shape applies to credentials: a stale GITHUB_TOKEN/GH_TOKEN exported by the shell profile takes precedence over a working stored login, so gh reports "invalid token" even though gh auth status also lists a good keyring account. Run the command with the variables unset instead of re-authenticating or editing credentials: env -u GITHUB_TOKEN -u GH_TOKEN gh pr checks. Do this when you are reading CI status to confirm a run, not only when pushing.

Fix divergence in shared config, not in the workflow

When you find a difference, ask where the fix belongs. A flag added to the workflow YAML fixes CI and leaves every local run diverging — so the next person hits the same confusion.

Prefer the file both sides read:

  • Test-runner flags → addopts in pyproject.toml, not the workflow's run:. (Import-mode is the classic one: a source directory on sys.path shadowing an installed compiled package is a config problem, and pinning --import-mode=importlib in addopts fixes local and CI together.)
  • Marker definitions, coverage thresholds, lint rules and target version → the project manifest.
  • Keep requires-python and the linter's target-version in sync; a mismatch means the linter applies rules for a runtime you don't support.

The workflow should read as make lint / make test plus the environment. When it contains flags the local target doesn't, that's the divergence.

Know which checks are actually gates

Not every command in the repo is a merge blocker, and treating them as equal wastes PRs.

# Which jobs are required is a repo setting, not a file — check it
gh api repos/OWNER/REPO/branches/main/protection --jq '.required_status_checks.contexts'

If CI runs the linter but not the type checker, then a pre-existing type error in an untouched module is not blocking your PR — don't fold a speculative fix for it into an unrelated change, and don't claim CI verifies types. The inverse matters too: a helper target like make quality-check that runs more than CI will show you reds that no one is gating on.

Finish by confirming the run, not by explaining it

"Passes locally" is a prediction. Wait for the real result:

gh pr checks --watch
gh run view --log-failed        # the failing step's output, not the summary

When a job is red, fix it in the same PR if the fix is feasible. If you believe it's pre-existing, prove it: check out the base commit and run the same command there. An unverified "pre-existing / out of scope" is how a base branch becomes permanently red.

Build the control without disturbing your branch: export the base commit to a scratch directory and run the one failing test there with the same interpreter.

T=$(mktemp -d) && git archive origin/main | tar -x -C "$T" && cd "$T"
/path/to/venv/bin/python -m pytest tests/test_x.py::test_y

Verify the control tested the base, not your branch. If the venv holds an editable install of your working tree, an import hook can resolve the package to your code ahead of the current directory, and the "control" silently re-tests your branch. Print the module's location from inside the control (python -c "import pkg; print(pkg.__file__)") and confirm it points into the scratch directory. Check this once per project; behavior differs by build backend.

If the failure reproduces on the base, say so with that evidence. If the cause is an unpinned dependency, the red check will move with the date of the install, not with your diff — see "Pin what gates the build" above.

A red check on files your diff never touched may be a stale base. If the branch was cut before a lint/format/test fix landed on the base, CI fails on the inherited problem even though your own change is clean. Tell the two apart by comparing against the base tip, not the merge-base:

git fetch origin
git diff origin/main --stat     # two-dot: what differs from the base as it is now
git diff origin/main...HEAD     # three-dot: only what your branch added

If the failing file shows up in the two-dot diff but not the three-dot one, the base moved under you: rebase (or merge the base in) instead of re-fixing the file, which would only collide with the fix already on the base.

Two traps in the log itself:

  • A step gated on an event (if: github.event.action == 'opened') is skipped when you re-run by pushing a commit. Green-on-rerun can mean not run.
  • A permissions failure at the last step (an HTTP 403 posting a comment) shows every build/test step green with a red X on the job — read which step failed before concluding the code is broken.

Checklist

Before running anything:
- [ ] Read .github/workflows/*.yml — commands, order, paths, markers, env, versions
- [ ] Local command uses CI's paths (not `.`) and CI's full marker expression
- [ ] Workflow env: block mirrored, including value shape (DB URL dialect, etc.)

Environment:
- [ ] venv created explicitly at CI's runtime version, with CI's extras
- [ ] Tools driven from that venv (`.venv/bin/python -m …` or `--no-sync`)
- [ ] `env -u VIRTUAL_ENV` when a shell profile exports one
- [ ] `env -u GITHUB_TOKEN -u GH_TOKEN gh …` when `gh` rejects a token but a stored login works
- [ ] Gating linter/formatter/toolchain pinned; local run uses the pinned version

Running:
- [ ] Every gate step run separately — a first failure hides the rest
- [ ] Formatter check run even when the linter passed (they are different tools)

Fixing:
- [ ] Divergence fixed in shared config (manifest/addopts), not only in the workflow
- [ ] Checked which jobs are actually required before treating a red as blocking
- [ ] Red on an untouched file: two-dot diff against the base tip; rebase if the base already fixed it
- [ ] Waited for the real run; any red either fixed here or proven on the base commit

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts20d3 checks · Risk SAFE
  • Gen Agent Trust Hub20d

    This skill provides guidance on replicating Continuous Integration (CI) environment gates locally to ensure parity between development and runner environments. It uses standard development tools and practices with no detected security risks.

  • Socket20d

    No alerts

  • Snyk20d

    Risk: LOW · No issues

Signed by skilld at 909f594. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 weeks ago

README badge

README badge for wdm0006/python-skills/reproducing-ci-locally