Index of every page on this site: llms.txt
All skills
starslingdev avatar

/ci-score

@9a84115
by StarSlingstarslingdev/skills134 stars
3
Run it yourself

Grades a repository's GitHub Actions configuration against CI best practices (the CI Score: eleven pass/fail configuration facts computed from the repo's own workflow YAML) and hands back concrete fixes for every gap, ranked by impact and risk, each with an apply-now option and a paste-able agent prompt. Use when: (1) the user asks to grade or score their CI, or audit CI configuration against best practices ("grade my CI", "CI score", "CI best practices audit", "how healthy is my CI config"), (2) re-scoring after workflow changes. Requires a local checkout of the target repository. Do NOT trigger for: CI speed, cost, or wall-clock audits, "why is CI slow", or optimization-opportunity analysis (use ci-speedup); security audits or posture review (ci-score is not a security audit — exactly two rubric checks, action pinning and OIDC token scoping, happen to be security-related); writing new workflows from scratch; non-GitHub-Actions CI.

Use this Skill: https://skilld.dev/gh/starslingdev/skills/ci-score

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

CHANGELOG.md

≈22k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Changelog — ci-score

All notable changes to this skill are recorded here (Keep-a-Changelog style; the skill is unversioned; the CI Score registry it carries is versioned separately as ci-score-vX.Y.Z inside references/ci-score-spec.json).

Note on PR/issue numbers. Entries below reference the pull requests and issues of the skill's pre-public development archive, which is not part of this repository's history. The numbers are kept for the maintainers' audit trail; they are not links you can follow here.

[Unreleased]

  • 2026-11-15 (#357) — Changed: CI Score's grade-affecting numeric engine limits and bounded-read ceilings are now recorded with reasons in the machine-readable spec and public methodology. An exhaustive contract test rejects value drift or a newly introduced numeric constant without a matching decision-record entry; scoring behavior is unchanged.

  • 2026-11-15 (#357) — Changed: the constant contract now also covers every prose copy of a bound value in the methodology and the spec's check facts (the nesting depths and the per-job read limit), so a sentence that drifts from its engine constant fails the suite. The constant scan reads signed, arithmetic and shifted values, names built from earlier constants, chained, annotated and augmented assignments, and public names. Scoring behavior is unchanged.

  • 2026-11-10 — Changed: the skill now states that the audited repository's workflow content is untrusted data, never instructions, and that a directive embedded in it is a prompt-injection attempt to report. The other shipped skills already said this; a repo-wide test now keeps the sentence in every skill.

  • 2026-11-07 — Fixed: shallow-checkout grading recognizes inline history operands. Parent-only operands remain exempt because the score cannot express their bounded depth, while branch comparisons still require full history.

  • 2026-11-07 — Fixed: shallow-checkout grading now follows common shell, interpreter, wrapper, workspace-path, Make and Snapcraft forms into one first-party file, and recognizes history reads inside command substitutions, conditions and quoted child-process calls. Parent-only reads stay exempt because the score's static fix recipe cannot yet print their bounded required depth.

  • 2026-11-05 — Fixed: shallow-checkout grading now sees history operations one hop inside a directly invoked first-party shell, Python, or Node script. Resolution stays inside the repository, unreadable paths fail closed, and scripts invoked by that script remain outside the bounded read.

  • 2026-11-03 — Added (OD-CS27, metadata-only): an Advisory notes section, rendered after the scored checks and labelled not part of the score. Its first note, ci.runner.swappable, fires when the same custom runner label is hard-coded in at least three jobs and suggests one repository variable per runner class. All eleven checks, the arithmetic and ci-score-v0.1.8 are unchanged. Report verification binds each advisory bullet to its stored note (rejecting missing, altered, duplicated or unrecorded bullets), rejects a section above the scored checks, and rejects any advisory in the scored table or stamp. The close names each advisory in one line, never as a fix slot. (#182)

  • 2026-10-29 — Fixed: the step-0 and close questions can no longer go unseen on Codex. When the host's question tool is experimental, the same question and options also end the plain message, in the same order, sent before the tool call (a call that never returns cannot hide them); the agent never says it is waiting on a choice it has not visibly shown; every question offers at least two options unless a step names the exception. The step-4 close, the close invariants and the close-contract reference now carry the same rule instead of reserving the printed list for hosts with no tool. Same fix as ci-speedup's shared interaction contract.

  • 2026-10-28 — Changed (registry ci-score-v0.1.8, OD-CS26): shallow checkout is not applicable when every PR-gating job calls another repository. The checkout is invisible locally, so it no longer earns a pass. Scores may fall; local jobs and local reusable-workflow calls remain applicable.

  • 2026-10-28 — Fixed (#161, registry ci-score-v0.1.7, OD-CS25): literal cache switches use the same verdict as resolved expressions. Off values no longer earn credit, and the evidence names their step and value. Explicit setup-uv enable-cache: auto receives the default's runner gate; enable-cache: true enables caching on any runner. Some scores deliberately fall; both frozen controls retain every fact and score.

  • 2026-10-28 — Fixed: every check now names workflows the scan could not parse, in both card evidence and the rendered report. States and scores are unchanged; the disclosure no longer appears on only three checks.

  • 2026-10-28 — Fixed: every workflow uses the strict YAML reader. A duplicate key leaves the workflow ungraded and names the key in the unread evidence and parse-error reasons. Workflow booleans use GitHub's YAML 1.2 spellings so distinct names such as ON and YES remain readable. Workflow reads use the explicit safe-loader lifecycle, preserving the Bandit B506 fix and cleanup on parsing errors. This corrects the existing read contract; the registry rules and version are unchanged.

  • 2026-10-28 — Fixed: strict YAML reads instantiate the existing SafeLoader subclass directly, removing ambiguous generic loader calls flagged by Bandit B506. Duplicate-key and Python-tag rejection remain intact, and parser resources are released on success and errors.

  • 2026-10-27 — Added (#156, registry ci-score-v0.1.6, OD-CS24, on top of ci-score-v0.1.5): the pinning check credits GitHub's workflow lockfile (.github/workflows/actions.lock, public preview 2026-10-14). The lockfile records the exact commit each action runs at while the YAML keeps its tags, so a repository that locked every action used to fail pinning. A tag reference now counts as pinned when the lockfile lists it under its own workflow and holds a full-commit entry for it, and the evidence says how many: "2 of 2 remote action references pinned, 2 via lockfile". Inside a local action, credit needs every reference to reach it through $/ (GitHub never follows or locks a ./ action) and an entry under every workflow that reaches it. A call to another repository's reusable workflow is pinned only by a full SHA, never by the lockfile: the gh actions-lock README states that such calls are not locked (github/gh-actions-lock#179, open), even though GitHub's docs page says reusable workflows are pinned, so until the tool locks them an entry for one enforces nothing. A lockfile grants nothing, and the evidence says so, when it cannot be read, is a symbolic link out of the checkout, sits at a path whose case differs from .github/workflows/actions.lock, carries a format version other than the two GitHub documents (v0.0.2, v0.0.3), or is malformed anywhere: a duplicated key, trailing text after a commit or version, an entry whose ref contradicts its key (GitHub's commit-keyed nested entries, which keep their tag as ref, stay valid), a listed reference with no entry, or a workflow key that is not exactly .github/workflows/<file>. Without a lockfile the evidence carries no lockfile wording (the reach fixes below can still change its counts).

  • 2026-10-27 — Fixed (#156): checks passed, or failed, on workflow content the scan never read.

    • A reusable workflow in the same repository (on: workflow_call) that a pull-request job calls was never graded on the pull-request path, so a full-history checkout inside it passed the shallow-checkout check. Called workflows are now followed (./ and $/, nested calls included, 16 deep at most) and graded with the pull-request events that reach them: the caller's events, narrowed by the calling job's own if: (a workflow that pull requests trigger directly also gains the events of a pull-request workflow that calls it). A calling job that can never run on a pull request reaches nothing. A called workflow's fetch-depth: ${{ inputs.x }} takes each calling job's with: value (any caller passing 0 fails it), else the input's default, plus the empty value a workflow that pull requests also trigger directly sees. On any job on the pull-request path, a fetch-depth set by an expression no rule resolves leaves the job ungraded and named in the evidence. A call that cannot be followed (outside the repository, missing, unparseable, holding a duplicated key, or more than 16 deep) is named in the evidence instead of being graded either way; a called workflow no pull-request path reads is left out of the caching and pinning checks too, so the "not read" note those cards carry is true. When the full-history offender is a called workflow, the evidence names it as one. The trigger-level checks (concurrency groups, cancelling superseded runs, path filters) still read only the workflows a pull request triggers, since those settings belong there.
    • Local composite actions were read only from .github/actions/. An action at ./ci/setup or $/tools/x was invisible: its cache step did not count (a wrong fail) and its unpinned actions were left out of the pinning count (a wrong pass). Every local action a step's uses: names is now read wherever it lives, including a directory named like ./ci/setup.yml, with nested references followed (16 deep, 256 files at most). A reference is never read when it resolves outside the repository (by path, or by a symbolic link, under .github/actions/ too), names no readable action.yml or action.yaml, is not a spelling git tracks, or names a file that does not load (a duplicated key, a YAML error, not a mapping); each is recorded in data_sources.composite_parse_errors with its reason. The spelling check reads git's own file list, the exact paths a runner checks out, so every disk gives the same answer: a wrong-case path does not exist on a Linux runner (the common case), so the scan refuses it rather than grade a file that may not run, and an action under .github/actions/ reached only through a wrong-case path earns nothing. Outside a git work tree the directory listing decides, and a directory that cannot be listed fails closed ("directory could not be listed").
    • A dependency cache switch set by an expression earns credit only when it resolves to a value that turns caching on. The switch is the input the action itself reads: bundler-cache for setup-ruby, enable-cache for setup-uv, cache-disabled for Gradle's actions and npm-install, cache for every other setup-* action; action and input names match in any case (Actions/Setup-Go is actions/setup-go). An expression on a key the action does not read (cache: on setup-uv) changes nothing: the action's default decides. ${{ inputs.cache }} inside a local action takes each caller's with: value, else the action's input default; inside a reusable workflow, each calling job's with: value, else the workflow_call default (an input the workflow declares nowhere has no value). Each value it resolves to is graded exactly as if written in place, by the same rule as #154's defaults, so an empty value on setup-node v5+ falls through to its automatic cache, and the step earns credit when any of them caches. Any other expression (${{ env.X }}, ${{ matrix.x }}, ${{ inputs.cache || 'npm' }}) used to count as a truthy string; it now earns nothing and is named on the card and in data_sources.unresolved_cache_expressions. Such a switch never falls back to the action's default-on cache (#154), which would otherwise read any expression as caching. A value written literally keeps its verdict.
    • The dependency-caching check, card and data_sources alike, reads a local action only from the workflows it grades: an action that only an unread called workflow runs earns nothing.
    • A job-level call to another repository's reusable workflow (jobs.ci.uses: org/shared/.github/workflows/ci.yml@main) was left out of the pinning count, so one pinned step read "1 of 1" and passed. It now reads "1 of 2" and fails; a SHA-pinned call counts as pinned. The evidence notes how many reusable-workflow calls were counted. A repository whose only remote reference is such a call moves from not applicable to a fail ("0 of 1").
    • Content the scan could not read is now named on the caching, pinning and shallow-checkout cards ("not read, so nothing inside was graded: …"), not only in data_sources: local actions not read (a path with no action.yml in the checkout says it may be checked out at run time), workflow files that do not parse, step lists that are not lists, and reusable-workflow calls not followed. A full-history job exempt only because a local action it uses, or one that action reaches, could not be read (missing, outside the repository, in the wrong case, or more than 16 deep) is still exempt, since it cannot be proven history-free; the card now says so ("exempt only because ./shared/setup could not be read") instead of claiming the job walks git history.
    • Owner decision (2026-10-27): dependency caching is not applicable only when a local action that is not in the checkout is the only thing that could explain the missing cache. Nothing readable shows a cache; no readable step is a setup-* action or carries a cache input; the action has no action.yml in the checkout (a wrong-case path, a path outside the repository or a file that does not load does not count), runs in a job on the pull-request path before an install, build or test step (readable local actions are followed in step order), and its own if: is not literally false; and no install anywhere runs without such an action before it in its job. A readable default-on cache action (#154: gradle/gradle-build-action, bahmutov/npm-install) counts as a step that could cache, as a setup-* step does. The card says so, and the "not read" note names the action. Anything else fails.
    • #154's step-level if: exemption for a full-history checkout applies on the whole pull-request path: inside a called workflow it is judged against the events that reach it, and a fetch-depth resolved from a caller's input counts only on a step that can run on a pull request.
    • Both frozen calibration controls score exactly as before (no fact or evidence string moves; a committed snapshot of every fact now enforces it). Other repositories can score LOWER: an action outside .github/actions/ or a called workflow can expose an unpinned action or a full-history checkout that was never read before. A repository can also score higher where an unread local action is the only place its dependency cache could be.
  • 2026-10-27 — Fixed (#154): a repository whose setup action caches dependencies by default no longer fails Dependency caching. setup-go v4+ and setup-uv v5+ with no cache input, setup-ruby with bundler-cache: true, setup-node v5+ when package.json names the package manager (v5: npm, yarn or pnpm; v6+: npm only), gradle/actions/setup-gradle, gradle/gradle-build-action and bahmutov/npm-install were all graded "no cache". The rules are the speed engine's own table, copied rather than imported, and a repo-level test pins the two by constant and by verdict. An explicit opt-out (cache: false, enable-cache: false, package-manager-cache: false, cache-disabled: true) still fails, as do versions before the default turned on. An explicit cache input other than false is credited on any runner, exactly as before, and input names are read case-insensitively in both directions: Cache: npm, Enable-Cache: true and Cache: pip now pass (they failed before), and Cache: false is an opt-out. The default-on credit is given only where the action really restores a cache:

    • A SHA-pinned setup action names no version, so it counts as caching unless its version comment (# v3.5.0, # tag=v3.5.0, # ratchet:owner/action@v3, # pin@v3, # owner/action@v3, # @v3, including after a quoted uses: or a flow-mapping }) names a release from before the default turned on, or names no release at all. A date (# 2024-05-01), a range (# 20.x) or a major the action never released is not a release. Gradle's and npm-install's actions always cache, so their comments are not read.
    • setup-uv's default (enable-cache: auto) caches only on GitHub-hosted runners, so with no input it counts only on one of GitHub's published hosted-runner labels (ubuntu-latest, ubuntu-24.04-arm, windows-2025, macos-15, xcode-27 and the like, not a custom label such as ubuntu-selfhosted-pool), and never inside a local composite action; enable-cache: true earns it anywhere, larger runners included, and the fix says so.
    • setup-node and setup-go read the checked-out workspace when their step runs, so their default counts only after a checkout of this repository to the workspace root has finished earlier in the same job. Before any checkout there is nothing to read; after a checkout into a sub-path or of another repository, the file the action reads is not the one this scan read; and a checkout still running in the same parallel: group, or started with background: true, has not finished. setup-go also needs a go.sum (at the root or at its cache-dependency-path), its cache key. A package-manager-cache written as an expression is read as its default, so it still needs a manager in package.json.
  • 2026-10-27 — Fixed (#154): a SHA-pinned setup-node whose v5 and v6 rules disagree for this package.json (for example packageManager: yarn@4.1.0) is credited when every release its version comment names caches (@<sha> # v5.0.0), by owner decision. No comment, or a comment naming v6, still earns nothing.

  • 2026-10-27 — Fixed (#154): a default the scan cannot decide is no longer a silent fail. A package.json that cannot be read, is not JSON or is not an object, a SHA or branch ref the setup-node rules disagree on, a version comment that names no release, a runs-on expression or runner group under setup-uv's default, a cache-dependency-path that is an expression, absolute, outside the repository or only ! exclusions, and a checkout still running or moved elsewhere are named in the Dependency caching evidence ("setup-node@v5 default cache could not be resolved: package.json is not valid JSON"), including when the check is not applicable, and recorded as data_sources.unresolved_setup_defaults. A package.json starting with a byte-order mark, which setup-node's own parse rejects, is decided off and named the same way ("earns no default cache"). An opt-out, a pre-default release and no checkout at all name nothing.

  • 2026-10-27 — Fixed (#154): a test job whose matrix is generated at runtime (matrix: ${{ fromJSON(...) }}) no longer fails Test sharding; it passes, because the job does run a matrix. A generated matrix on a job that is not test-like earns nothing, the same rule a written-out matrix gets.

  • 2026-10-27 — Fixed (#154): a repository whose every job calls a reusable workflow from another repository no longer fails Job timeouts. A calling job cannot set timeout-minutes, so the check is now not applicable there, the same handoff Dependency caching and Test sharding already make. One job that runs its own steps keeps the check graded. A repository with no other applicable check now gets the all-not-applicable refusal instead of a 0.

  • 2026-10-27 — Fixed (#154): a full-history checkout step whose own if: can never be true on a pull request (for example if: github.event_name == 'push') no longer fails Shallow checkout. The same fail-closed reading the job-level if: gets applies: a condition it cannot fully resolve still counts.

  • 2026-10-27 — Changed (#154): the CI Score registry is bumped to ci-score-v0.1.5 (OD-CS23). Job timeouts gains a not-applicable gate and the shallow-checkout reachability exemption widens from jobs to steps, and the rubric's own rule bumps spec_version when a gate moves. Check count (11), bands, formula, refusals and scorer arithmetic are unchanged. Run against the previous engine, both frozen calibration controls produce identical facts. A test pins those facts and re-checks both controls, and an explicit list of OD-CS23 cases, against the previous engine for any move away from credit; a later change that drops a verdict on purpose adds it to an allowlist. Committed corpus and synthetic stamps carry the new spec_version field and nothing else.

  • 2026-10-10 — Fixed: actions referenced with GitHub's self-repository prefix (uses: $/path, shipped 2026-07-30) are local to the repository, exactly like uses: ./path, but were counted as unpinned remote actions. A repository using the prefix in 24 steps read "111 of 135 remote action references SHA-pinned" and failed the pinning check, with fix advice naming files that had nothing to pin; it now reads "111 of 111" and passes. The shallow-checkout check's git-history exemption also now opens a $/-referenced composite action (and fails closed when it cannot be read), as it already did for ./, and follows nested references: a git-history command inside a local action that the named action itself calls (./ or $/, with a bare or quoted uses: key) exempts the full-history checkout too, and an unreadable nested reference fails closed the same way. A nested reference that resolves outside the repository root ($/../x, $//abs) or cannot be opened at all also fails closed instead of reading as clean or stopping the scan. A malformed $/ job call carrying an @ref suffix is no longer listed as a delegated remote reusable workflow.

  • 2026-10-09 — Fixed (#120): steps inside a GitHub Actions parallel: group (shipped by GitHub 2026-06-25 on github.com and GHEC; not GHES) were invisible to every check that reads steps. A - parallel: entry carries neither run: nor uses:, so its child steps were skipped: the only cache step, an unpinned action, a fetch-depth: 0 checkout, the history op that exempts it, a --filter build or a test command could sit inside a group unseen, and the check passed or failed on a step it never read. One step walker now yields every leaf step in declaration order, and every step read routes through it: dependency caching (and its applicability signal), build caching, shallow checkout (and its git-history exemption), change-scoped builds, pinned action SHAs, and the automation-only refusal's test/build/install signals. The wait: / wait-all: / cancel: steps are skipped and counted; background: true steps are ordinary steps and were already read. Inside parallel: groups nothing is skipped without being counted and named by file, job, step position and reason: a group in a shape GitHub does not document (inside a local composite action, a group inside a group, beside run: / uses: on one step, or a single step where a list belongs) is read anyway, defensively, and kept out of the side-by-side count; a group that cannot be read (not a list, empty, contains itself through a YAML alias, nested more than 64 deep, or past a 10,000-entry read limit per job) is counted as unread; a non-step entry inside a group is counted. The depth cap and read limit mean a hostile alias chain can neither crash the run nor stall it, and any failure in the step walk is now a data_sources.ci_score_error marker (exit 3), never a traceback. Unread groups are disclosed, not scored around, matching how an unparseable workflow file is handled. A job whose steps: is not a list and a local composite action that fails to parse are now recorded by name too (data_sources.unreadable_step_lists, data_sources.composite_parse_errors, present only when non-empty). The findings document records data_sources.parallel_steps whenever a repo has a parallel: key, readable or not, and the report header carries a "Parallel steps" row; verify_report.py re-derives that row on its own and fails a report whose row is missing, altered, or present without the record. Repos with no parallel: group (including ones using only background: with wait: / cancel:) produce byte-identical output. No spec_version bump: verdicts move only for repositories using parallel:, and the census over the calibration controls (31 controls, 1 using parallel: on its default branch as of this change: getsentry/sentry) re-read that one control with all eleven verdicts unchanged, so the registry is unchanged.

  • 2026-09-30 — Changed: the Dependency caching check's "Why it matters" text (engine, methodology, example report) now states its exception: a cache that measurably costs more than it saves should be removed, this check reads configuration only and cannot see that measurement, and a repo that removed such a cache still loses the point. Documentation only; no scoring change, no spec_version bump.

  • 2026-09-02 — Changed: the CI Score registry is bumped to ci-score-v0.1.4 (OD-CS22). The two shallow-clone exemptions below move a gate — one frozen calibration control goes from 9/11 to 10/11 on identical workflow YAML — and the rubric's own rule bumps spec_version when a gate moves. Check count (11), bands, formula, refusals and scorer arithmetic are unchanged. Committed corpus and synthetic stamps carry the new spec_version field and nothing else: no corpus score moves.

  • 2026-09-02 — Fixed: pull-request relevance was decided at the WORKFLOW level, so a job that can never run on a pull request was still graded as if it gated one. A workflow triggered by pull_request can hold a job carrying its own if: that is never true for a pull-request event — a heavy job scoped to pushes on a release branch, or to manual dispatch — and no pull request ever waits for it. The shallow-checkout check now skips such a job. The condition parser is deliberately tiny and fails closed: it reads only github.event_name compared with a string literal by == or !=, combined with &&, || and parentheses, and treats anything else — a variable, a function call, any other context reference, malformed text — as possibly pull-request-reachable, keeping the job on the PR path. A wrong exclusion would silently drop a real finding, so ambiguity always resolves toward keeping the job. The workflow's own trigger list is consulted, so event_name != 'pull_request' does not exempt a job on a workflow triggered by pull_request_target, where that condition is true.

  • 2026-09-02 — Changed: the rubric now states two consequences of that rule that it previously left to the reader. A job scoped to the merge queue alone (github.event_name == 'merge_group') is off the pull-request path, because a merge-queue run is not a pull-request event and no pull request waits for it. And a job switched off with a bare literal — if: false or if: ${{ false }} — is still graded, because the grammar reads no literal; that is a known miss in the fail-closed direction, costing at most a finding that was already being reported, and both behaviours are now pinned by tests so a change to either is deliberate.

  • 2026-09-02 — Fixed: the shallow-checkout check (ci.checkout.shallow-clone) failed a repository for any fetch-depth: 0 on a pull-request-gating workflow, including on jobs where full history is load-bearing — changelog generation, a merge-base or base-SHA diff, git describe, changeset versioning. Following that advice breaks the job. The speed-audit skill's equivalent pattern has always carved those jobs out, so the two shipped skills could give one repository contradictory advice about the same line of YAML. The check now applies the identical carve-out, per JOB rather than per workflow: a job that walks git history is not an offender, while another job in the same file that takes full history for nothing still is. The carve-out fails closed — anything that might read history (a local composite action whose file cannot be read, a changeset/release/version job name) is treated as needing it, so the cost of a miss is a lost finding, never a fix that breaks a build. A pass produced by the carve-out now says so in its evidence instead of claiming no workflow checks out full history.

  • 2026-09-02 — Added: a repo-level parity test pins the git-history carve-out on both sides. The predicate is duplicated rather than imported — each skill must install and run standalone — so the test loads both copies and fails if they ever disagree, on the pattern text or on any input in its battery.

  • Fixed: registry security scanners attributed the frozen third-party control fixtures' workflow files to this repository, because they sat at real .github/workflows/ paths that scanners parse as this repo's live automation. The fixtures now ship under fixture-suffixed, non-workflow paths (tests/fixtures/checkouts-src/, with each .github path segment renamed and every file given a .fixture suffix) and are materialized back to their exact original layout at test time. The relocation is byte-identical — a committed manifest and test_fixture_cloak.py prove the round trip is lossless — and no control score changed. test_fixture_cloak.py also scans the whole skill tree (not just the two known control names) so a new fixture checkout added later under a raw .github/workflows/ path turns CI red instead of silently re-introducing a scanner-attributable workflow.

  • Fixed: the repository-name parser's negative-control example — a fake look-alike host used to prove typosquats are rejected — appeared as a literal domain in a docstring, a test table, and this changelog, and a registry security scan read it as a real malicious URL. The test still exercises the identical attack string (now constructed at runtime); the prose describes the class without naming a domain.

  • 2026-07-31 — Public launch. Ported the finished skill from the pre-public development archive into the public starslingdev/skills repo at CI Score registry ci-score-v0.1.3. The port is the launch: it moves a completed, dogfooded skill (frozen scorer + rubric, ranked recommendations, close/kickoff protocol, offline controls) unchanged. The graded third-party calibration corpus and the internal build/launch specs deliberately stay in the development archive and are not part of this repository; a new tests/test_ci_score_install_surface.py invariant keeps that maintainer-only data out of the installable skills/ci-score/ tree.

  • Close/kickoff protocol extracted to a reference doc (2026-07-30, behavior-neutral). SKILL.md's ## Close contract section (banner rules, stake kinds, menu shapes, kickoff protocol, informed consent, ship/post-ship/post-merge rules, PR requests, refusal closes) moved verbatim to references/close-contract.md; SKILL.md dropped from 474 to 233 lines (230 for the bare extraction, +3 for the review clarification below), back under the 500-line soft ceiling with headroom. In its place, a compact ## Close contract — invariants section keeps the non-negotiables at point of use — banner-verbatim/no-fabricated-score, close-text-then-question-same-turn, menu shapes, save-pick-only-writes-the-report, one-rec-per-approval / stop-at-the-diff — each with its live-miss anchor, plus an imperative instruction to read the full protocol before every close (text position is behavior here; burying these regressed live runs before). One additive rule while extracting (owner-sanctioned): the ask-before-network norm now also covers apply-time READS — any network read during an apply or its investigation (git ls-remote for SHAs, gh api reads like branch protection) is disclosed in the offer or at the moment of need (observed live 2026-07-30: a branch-protection read during a path-filters investigation was protective but undisclosed). Coupled tests repointed to the reference (tests/test_close_contract.py honesty pin; the banner/delta-bar formula re-derivation in tests/test_ci_score_contract.py); no rubric or scorer behavior changed. Two review clarifications on top of the verbatim move: (1) step 4 of the Flow now points explicitly at references/close-contract.md as the first thing to read at the close (the invariants section is the point-of-use summary, not the whole protocol), and a new test_close_contract.py guard pins that entrypoint handoff — the link, the read instruction, and the invariants must stay in SKILL.md or the suite goes red, so the relocated protocol can't become present-but-undiscoverable (Greptile P2). (2) One move-introduced dangling pointer corrected inside the reference: "the interaction contract above" → "SKILL.md's interaction contract", since that contract stays in SKILL.md and nothing is "above" it in the new file (the sole deviation from a byte-verbatim body; the sentence's meaning is unchanged and its mechanics are restated inline).

  • A reported merge starts a fresh banner-led round (owner dogfood 2026-07-30): after the user merged the shipped PR, the session re-scored the merged base but buried the new score in prose and jumped straight to a fix menu — the fresh banner only appeared on manual re-invocation. Now: reporting the merge — or, post-merge, asking for a fresh score or what to do next — triggers a full first close on the merged base (fresh collector banner, updated recommendations, first-close menu, no ship option since nothing is applied yet). Review-loop scoping on the same ruling: an incidental post-merge question (a status check, recalling the earlier number, explaining a finding) is answered on its own terms, not hijacked into a surprise re-grade; and the banner-led rule governs a re-score of a real committed/merged base presented as the current standing — the in-loop post-apply re-score against the dirty working tree keeps its before → after delta bar and is never turned into a fresh banner.

  • Post-apply re-offer leads with "Commit this branch and open a PR" (owner 2026-07-30, live apply run on the docs repo): after a fix was applied and re-scored, the menu dropped the user straight into the next recommendation — no visible path to review/land the work, which is not how a developer works. The stop-at-the-diff rule stands, but the explicit ask it requires is now a first-class menu option: slot 1 post-apply = commit + push + open a PR (body per the PR-request rules: CI Score before → after from stamps, applied recs, caveats; never merge), then ONE next-fix slot, a different recommendation, save last. An unshipped session's close names the branch so the work is findable. Review-loop reconciliations on the same ruling: a ship pick ENDS the loop (close on the opened PR, no silently appending further commits to a branch whose PR is already open); the "stop-at-the-diff" gotcha no longer says "the user commits" flatly — the agent commits/pushes/opens the PR on the ship pick (still never merge), while the apply "yes" alone still authorizes nothing downstream; and the save-pick's terminal description now also names the branch when a fix was applied but not shipped.

  • Two owner rulings from the docs-repo dogfood (2026-07-30): (1) the close menu is now multi-slot, adopting ci-speedup's shape — up to two fix slots ranked best-first (a one-edit-closes-both bundle counts as one slot; each slot carries its own consent scope and risk note), then "a different recommendation", then the verbatim save option last (the platform question tool caps at 4 options). A live run improvised exactly this shape and it read well. (2) Ranking tie-break: among recommendations equal on impact AND risk, the stake kind now decides — security, then reliability, then speed/cost — instead of arbitrary registry position (the two high-impact/low-risk security checks tied with the concurrency/trigger checks and sorted below them purely by registry order — the ci.hygiene.* family never shares that tie group). Impact and risk still dominate; renderer-only (OD-L5), no registry change, verifier parity automatic (it recomputes with the same rank function).

  • Sandbox-aware gh probe in the clone path (2026-07-30, mirroring ci-speedup's gh-gate fix): step 0's gh repo view access check can false-fail in restricted agent shells (Codex) where keyring credentials are unreachable — retry with host access before reading a failure as no-access. (A live Codex ci-speedup run told a logged-in user their auth had expired off exactly this probe shape.)

  • Codex question tool: attempt-first, mirroring ci-speedup (owner 2026-07-29): the interaction contract had talked itself out of the Codex question tool ("expect the plain-message branch to carry most Codex runs") — so a live Codex run never attempted the call and went straight to the printed list. Now worded exactly like ci-speedup's proven contract: name both identifier forms, check for the tool and attempt the call first, fall back to the plain-message option list only when the tool is genuinely absent or the call fails. Also corrected the provenance note: the whole interaction contract mirrors ci-speedup's (it was mislabeled as partly ci-score's own extension).

  • Close completion = the question call, never the text (regression fix #2, quasar dogfood 2026-07-29): a second live close ended with no question at all — its text referenced "the last option below" and the turn ended, leaving the user to type a pick freehand. Root cause: the contract itself scripted the agent to narrate the menu in prose (the report note said "the last option below saves it"), so a finished- sounding text read as a finished close. Restructured: step 4 is explicitly two parts in one turn (close text, then the question call); the options may appear ONLY inside the question; prose references to "the option below" are banned; and the close is defined as unfinished until the question is asked. Review-loop tightening on the same fix: the completion rule now explicitly covers the in-loop "next?" re-offers (not just the first close, or miss #2 recurs one loop later); the step-4 close text is stated to be a separate ordinary message, never packed into the question call; and the "end by asking" mandate is reworded so it forbids a message that leaves the choices unasked without forbidding the sanctioned no-tool plain-message fallback (that fallback message IS the question and does complete the close). Made explicitly Codex-safe (owner directive, parallel to ci-speedup's interaction contract): step 4(b) is platform-conditional with ONE definition — the structured-question tool where one exists (AskUserQuestion on Claude Code), and where none does (most Codex runs) the question IS the numbered option list printed as the message's FINAL block (same options, same order, save last and verbatim), the turn ending on it. The prose-menu ban is scoped to menu NARRATION ahead of the question, expressly NOT that final option list. Cites a third live miss: a Codex run that closed on a bare "Apply recommendation #4 (job timeouts) now?" with no options — the same optionless close on the tool-less platform.

  • Best-practices audit follow-ups (2026-07-29): contract prose standardized on "score" (the noun "grade" survives only in the description's user-utterance triggers and the stamp's literal grade field; the letter band was already never rendered); the methodology's inline link to the spec JSON replaced with plain text so no reference file links to another reference file (references stay one level deep from SKILL.md). From a /skills-best-practices audit of the branch; tests and evals pinning the old wording updated in the same pass.

  • Close-offer mandate front-loaded (regression fix, quasar dogfood 2026-07-29): a live run delivered a contract-perfect close except it ended in a typed "Want me to apply #1?" — no structured options, no save choice. The ONE-structured-question mandate had been buried mid-sentence fifteen lines into the close paragraph by successive additions (instruction decay by burial — the same failure mode the review-loop skill documents for spawn prompts). The mandate is now its own bold rule at the point of action: ending the close on a bare, optionless offer is a violation even when the content is right — what is owed is the fixed options + the save choice, delivered through the structured-question tool where one exists, and otherwise through the contract's plain-message fallback carrying the same options in the same order (the fallback stays sanctioned — the ban is on dropping the options, not on prose per se).

  • Close recs name their stake KIND, not just size (owner dogfood 2026-07-29, quasar run): "high impact / low risk" reads identically for a security exposure and a speed win, so the close's per-recommendation line now leads with the stake kind in one plain word, derived mechanically from the check_id family (ci.security.* → security, ci.hygiene.* → reliability/cost, others → speed/cost — never invented per run). The stake kind stays qualitative — a category and its mechanism, never a saved-seconds/runner-minute/dollar figure — so it cannot drift into the measurement the Boundaries ban forbids. The report-exists note also now says the report explains why each practice matters. Contract + evals only.

  • Interaction contract: confirm the target first, structured questions at both ends (owner 2026-07-29, adopting the ci-speedup convention): (0) a new Flow step 0 confirms the scoring target before the collector runs — default is the detected checkout (<owner/repo> or path), one structured question, "a different repo or path" as the alternative; load-bearing because the report file lands in the scored repo's root, so a wrong working directory writes a file into the wrong project. (4) The kickoff offer (and each later in-loop approval) is now ONE structured question via the platform's question tool where one exists (AskUserQuestion on Claude Code; Codex's user-input request tool; a single plain message with the same fixed options only when neither exists): apply the named recommendation (option carries the informed-consent scope + risk note) / a different recommendation / None, just save the report (.md) last and verbatim, no re-offer after a save pick. evals.json close-contract expectation updated to match. Contract-only — no script changes. The report itself is now opt-in like ci-speedup's (owner dogfood 2026-07-29): rendered and verified in the scratch workdir, written into the target repo as ./ci-score-report.md ONLY when the user picks the save option — no other pick writes it, and no repo path is shown before that pick. Side benefit: an unsaved report no longer leaves the untracked file that flagged the next run's provenance -dirty. Review-loop refinements folded into the same contract: the report is now re-rendered after every re-score (not only when the pass/fail stamp flips) so a save pick always copies the freshest report; an explicit out-of-band save request mid-loop is honored without ending the loop (only the structured "None, just save" pick is terminal); and step 0 falls to a plain-message / name-the-target ask when the working directory is not a git checkout (no default to confirm).

  • Toolkit-review hardening round (2026-07-29): four-agent toolkit pass on the PR found no critical issues; four small fixes folded in. (1) The practice-page census test now carries the two post-freeze pages and asserts the coverage note accounts for all three unmapped pages by name — the exact rot OD-CS21 corrected can no longer recur silently. (2) A malformed stamp (value present, tallies missing) prints NO banner instead of a contradictory "0 pass · 0 fail" box. (3) The HEADER invariant now also runs on collection refusals that carry a commit (no_parseable_workflows renders a real header), red-proven. (4) SKILL.md banner phrasing corrected to <owner/repo>. Pushed back with rationale: a verify guard for the terminal banner (unit tests already re-derive its math; verify_report guards report artifacts, and the banner is collector stdout copied verbatim), and a stale pre-existing archive line in an old decision-log entry (point-in-time record, out of scope).

  • Registry footnote corrected in place (OD-CS21, owner 2026-07-29; no spec_version bump): the spec's practice_coverage_note claimed exactly one published best-practices page lacked a scored check; two pages shipped after it was written. Corrected to account for all three (keep-advisory-checks-non-blocking - no check, branch-protection semantics; replace-fixed-sleeps-with-polling - no check, step-content judgment; bound-job-timeouts - scored, null slug binding, renderer supplies the link). Metadata-only: zero computation, stamps, or calibration rows affected, which is why the version number does not move (rationale recorded in the spec's own decision_log). Surfaced by the an internal-dev-repo session working against the pinned vendored copy.

  • Banner is now pre-drawn by the collector, copied verbatim (dogfood 2026-07-29): a live session drew the close banner's 30-block bar freehand and mis-counted it (29 blocks, misaligned box), so the collector now prints the complete banner box to stdout (value line, 30-block bar via the same round-half-up integer math as the card gauge, pass/fail/n-a tallies, and <repo> @ <7-hex-sha> keeping any -dirty suffix) right after its summary line — the SKILL.md close now says to COPY that box verbatim and never redraw it by hand. Box borders are always equal width (grows to fit a long slug); no banner is printed for refusals or scoring errors. Tests pin bar length, round-half-up fill, equal-width borders, -dirty preservation, slug→basename fallback, and no-banner-on-refusal. A second dogfood round (a live dogfood repo's longer slug) caught the widened box padding its longest row flush against the right border — the width math now reserves a 2-space right margin for every content row, test-pinned. Also hardened in the same pass: _repo_slug now recognises git://, explicit-port, non-git scp-user, and mixed-case github.com remotes (and still rejects look-alike hosts that merely begin with github.com), the no-remote header cell no longer falsely asserts "no GitHub remote detected" (an unrecognised URL also lands there), a slug-without-commit no longer emits a broken /commit/? link, and the HEADER invariant's SHA/dirty checks are scoped to the provenance header region (a 7-char SHA echoed lower in the report can't stand in for the table).

  • Report header: title + provenance table (2026-07-29): the report now opens ci-speedup-style — # <owner/repo> — how does your CI configuration score? over a metadata table stating exactly what was scored: repository (GitHub-linked when a github.com origin exists, local path otherwise), scored commit (linked short SHA; a -dirty run is labelled "tree was dirty" in the table, not just the raw suffix), workflows scanned, rubric version + check count, and run date with the local-checkout-only note. The collector records a new display-only repo_slug field (parsed from the local remote.origin.url config — no network call; absent when the origin is missing or not GitHub). verify_report.py gains a HEADER invariant: the report must quote the document's own short SHA, must say the tree was dirty when commit_sha is -dirty, and must never claim dirt on a clean run — each provably red. No registry change.

  • Terminal score visuals — gauge, close banner, delta bar (2026-07-28): three number-only visuals, every number read straight off the ci_score stamp. (1) The score card's first line is a 25-block gauge (CI Score 38/100 ▏█…░▕ 3 of 8 checks pass · 3 n/a); filled = round-half-up(value × 25 / 100), and the · N n/a tail shows only when the not-applicable count is positive. The ## CI Score: **{value}/100** line stays as the card's second line beneath the gauge (kept so verify_report's CARD-headline invariant is undisturbed). (2) The LLM close opens with a 30-block banner carrying the score, pass/fail/n-a tallies, and <owner/repo> @ <short-sha> — keeping the -dirty suffix whenever the document's commit_sha carries one, so a dirty-tree run stays visibly dirty on the banner (dogfood 2026-07-29); refusals and scoring errors keep their plain-sentence closes (no banner). The banner example art uses deliberately illustrative numbers with an explicit read-from-the-stamp-only note, so an obeying agent can't look correct by parroting the example. (3) The kickoff confirmation shows a before → after delta bar (two 25-block gauges + numeric delta + the check that flipped) when an approved fix's re-score moves the value. verify_report.py gains a GAUGE↔STAMP invariant — the gauge's shown value must equal the stamp's, its filled-block count must equal round-half-up(value × 25 / 100), and its N of M checks pass · K n/a tail must equal the stamp's own tallies; provably red by doctoring a rendered bar, value, or tail. A doc-consistency test also re-derives the shipped SKILL.md banner/delta example art from the formula so the illustration can't silently drift. No registry change (frozen spec untouched — presentation only).

Fixed

  • Report is now self-contained — broken methodology-file links removed (owner hit it live) (2026-07-28): the score card linked each check name to the installed skill's methodology file (/abs/path/….md#anchor). Those links broke in common macOS viewers (which read path.md#anchor as a literal filename → "no such file") and were already dead on GitHub and other machines. The card's check-name links are now IN-DOCUMENT anchors, and the report renders a final "What each check means" appendix — one subsection per check (the check's fact and not-applicable condition verbatim from the registry, the why-it-matters line, the guide link, and the check's Sources: line), generated at render time, never hand-authored. No filesystem path appears in the report. verify_report.py gains an invariant that every check the card links has its appendix subsection in the same document. The methodology file keeps its explainer sections (it serves the website mirror and direct readers); only the report stops depending on it. The Sources links are single-sourced in render_report._CHECK_SOURCES and feed BOTH the appendix and the methodology explainers (a census test asserts they can't diverge); every URL re-verified live. This vindicates the earlier toolkit review that flagged the report should not depend on an external file.

  • Install-signal regex hardened — ReDoS removed + build-tool coverage (review round 7) (2026-07-28): the dependency-install regex ran over an arbitrary (untrusted) target repo's run: text with a (?:\s+-{1,2}\S+)* group whose two-way dash split made it exponential — a crafted run: line (a manager token + ~20 dash-flags + no reachable verb) could hang the collector indefinitely with no timeout or error. De-ambiguated to a single -\S+ decomposition per flag token (behaviour-preserving, now LINEAR), verified with a pathological-input test. Also closed a masking gap the same review found: build tools invoked with a project-path/goal target between the tool and the verb (./gradlew :app:test, mvn -pl app -am install, dotnet tool install) now match via a BOUNDED lazy window, and the i shorthand (npm i/pnpm i/ bun i) is recognised. Added pinning tests for the by-design boundaries (root-only manifest probe → subdir-only manifest is n/a; install inside a local composite action counts) and a linear-time guard.

Changed

  • Automation-only repositories are not scored (OD-CS20 — third refusal, part of the v0.1.3 bump) (2026-07-28, owner-directed): a repository whose visible workflows do NO project build or test — only automation (bots, releases, triage) — is now REFUSED with reason automation_only ("Not scored: this repository's workflows show no build or test activity — what is visible is automation (bots, releases, triage), not the project's CI"), rather than given a technically-honest-but-absurd number that would corrode trust in every real score. The predicate is structural and conservative: refuse only when a repo has no test signal (a test-like job OR a test-runner command — pytest/go test/npm test/cargo test/rspec/… — so a real test-only repo whose job is named ci still scores) AND no build signal (build tool, a build-or-lint job or command, a container-build action) AND no dependency-install command in any workflow — any one scores it, so a small-but-real gate (lint-only job, a single build job) is never falsely refused; a setup-node/setup-bun action does not count (issue-triage bots use those to run their scripts), nor does manifest presence, and a repo delegating its CI to a cross-repo reusable workflow is never refused. compute_ci_score gains one refusal branch; the score arithmetic and the 11-check registry are unchanged; the card/report render it through the existing refusal path. Motivating case: anthropics/claude-code scored a real-looking number off workflows that are all issue-triage bots and release automation. Blast radius verified, not assumed: of the 31 calibration repos, only three have no test-like job (claude-code, coolify, leonardo); on re-collecting their workflows the predicate refuses exactly ONE — anthropics/claude-code (now "not scored") — while coolify keeps its value (real build-push jobs) and leonardo keeps its 0 (its pull_request ci runs a real pnpm install). One calibration row becomes "not scored"; every scoring row is unchanged.

  • Per-check external sources added to the methodology explainers (2026-07-28, owner-directed, dogfood round 6: the score had no citations anywhere): every one of the eleven explainer sections gains a Sources: line with 1-3 authoritative external vendor-documentation links (GitHub Actions docs, actions/cache, actions/checkout, Turborepo, Nx, Gradle, Playwright, Jest, pytest-xdist) — never blogs. Every link was verified live (HTTP 200 after redirects) at commit time; the census test asserts the Sources line's presence and shape (no network in tests).

  • Scale-dependent advice conditioning restored (2026-07-28, owner-directed, dogfood round 6): the three scale-dependent recommendations — test sharding, build caching, change-scoped builds — now state their payoff condition plainly in the ADVICE layer (impact_note + recipe caveat + methodology "Why it matters"), never presenting as unconditionally high-impact on a repo whose suite/build is evidently quick. Test sharding carries OPT24's original detection heuristic — the measured catalog's five-plus-minute test-job threshold — as a plain-English precondition. Provenance: the config port had dropped the source catalog's precondition; this restores it. Scoring is unchanged (advice layer only). SKILL.md's recommendations guidance gains the matching rule.

  • Masking hole closed: three-signal applicability + broadened ecosystems (bot + adversarial review) (2026-07-28): a manifest+install-verb-only gate still MASKED any ecosystem outside JS/Python/Go/Rust — a .NET, Elixir, Scala, or JVM-monorepo repo that installs dependencies but has neither a listed root manifest nor a listed install verb fell to not_applicable and inflated its grade (a regression from v0.1.2, where it failed). Fixed three ways: (1) a language setup action (actions/setup-node/setup-python/setup-java/ setup-dotnet/setup-go/setup-ruby/erlef/setup-beam/… — excluding non-runtime setup-* like docker/setup-buildx) is now a general applicability signal that does not depend on enumerating every ecosystem; (2) the manifest probe adds pom.xml, build.gradle(.kts), *.csproj/ *.fsproj/*.vbproj, packages.config, mix.exs/mix.lock, build.sbt, *.gemspec, cabal.project/*.cabal, stack.yaml, Package.swift; (3) the install-command regex adds mvn/gradle/dotnet/mix/cabal/stack build-and-resolve commands. The check is applicable when ANY of the three signals is present; not_applicable only when NONE (a genuinely dependency-free repo — the only disclosed residual is a truly exotic stack with no root manifest, no recognised install command, and no setup action). Also hardened the install regex to require the verb ADJACENT to the manager (so npm run ci / yarn run add-x no longer false-match) and to strip shell comments before matching (so a commented-out install can never cause a false FAIL — honouring the module's structure-not-substrings invariant). The v0.1.3 post_freeze_receipts summary is corrected to record the two test-sharding row moves (it had wrongly said no row moved). Blast radius unchanged: the three-signal union is strictly more applicable, so it can only remove n/a cases, never add them — no calibration row moves.

  • Post-merge coherence + toolkit hardening riders (2026-07-28): the two explainer sections whose not-applicable conditions v0.1.3 changed are regenerated verbatim from the new registry (the other nine verified already-verbatim); plus the second toolkit pass's five follow-ups: an anchor round-trip test (card _anchor == GitHub's slug for every registry label), a card link-emission test, a content pin on the cache recipe's manifest caveat, the methodology summary-table row aligned to the registry label ("Superseded runs cancelled"), and a stale "grade from the stamp" docstring refreshed to score wording.

  • CI Score registry bumped to ci-score-v0.1.3 — two applicability gates added (OD-CS19) (2026-07-28): two checks gain the applicability gate the rubric's own principle already demanded — a mechanism this fact cannot see is never failed.

    • ci.cache.dependency-cache is not_applicable only when the repo shows NO dependency-install signal at all — an install-signal gate, not a file-only one (see "Round-6 retarget" below). Applicable when ANY of three signals is present: (i) a dependency manifest at the repo root (probe over package.json + JS/TS lockfiles, requirements*.txt, pyproject.toml, setup.py/setup.cfg, Pipfile(.lock), poetry.lock, uv.lock, go.mod/go.sum, Cargo.toml/Cargo.lock, build.gradle(.kts), pom.xml, *.csproj/*.fsproj/*.vbproj, packages.config, mix.exs/ mix.lock, build.sbt, *.gemspec, cabal.project/*.cabal, stack.yaml, Package.swift, Gemfile(.lock), composer.json/composer.lock, pubspec.yaml/pubspec.lock), (ii) a dependency-install command in a step (pip/npm/pnpm/yarn/bun/ poetry/uv/composer/bundle/gem/conda with an install-verb adjacent, plus mvn/gradle/dotnet/mix/cabal/stack build commands, go mod download / cargo fetch), or (iii) a language setup action (setup-node/setup-python/setup-java/setup-dotnet/…). Only a repo with NONE of the three is not_applicable — with nothing installed there is nothing to cache, so a missing cache is never a fail. A cache that IS configured still passes regardless of the signal; the existing reusable-delegation n/a is unchanged and takes precedence.
    • ci.parallel.test-sharding is not_applicable when no test-like job (test/spec/e2e/integration/unit) and no shard-like matrix axis exists in any parsed workflow — you cannot shard tests you do not have. A test job WITHOUT a matrix still fails (you have tests, shard them); the existing reusable-delegation n/a is unchanged and takes precedence.
    • Live discovery + Round-6 retarget: found by dogfooding the public skills repo, which installs its dependencies with pip install pytest pyyaml and caches none. An early file-existence-only manifest gate was the wrong instrument on two counts, exposed in round-6 dogfooding: (a) it missed the motivating repo — the skills repo actually carries a pyproject.toml (pytest config, dependencies=[]), so a file gate leaves it applicable but for the wrong reason, not the install it runs; and (b) a file-only gate would mask real waste — a repo with no manifest file but a bare inline pip install in every run would go not_applicable, silently dropping a genuine fail from the denominator and INFLATING the grade. So the gate was retargeted to install-signal semantics: the skills repo is now honestly applicable+fail (it installs deps and caches none — a real, two-step fix), and a manifest-less inline-install repo is applicable+fail (no masking).
    • Scope of code change: practice_facts.py only (an install-signal probe = manifest file-existence OR a step-code install regex, and a job-shape probe). The scorer arithmetic (ci_score.py) is unchanged — both gates land another not_applicable state, which already leaves the denominator; the check count stays 11. The dependency-cache fix recipe (render_report.py) is reworded to match: a fail means the repo DOES install dependencies and caches none (n/a, not a fail, only when it installs nothing), so the recipe wires a cache to a manifest, or adds the manifest first when deps are installed inline.
    • Blast radius (computed by re-running the scorer, never hand-derived): over the full 31-repo calibration table under v0.1.3, two rows move — both small/atypical repos with no test-like job at all, where the test-sharding gate fires: anthropics/claude-code 44 → 50 and coollabsio/coolify 22 → 25 (F → F, value only). adobe/leonardo also has no test job but already scored 0 (0 passed), so it holds at 0. Every other row is unchanged. The sharding gate is determinable from committed data (workflow_job_graph carries every job id + name, and a committed sharding fail implies no shard-axis existed); the dependency-cache install-signal gate is not recoverable from committed findings, but it is strictly MORE applicable than the file-only gate it replaced (a manifest OR an install command keeps it applicable), so it can only remove n/a cases, never add them — and every dep-cache-fail repo in the set both carries a standard root manifest AND runs an install step, so that gate fires on none of them. The six committed corpora each keep a test-like job (requests's sharding fail stays fail) and the two frozen checkout controls hold at 82. Per-row receipt: calibration/dry-run-2026-07-28-v0.1.3.md.
    • Re-stamp scope + reasoning: the corpora fixtures do NOT carry the repo tree, so their practice_facts cannot be recomputed from source; they were re-stamped by re-running the scorer over the committed facts (arithmetic only), which bumps the stamp's spec_version and nothing else. That is honest because every corpus's facts are consistent with the gates NOT firing, verified from the committed evidence strings: deepgram-python-sdk is the only dependency-cache fail and is a real Python SDK (manifest present → the fail stands); requests is the only sharding fail and its CI has real test jobs (the gate needs no test job to fire → the fail stands); every other corpus passes both checks (cache hits / test-with- matrix present). The two checkout controls DO carry the tree and were fully recomputed via the real collector — both pass dependency-cache (cache hits present) and test-sharding (real test jobs), holding at 82 / 82. The collected/ calibration receipts stay their dated snapshots.
  • Every check name on the card links its explainer (2026-07-28, owner: "the report is sparse - no path to learning more about each check"): the methodology gains an anchored "Each check, explained" section per check (fact verbatim from the registry, not-applicable condition, why-it-matters, guide link where a practice page exists - GENERATED from the registry so it cannot drift), and the score card links every check label to its section in the installed skill. A census test pins registry<->explainer coverage.

  • Number-only presentation (owner, 2026-07-28): surfaces render the numeric score ("82/100"), never the letter band. The bands remain defined in the frozen registry and stamped beside every value (no registry bump - presentation only). Rationale: with 8-12 applicable checks the instrument cannot distinguish adjacent values, and the number states that honestly; a numeric form is also softer on a public page than a report-card letter, and matches the category convention (Mintlify Agent Score, React Doctor). Card headline, collector summary, verify_report CARD invariant, close contract, and methodology all updated; format-pinning tests moved to the new form and assert the letter is NOT rendered.

  • Dogfood round 5 (self-scoring the public skills repo) (2026-07-28): two owner-ratified refinements. (1) Cleanest-clean-win offer: the kickoff offer defaults to #1, but when #1 is not a clean apply on this repo (missing prerequisite, degraded payoff - with evidence stated), the agent may offer the highest-ranked clean apply instead, saying why it skipped ahead; the report's ranking is untouched. Ratifies the live session's judgment (dep-caching ranked #1 but the repo has no manifest to key a cache on; timeouts offered instead). (2) Dependency-cache recipe gains a manifest precondition (the sharding-guard pattern): without a lockfile/manifest the cache input has nothing to hash - add one first; the fix does not apply as written.

  • Security positioning: not-a-security-audit framing, no sibling pointers (2026-07-28, owner-decided): the public skill must not point at non-public siblings, so every ci-secure mention is gone from shipped surfaces. The framing is positive scope-statement instead: ci-score grades best practices, and exactly two of its eleven checks (action pinning, job-scoped OIDC tokens) happen to be security-related - it is not a security audit and claims nothing further about workflow security (description, overview, and routing contract all carry it; security prompts route to "none"/do-not-trigger).

  • Skill renamed ci-advisor -> ci-score (OD-L6) (2026-07-28): the owner's own invocation reflex (/ci-score) proved the entity->tool name hop real; the skill name now equals the metric, the website URLs, and the data dir - OD-W1's one-name rule extended to its last holdout. v1 scope unchanged (grade + fixes, eleven checks). The report filename becomes ./ci-score-report.md. "ci-advisor" is parked as a possible future speed+practices release, constrained by the hard separation rule.

  • Fresh-report rule (dogfood round 4) (2026-07-28): the owner caught the report going stale after applies ("now slightly stale since it predates the pin fix") - the kickoff re-scored but never re-rendered. The contract now requires re-running render + verify whenever a re-score changes the stamp, so ./ci-advisor-report.md always matches the current state.

  • CI Score registry bumped to ci-score-v0.1.2 — ci.trigger.draft-gate removed (OD-CS18) (2026-07-28): the Draft-PR-gating check is dropped from the registry (12 → 11 checks). Rationale: it is a contested COST PREFERENCE, not a consensus best practice — many teams push draft PRs precisely to get CI feedback before review; in the B4 18-repo sweep the top-band (85) exemplars (vitejs/vite 90, home-assistant/core 90) each failed ONLY this check; and it is the one check with no published best-practice page behind it (practice_slug null). Pre-publication timing, the cheapest moment for a rubric fix. The scorer arithmetic (ci_score.py) is unchanged — it iterates the registry, so a smaller registry simply scores over fewer checks; the draft-gate fact computation (practice_facts.py), its _FIX_TABLE entry (render_report.py), and its methodology row are removed.

    • Re-stamp scope: every committed corpus fixture (tests/fixtures/corpora/*/findings.json) and the synthetic stamped-fixture.json were recomputed under v0.1.2 (their practice_facts are kept verbatim — the extra draft-gate fact is a harmless superset the scorer ignores). The 25 maintainers/ci-advisor/calibration/collected/* receipts were left as their dated v0.1.1 snapshots so each stays coherent with its un-restamped report.md; the recomputed 31-repo spread is calibration/dry-run-2026-07-28-v0.1.2.md.
    • Calibration delta (numeric, computed by the scorer over the 31-repo table). Every row shifts by a few points because ~60% of the set was failing the removed draft-gate check (promotions only, none demoted): OneSignal-Flutter-SDK, requests, oven-sh/bun, cal.com, Infisical, nx and redpanda all rise; the two B4 positive controls move mastra 83 → 82 and better-auth 75 → 82 (both 9/11); langfuse and grafana 83 → 82, deepgram 30 → 33, mastodon holds at 100, coolify/leonardo/claude-code hold. The complete per-row before/after (numeric) is calibration/dry-run-2026-07-28-v0.1.2.md.

Fixed

  • Scoring-failure marker rendered from its real (string) shape (2026-07-27): render_card._render_score_card emitted the "CI Score unavailable" line only for a {"error": ...} dict, but collect_config records a scoring failure as a plain string (f"{type(exc).__name__}: {exc}"). On every real scoring failure the card silently dropped the error and the B2 report verifier (verify_report.py) then went red with an opaque message. The card now honours both shapes, so a recorded failure is surfaced and the report↔verify pair stays green. Covered by a new contract cell built from the collector's actual output.

Added

  • Stakes-first TL;DR per recommendation (2026-07-27): every rendered recommendation now opens with a one-line plain-English headline (the tldr field, required for every fix-table entry and asserted present by test_fix_table_covers_every_registry_check) that states why the gap matters before the mechanics. The headlines describe the mechanism the fix changes — they deliberately do NOT promise repo-specific measured outcomes (no "minutes into seconds", no "divides the wait by N", no "makes it free"), staying consistent with the adherence-not-speed disclosure: the grade measures configuration, not timing this scan cannot see.

  • Dogfood round 3 (PR-body score receipts + consent precision) (2026-07-27): (1) PR requests get a receipt-bearing description - opening a PR is never part of an apply (explicit ask only); when asked, the PR description states the CI Score before → after (read from the confirming re-scores' stamps, never recomputed), names the applied recommendation(s), and carries the intermediate-state caveat when one applies, so the reviewer understands why the PR exists from the description alone. (2) Consent scope prefers files over occurrences

    • state how many FILES a fix will touch, not just how many occurrences. The finding's example list is capped at three, but the offer is not capped by it: enumerate the real offender set from the checkout (the same scan the apply must run to fix every offender) and state that true file count; only when the scope genuinely can't be enumerated ahead of the edit does the offer fall back to the occurrence count, and then it names the diff — not the count — as the authoritative scope, so the offer never implies fewer files than the apply will change. (3) Network-lookup disclosure - an apply that needs network lookups (tag→SHA via git ls-remote) says so in the offer, the same ask-before-network spirit that governs cloning.
  • Dogfood round 2 (owner's second live session; apply flow exercised end-to-end) (2026-07-27): three owner rulings folded in. (1) Apply endpoint enforced: the yes covers the EDIT only - the live agent read "yes do #1" as license to commit AND push a branch; the contract now says stop at the diff, with commit / push / PR each needing its own explicit ask (pushing is outward-facing). (2) Deliberate-absence check ratified: the agent spontaneously caught that the repo's full-depth checkout exists because an immutability test walks git history, and advised skipping that rec - now required: a quick scan for visible evidence a failing practice is deliberate, presented as "likely deliberate, I'd skip it" with the reason. (3) Intermediate-state honesty ratified: the agent's unprompted "a group without cancel-in-progress queues, it doesn't cancel" is now required - when an applied fix only fully pays off with a sibling rec, say so and name the sibling. Session also confirmed round-1 fixes live (report home, named-pair bundling, per-rec consent held under a different model).

  • Dogfood round 1 fixes (owner's first live session, an internal-dev-repo) (2026-07-27): three outcomes from real usage. (1) Report home = the ci-speedup convention (owner-decided): the sanitized report renders to an accessible working-directory path (./ci-advisor-report.md) so the handoff prompts outlive the session; raw findings.json stays in scratch; never auto-committed; Gotchas document the report-in-tree -> next-run -dirty interplay honestly. The close contract's absolute-path line was corrected in the same move (it still claimed the report "lives in the scratch workdir, not the repo" - now points at ./ci-advisor-report.md). (2) Same-edit bundling allowed, named explicitly (owner-decided): when a single edit closes two checks at once (one concurrency block with cancel-in-progress satisfies both the concurrency-group and cancel-superseded checks - a superset, not two identical recipes), that one edit may be offered as a single apply naming both checks; two edits that merely touch the same file stay separate approvals - the live agent did exactly the one-edit case and the owner ratified it. (3) Impact notes state what impact scales with: the owner asked "do concurrency groups even matter at this repo's volume?" - fair; the generic high-impact tier overstates on low-traffic repos, so the concurrency/cancel impact notes now say impact scales with push frequency (cheap hygiene, not meaningful savings, on quiet repos).

  • Spec-compliance audit fixes (three-verifier pass over B1-B4) (2026-07-27): three parallel verifiers audited the implementation against both specs; all numbers held (18/18 grades, 17/18 frozen-row matches, zero owner-decision violations), six process findings fixed: (1) absence-type findings now NAME their target workflows in files on fail (four PR-path checks + job-timeouts; capped-3 examples, evidence carries the true count) - the fixing agent re-derives nothing, closing the deferred B4 finding in-wave; (2) NEW sparse-checkout refusal in the collector (sparse_checkout, exit 2)

    • sparse mode hides missing files from the dirty flag, so a clean-HEAD partial view could stamp a confident inflated grade; (3) evals/evals.json now encodes the three B4 validation layers where the spec places them; (4) live refusal-repo smoke run and recorded (antirez/sds: honest refusal, report renders, verify OK); (5) sweep record corrected (17/18 calibration matches, not 7 - the draft understated its own coverage) and (6) the django-class precision recorded: fully off-tree external CI has no in-tree marker, so the W1-D judgment item can never be reduced to marker-scanning.
  • Cannot-see-mechanism caveats (B4 sense-check) (2026-07-27): the owner's grade sense-check on the 18-repo sweep found three honest-but-misleading grades, all one root cause - practices delivered via mechanisms outside workflow YAML (nx distributes tests via Nx Cloud agents yet failed test-sharding, and the recipe would have advised adding a shard matrix on top; bun/django are Buildkite/Jenkins-primary and were graded on auxiliary GHA workflows). The report's recommendations section now carries a general external-CI disclosure, the sharding recipe carries an explicit do-not-apply-over-distribution caveat, and both are test-pinned. The Wave-1 publish-gate floor gained the measured datum (bun/django at 9 applicable sail past a <8 floor - the human external-CI judgment item is load-bearing).

  • B4 (partial) — exact-match controls + 18-repo sweep green (2026-07-27): frozen input-surface checkouts for the two calibration positive controls committed under tests/fixtures/checkouts/ (upstream SHAs recorded); tests/test_b4_controls.py runs the FULL collector path on each, offline, and asserts the calibration grade exactly (mastra 83, better-auth 75) - an exact-match failure now always means engine drift, never input drift. The 18-repo live dogfood sweep ran clean (18/18 collect/render/verify green, 7/7 calibration cross-checks matched on live HEADs, kickoff apply loop exercised end-to-end on a throwaway clone: full scope, oracle flip, expected -dirty confirmation). Sweep record: maintainers/ci-advisor/. One open follow-up logged (absence-type findings cite zero files). B4 remains OPEN behind the owner's interactive dogfooding; SHIP is owner-gated.

  • Contract hardening from the post-merge adversarial pass (2026-07-27): two skeptic reviews read the live SKILL.md as the obeying agent and found 15 text-level defects; all folded in. Kickoff protocol: pre-existing uncommitted work is now protected (never stash/reset/discard work the agent didn't create; ask on a dirty tree), the still-failing re-score loop is BOUNDED (stop only after the recipe reached the full offender scope - the evidence's count, not just its up-to-three examples - never mutate YAML to force a pass), the re-score is scoped as a config-fact oracle (not proof CI runs green), informed consent before apply (state file-count scope; surface risk notes for skip-CI recs in the offer itself), and a blanket "apply all" is explicitly not per-recommendation approval. Close contract: the recs/offer block is gated to scored-with-fails (clean repos and refusals no longer get an absurd "Apply #1?"), and the report's absolute path is stated. Flow: <workdir> defined (scratch outside the checkout, per-repo subdir), verify-failure gets a re-render + stamp-fallback recovery instead of a dead end, self-cloning is a valid acquisition path (ask first; third-party grades stay for the user's eyes), and the no-speed-claims boundary is operational (route mid-session speed questions to ci-speedup). Recipes: sharding guarded on runner support, pinning never fabricates SHAs (network lookup or hand back; Renovate = follow-up suggestion), change- scoped warns it needs a merge-base and must not combine with the shallow-clone fix on the same workflow.

  • B3 — SKILL.md goes live + close contract + routing contract (2026-07-27): the NOT LAUNCHED placeholder is replaced by the real skill contract: triggering description per OD-L4 (explicit do-not-trigger routing to ci-speedup/ci-secure), the collect -> render -> verify -> close flow (verify_report must print OK before a report is presented), the close contract wired live (open with the grade from the stamp, refusal/error/ absent handled honestly, never fabricate; top ranked recs; the OD-L5 "Apply recommendation #1 now?" kickoff with per-approval apply), a Gotchas section, and Boundaries. The kickoff/handoff promise is honest about the three-example offender cap in _practice_facts (files truncates to 3): the fix applies everywhere the practice is missing and the re-scored check — not the listed files — is the completion oracle, so a >3-offender check is never falsely reported fixed. New: evals/prompt-routing.json (the committed OD-L4 routing contract; ambiguous middle -> ci-speedup) guarded by tests/test_routing_artifact.py, and evals/evals.json (4 scenarios incl. a should-not-trigger case). skills-best-practices audit run: 3 warnings found and fixed in the same PR (Gotchas section, evals.json, methodology TOC). Continuous-dogfood pass: collect->render->verify green on 3 real repos (the public skills repo and two internal-dev-repos).

  • B2 — report + ranked recommendations (OD-L5) (2026-07-27): scripts/render_report.py renders the full report from a findings document only: score card (stamp-only, unchanged) + the adherence-not-speed disclosure beside it + one recommendation per FAILED check ranked by impact x risk (cheapest-safest first within a tier), each with honest impact/risk notes (path filters say plainly they can skip CI that should run), a concrete YAML fix recipe, the registry-bound best-practices page link (fix-table override only for the one page that postdates the freeze: bound-job-timeouts), and a paste-able agent handoff prompt grounded in the document (evidence + files + scored commit quoted; capture-once). Ranking tiers live in the renderer's _FIX_TABLE, never the frozen registry. scripts/verify_report.py is the one-command PASS/FAIL invariant checker (stamp<->card verbatim agreement, stamp recount, disclosure presence, every FAIL fully recommended, rank-order-follows-the-table, cited files = the stamp's own). Contract cells: tests/test_render_report.py (12 cells; the checker proven RED on all six invariants — a drifted stamp, a rec-less FAIL, a missing disclosure, a mis-ordered ranking, an internally inconsistent stamp, and a rec citing a file the stamp never cited).

  • B1 — the entry point (2026-07-27): scripts/collect_config.py, the skill's front door. Local-checkout-only per OD-L2 (a partial fetch silently inflates grades, so outside a git checkout it refuses politely — never guesses); offline by construction (contract cells run with sockets booby-trapped); parses .github/workflows/, runs _practice_facts → compute_ci_score, writes findings.json with the ci_score stamp. Every outcome is stamped: no-workflows → the spec's no_workflow_yaml refusal in the stamp (exit 0 — a refusal is a result); not-a-checkout → collection_refusal (exit 2); scoring failure → data_sources.ci_score_error, never a partial stamp (exit 3); unparseable workflows counted + named in data_sources.workflow_parse_errors; workflow files present but none parseable → collection_refusal (no_parseable_workflows, exit 2) rather than a deflated F(0) computed from zero readable documents (that would assert false "absent" facts). A --repo target inside a checkout is normalized to the repo top level so a subdirectory target scores the whole repo, never a partial (grade-inflating) view. Provenance = full-repo HEAD SHA with an honest -dirty suffix (untracked files included; an unverifiable tree state — git status failing — is marked -dirty, never certified clean) that truncation never hides. STARSLING_LOG_LEVEL DEBUG lines record counts, names, and states — never file contents. Contract cells: tests/test_collect_config.py (11 cells incl. offline, determinism, subdir normalization, all refusal/error paths).

  • Born by extraction from ci-speedup (2026-07-16, #237). ci-advisor is the new private home of the CI Score, lifted whole out of ci-speedup so that skill ships score-free:

    • scripts/ci_score.py — the pure-function scorer (moved byte-identical).
    • scripts/practice_facts.py — _practice_facts and its parsing helpers, lifted out of ci-speedup's scan.py (copied self-contained; no cross-skill imports) so ci-advisor owns its input end-to-end.
    • scripts/render_card.py — _render_score_card (+ a copied _flatten_cell) lifted out of ci-speedup's blocking_path.py.
    • references/ci-score-spec.json — the frozen CI Score v0.1.1 registry (moved byte-identical; never edited).
    • references/ci-score-methodology.md — the human-facing rubric.
    • Tests rebuilt against fixtures only (no imports of ci-speedup scripts, no reads of ci-speedup's reports/): tests/test_ci_score_contract.py (recompute-from-stamps over the six scored corpus fixtures + stamped-fixture.json), tests/test_ci_score_spec.py (spec shape), tests/test_close_contract.py (the relocated close-honesty pin).
    • WIP SKILL.md (explicitly NOT LAUNCHED), carrying the relocated close contract as ci-advisor's future close guidance.

Fixed

  • Guarded the yaml import in practice_facts.py (2026-07-27): was a bare import yaml that would kill a clean install with a raw traceback (PyYAML is not stdlib and the skills CLI installs no deps); now the loud install-hint + sys.exit(1) pattern the ci-speedup scanner uses.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

2 warnings1d3 checks · Risk SAFE
  • Gen Agent Trust Hub1d

    The ci-score skill is a secure auditing utility designed to analyze GitHub Actions configurations for speed and hygiene best practices. It uses local Python scripts to perform structural analysis on repository YAML files and generates ranked recommendations for configuration improvements. The skill includes robust safeguards against indirect prompt injection by explicitly instructing the agent to treat analyzed content as untrusted data. All operations are local-first, ensuring data privacy, and the skill's dependencies and external references are limited to trusted or vendor-owned domains.

  • Socket1d

    3 alerts: gptAnomaly

  • Snyk1d

    Risk: MEDIUM · 2 issues

Signed by skilld at 9a84115. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 16 hours ago.

Activeupdated 2 weeks ago

README badge

README badge for starslingdev/skills/ci-score