Index of every page on this site: llms.txt
All skills
nielsmadan avatar

/agent-models

@9fbe49f
Run it yourself

Refresh model tiers for claude, codex and pix session shortcuts, the OpenRouter coding models for Pi, OpenCode and clor, or the extraction models for distill and jina-fetch. Uses Artificial Analysis intelligence versus cost per task for coding and measured benchmarks for extraction. Use for "pick models", "update the models", "refresh model tiers", "update session presets", "refresh extraction models", or questions about current model bindings. Cursor agent shortcuts are outside this skill's targets.

Use this Skill: https://skilld.dev/gh/nielsmadan/agentic-coding/agent-models

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈130 tokens for metadata: the name and description. ≈4.2k when used: this file. ≈4.4k more on demand in 2 files.

Description uses 6.5% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈520 characters, or 6.5%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Agent Models

Maintains three independent selections:

  • Session tiers: low, mid, high, xhigh, max for claude, codex, and pix. Anthropic only for Claude; OpenAI only for Codex and pix.
  • OpenRouter coding: low, mid, high-main, high-fallback for plain Pi, OpenCode and clor.
  • Extraction: a measured default and fallback shared by distill and jina-fetch.

The following constraints apply to the OpenRouter coding selection. Session tiers have their own rules in session tiers.

  • Never pick a Grok / xAI model. rank_models.py filters them out entirely.
  • Cap per-token price at $2/M input and $10/M output. rank_models.py excludes anything above it (--max-price-in / --max-price-out to change). Real sessions bill by token, and a model past the cap made a poor high-fallback in November 2026 even where it won $/task.
  • Prefer alternative labs. OpenAI, Anthropic and Google are eligible but only win a tier when clearly ahead of the best alternative; the point of these harnesses is to get a read on non-frontier-lab models.
  • Rank on Intelligence Index vs cost per Intelligence Index task, the two axes of the chart of that name on artificialanalysis.ai. This replaced the Agentic Index in September 2026, when AA stopped publishing it per model. The Coding Agent Index (/agents/coding-agents) was considered and passed over: it covers ~19 harness+model pairs, so most candidates are missing from it.

Instructions

A normal invocation refreshes all three selections: session tiers, OpenRouter coding and extraction. Include the Claude, Codex and pix tiers in the same proposal, application and verification as the existing models. The existing --coding option limits a run to both coding selections; --extraction limits it to extraction. A request naming particular harnesses narrows the scope; it does not authorize refreshing the others. A read-only question only inspects the current bindings. Benchmark artifacts may be written before approval; model configuration may not, unless the user's request already authorizes the picks or a choice among named alternatives.

Step 1: Rank the candidates

Skip Steps 1 through 2a for --extraction.

python3 .claude/skills/agent-models/scripts/rank_models.py

It fetches /leaderboards/models (every model) and /models (a richer record — effort, token counts — for a featured subset), parses both out of the RSC payload each page server-renders, joins them on model/effort record id, resolves each model to an exact OpenRouter id, and prints a table plus the cost/intelligence Pareto frontier. Leaderboard-only records carry effort only in their name, which the script parses (GPT-6 Sol (low)). Add --no-cache to bypass the 6h fetch cache, --all to show excluded models, --json for machine-readable output.

Do not read the chart through a fetcher instead. The page is JS-rendered: WebFetch and jina-fetch both return prose with no numbers in it.

Read the Frontier and Cost cliffs sections — that is where the tiers are.

Step 2: Choose four models

Work off the frontier, cheapest first. Marginal cost per index point is the signal: a segment that buys many points cheaply is inside a tier, a segment that buys almost nothing for a lot of money is the cliff between tiers.

Tier Rule
low Cheapest frontier point worth running at all. Its natural home is the terse one-shot slots (occli, the haiku rung) — it only earns the default as well if it is also cheapest per input token.
mid The next frontier point up. Usually the best value on the board, and usually the right default.
high-main Highest intelligence index before the first cost cliff. Buying past a cliff is the mistake this ranking exists to prevent.
high-fallback Best model from a different creator at comparable capability — within ~6 index points of high-main, at similar cost. A second opinion from the same lab is not a second opinion.

Prefer the frontier points themselves. When the frontier has three points below the cliff, that is the ladder — take them in order rather than reaching off-frontier to fill a tier. A tier filled with a dominated model is worse than a tier that repeats the one above it.

Apply the alternative-lab tiebreak: when a big-lab model and an alternative are close in a tier, take the alternative. Take the big-lab model only when it is clearly ahead at comparable-or-lower cost — "clearly ahead" means ahead on the ranking axes, not ahead on one axis while behind on the others.

Note each pick's effort column (max, xhigh, high). The benchmark number is for that effort level, so the effort has to carry into the config — a model recorded at max and run at low is not the model that was measured.

Sanity-check the ladder before presenting it

Four checks. Each one has caught a real bad pick; run all four.

  1. Cost must not invert. low ≤ mid ≤ high-main on $/task. A mid that costs more than high-main is not a step up, it is a dominated pick — re-pick it, or collapse the tier into high-main and say so.
  2. No dominated pick wins a tier. If a candidate is behind another on both index and cost, it cannot take a tier on any other merit. (Under the old Agentic Index, GPT-5.6 Sol nearly took mid at $0.953 while GLM-5.3 sat above it on the index and below it on price.)
  3. Check input price and tokens/task, not just $/task. $/task is a benchmark blend. Real agentic sessions re-send their whole context every turn, so the bill tracks price_in far more closely than $/task — a terse model can win $/task and still be the more expensive one to actually run.
  4. Name which tier holds the default, out loud. It is whichever tier is cheapest per input token among those that can carry everyday work — not automatically low. See references/targets.md.

Getting tokens per task

The printed table has no token columns, and --json does not carry them either. They exist only on the /models records (a featured subset) — leaderboard-only models have none. Pull them with the script's own parser:

import importlib.util
spec = importlib.util.spec_from_file_location("rm", ".claude/skills/agent-models/scripts/rank_models.py")
rm = importlib.util.module_from_spec(spec); spec.loader.exec_module(rm)
for r in rm.aa_models():
    otp = r.get("intelligenceIndexOutputTokensPerTask") or {}   # reasoning / answer / output
    can = r.get("canonicalIntelligenceIndexTokenCount") or {}   # suite totals, for the in:out ratio
    # input tokens per task = otp["output"] * can["input"] / can["output"]

otp["output"] checks out exactly against cost.output ÷ price_out, so the derivation is sound. Use it whenever a tier decision turns on verbosity.

Pricing a real workload

To compare candidates on how this machine actually bills rather than on the benchmark blend, replay a measured session's token split at each model's prices. rl usage session <id> gives the split; cache reads bill at roughly 12% of the input rate (measured on OpenRouter, muse-spark: $0.15/M read vs $1.25/M input):

cost ≈ input_M × price_in  +  cache_read_M × 0.12 × price_in  +  output_M × price_out

Validate it against the session's own recorded cost before trusting it — pi records per-turn usage.cost, so the replay can be checked to the cent.

Step 2a: Choose the session tiers

Follow the ranking and selection rules in session tiers for Claude's Anthropic ladder and the shared Codex/pix OpenAI ladder. Carry those picks into the combined proposal in Step 3.

Step 2b: Benchmark extraction candidates

Skip this step only for --coding. Read docs/tests/extraction-models/README.md and use the small suite through the existing runner. Refresh exact model IDs, prices and provider availability from OpenRouter; include bin/extraction.py's default and fallback as controls plus at most four candidates. Exclude Grok/xAI. Never choose extraction models from the Intelligence Index alone.

python3 benchmark/bench-models.py --small benchmark/extraction-candidates.txt benchmark/runs/YYYY-MM-DD-extraction --repeats 3 --workers 3 --budget 5 --timeout 60

Use a fresh output directory for every run. The defaults bound each refresh to 144 requests and a $5 conservative spend guard; do not expand either without the user asking. No automatic fallback runs in the benchmark. Inspect summary.json, responses.jsonl, and metadata.json: incomplete runs, API errors, unknown costs, and missing provider data remain explicit. Review every quote failure against the saved source before calling it fabrication. Reject fabricated quotes and answers to the wrong subject; among reliable candidates compare observed latency and cost. Prefer an independently served fallback when its speed and quality are acceptable. If all candidates pass, report that this small suite did not distinguish quality.

Step 3: Present the proposal, and stop

Include the five session tiers per lab, with native model IDs, efforts, scores, task costs and cliffs, alongside the OpenRouter and extraction proposals below.

Show a table: tier, model name, creator, intelligence index, cost per task, effort, and exact OpenRouter id. Say in one line per tier why it won, and name the runner-up so an override is a one-word reply.

Call out anything surprising — a tier that did not move, a pick that is off the frontier, a cliff that shifted, a current model that has dropped out of the ranking entirely.

For extraction, show full model names, exact IDs, passed/attempted counts, API errors, latency, known cost and unknown-cost count, and provider names. Recommend a default and fallback with the limits of the evidence. Link the saved results.

Then show the binding table from references/targets.md and wait for confirmation. Do not change model configuration before the user approves the picks and bindings for the requested scope. Approval of a coding ladder does not authorize an unmentioned extraction-model change.

Step 4: Apply

Apply the approved session tiers using their targets.

Read references/targets.md and update the targets in the approved scope. It carries the exact key names, prefix rules (openrouter/ on some keys, not others), the [1m] context suffix on clor, and which files are loadout sources vs generated.

Preserve unrelated content in every file: pi's openai/* entries, the clor continuation backslashes, CLOR_MODEL / CLOR_EFFORT overrides.

For extraction, update DEFAULT_MODEL and FALLBACK_MODEL in bin/extraction.py, which both CLIs import, and the matching MODELS pair in mods/dev-phase/pi/extension.ts (see the extraction targets). Preserve --model overrides and JINA_FETCH_MODEL. Update the current selection in docs/tests/extraction-models/README.md with the supporting run link. Do not rerun the paid benchmark just to apply a pick.

Step 5: Sync and verify

Run the session-tier checks for session changes; those shell sources and the project skill need no loadout sync.

For extraction changes run:

python3 -m unittest discover -s bin -p 'test_extraction.py'
python3 -m unittest discover -s bin -p 'test_distill.py'
python3 -m unittest discover -s benchmark -p 'test_extraction_benchmark.py'
claude plugin test mods/dev-phase
jina-fetch --help
distill --help

No loadout sync is needed for extraction constants. For coding changes:

loadout sync --global     # required for opencode; needs an unsandboxed shell
source ~/.airc            # reloads clor / ocs / occli
loadout check --global    # must be clean
prat --config pratfall/config.toml config validate

pratfall/config.toml is symlinked to ~/.config/pratfall/config.toml, so it needs no sync — but a malformed profile only surfaces at the next /second-opinion or occli, hence the validate.

loadout sync writes outside ~/wrksp, so it needs claude-raw or a shell the sandbox does not cover. If it fails on a grant, report it — do not relocate files.

Then confirm what landed:

grep -n "MODEL=\|CLOR_MODEL:-" .airc.d/claude.zsh   # 4 model slots + the start alias
grep -n "openrouter/" .airc.d/opencode.zsh pratfall/config.toml publish/overrides/second-opinion/SKILL.md
python3 -c "import json;d=json.load(open('loadout/settings/pi.json'));print(d['defaultModel'],d['defaultThinkingLevel'],d['enabledModels'])"

Confirm the start alias resolves to the default tier — that is the one edit with no model id in it, so a stale value survives every id-based check. Confirm the advisor-* profiles match the approved advisor models and reasoning levels, preserving the user-selected overrides in references/targets.md. The local /second-opinion skill names no model, but the public override in publish/overrides/ does — check both stayed in step.

Report the diff summary and leave the commit to the user.

Examples

Example 1: Routine refresh

User says: "refresh the model tiers"

  1. Run rank_models.py. Frontier comes back GPT-5.6 Luna ($0.049, 46.9) → GLM-5.3-Flash ($0.087, 58.2) → GLM-5.3 ($0.683, 59.1) → Claude Opus 5 ($2.337, 59.2), with a cliff flagged after GLM-5.3 at $23.23 per extra index point.
  2. Three frontier points sit below the cliff, so they are the ladder: low = GPT-5.6 Luna (max), mid = GLM-5.3-Flash (max), high-main = GLM-5.3 (max). Luna is a frontier lab but takes low cleanly — the best alternative near its price is MiniMax-M3 at 36.1 for $0.139, so Luna is 10.8 points stronger and 2.8x cheaper. high-fallback = Qwen3.8 2.4T A95B — different lab, 57.1 at $0.807, beating Kimi K3 on both axes.
  3. Run the four sanity checks. Costs rise across the ladder ✓, no pick is dominated ✓. But check 3 fires: Luna costs more per input token than mid ($0.20/M vs $0.15/M) and only wins $/task on terseness (136k tokens/task vs 700k). So mid holds the default — low keeps occli and the haiku rung.
  4. Present the four with runners-up, the default-tier call, and the binding table. Wait.
  5. On approval: the six targets, loadout sync --global, source ~/.airc, loadout check --global.

Result: pi, OpenCode and clor default to GLM-5.3-Flash; clor opens on sonnet so it lands on that default; subagents take it too; occli and the haiku rung keep Luna.

Example 2: Read-only question

User says: "what is clor running right now?"

Read .airc.d/claude.zsh and report the three tier mappings. Do not run the ranking or edit anything — this is a question, not an instruction to re-pick.

Troubleshooting

rank_models.py returns few or no models

Cause: Artificial Analysis changed its payload shape. In November 2026 the leaderboard dropped its top-level slug; model/effort record IDs still joined it to the detail page. A release slug alone cannot distinguish effort variants. Solution: Re-run with --no-cache first — a truncated cached fetch looks the same. If still empty, inspect the cached HTML under $TMPDIR/agent-models/ for self.__next_f, "intelligenceIndex" (the leaderboard record key) and intelligenceIndexOutputTokensPerTask (the /models record key). Report that the parser needs updating rather than falling back to reading the rendered chart, which has no numbers in it.

A model is excluded as "no OpenRouter id"

Cause: Either it genuinely is not on OpenRouter, or the AA and OpenRouter names differ by more than the resolver's fallbacks handle. Solution: Check by hand against https://openrouter.ai/api/v1/models. If it exists, extend resolve() in the script. Never paste an id you have not seen in that API response — deepseek/deepseek-v4-pro and deepseek/deepseek-v4-pro-0813 are different builds, and the plain-looking one is the older one.

Pi warns Model "…" not found for provider "openrouter"

Cause: Pi's cached model catalog predates the model. It still runs, using the id verbatim. Solution: pi update refreshes the catalog. Do not hand-edit ~/.pi/agent/models-store.json — Pi overwrites it.

OpenCode still runs the old model after editing

Cause: loadout/settings/opencode.json is a loadout source. The file OpenCode reads is ~/.config/opencode/opencode.json, which is generated. Solution: Run loadout sync --global from an unsandboxed shell. Never edit the generated file — loadout sync refuses to overwrite a hand-edited output.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 alert1mo3 checks · Risk HIGH
  • Gen Agent Trust Hub1mo

    This skill manages AI model configurations by fetching data from external services and updating various local files. It includes instructions for the agent to bypass sandbox restrictions and modifies shell environment files, which poses a significant risk for persistence and arbitrary command execution.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 9fbe49f. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 4 days ago.

Activeupdated 6 days ago

README badge

README badge for nielsmadan/agentic-coding/agent-models