---
name: agent-models
description: Refresh model tiers for claude, codex and pix session shortcuts, the OpenRouter coding models for Pi, OpenCode and clor, or the extraction models for distill and jina-fetch. Uses Artificial Analysis intelligence versus cost per task for coding and measured benchmarks for extraction. Use for "pick models", "update the models", "refresh model tiers", "update session presets", "refresh extraction models", or questions about current model bindings. Cursor agent shortcuts are outside this skill's targets.
---

# Agent Models

Maintains three independent selections:

- **Session tiers:** `low`, `mid`, `high`, `xhigh`, `max` for `claude`, `codex`,
  and `pix`. Anthropic only for Claude; OpenAI only for Codex and pix.
- **OpenRouter coding:** `low`, `mid`, `high-main`, `high-fallback` for plain Pi,
  OpenCode and clor.
- **Extraction:** a measured default and fallback shared by `distill` and `jina-fetch`.

The following constraints apply to the **OpenRouter coding** selection. Session
tiers have their own rules in [session tiers](references/session-tiers.md).

- **Never pick a Grok / xAI model.** `rank_models.py` filters them out entirely.
- **Cap per-token price at $2/M input and $10/M output.** `rank_models.py`
  excludes anything above it (`--max-price-in` / `--max-price-out` to change).
  Real sessions bill by token, and a model past the cap made a poor
  high-fallback in November 2026 even where it won `$/task`.
- **Prefer alternative labs.** OpenAI, Anthropic and Google are eligible but only
  win a tier when clearly ahead of the best alternative; the point of these
  harnesses is to get a read on non-frontier-lab models.
- **Rank on Intelligence Index vs cost per Intelligence Index task**, the two
  axes of the chart of that name on `artificialanalysis.ai`. This replaced the
  Agentic Index in September 2026, when AA stopped publishing it per model. The
  Coding Agent Index (`/agents/coding-agents`) was considered and passed over: it
  covers ~19 harness+model pairs, so most candidates are missing from it.

## Instructions

A normal invocation refreshes all three selections: session tiers, OpenRouter
coding and extraction. Include the Claude, Codex and pix tiers in the same proposal,
application and verification as the existing models.
The existing `--coding` option limits a run to both coding selections;
`--extraction` limits it to extraction. A request naming particular harnesses
narrows the scope; it does not authorize refreshing the others. A read-only
question only inspects the current bindings.
Benchmark artifacts may be written before approval; model configuration may not,
unless the user's request already authorizes the picks or a choice among named alternatives.

### Step 1: Rank the candidates

Skip Steps 1 through 2a for `--extraction`.

```bash
python3 .claude/skills/agent-models/scripts/rank_models.py
```

It fetches `/leaderboards/models` (every model) and `/models` (a richer record —
effort, token counts — for a featured subset), parses both out of the RSC
payload each page server-renders, joins them on model/effort record `id`, resolves each model to an
exact OpenRouter id, and prints a table plus the cost/intelligence Pareto
frontier. Leaderboard-only records carry effort only in their name, which the
script parses (`GPT-6 Sol (low)`).
Add `--no-cache` to bypass the 6h fetch cache, `--all` to show excluded models,
`--json` for machine-readable output.

Do **not** read the chart through a fetcher instead. The page is JS-rendered:
`WebFetch` and `jina-fetch` both return prose with no numbers in it.

Read the `Frontier` and `Cost cliffs` sections — that is where the tiers are.

### Step 2: Choose four models

Work off the frontier, cheapest first. Marginal cost per index point is the
signal: a segment that buys many points cheaply is inside a tier, a segment that
buys almost nothing for a lot of money is the cliff between tiers.

| Tier | Rule |
|------|------|
| **low** | Cheapest frontier point worth running at all. Its natural home is the terse one-shot slots (`occli`, the haiku rung) — it only earns the default as well if it is *also* cheapest per input token. |
| **mid** | The next frontier point up. Usually the best value on the board, and usually the right default. |
| **high-main** | Highest intelligence index *before* the first cost cliff. Buying past a cliff is the mistake this ranking exists to prevent. |
| **high-fallback** | Best model from a **different creator** at comparable capability — within ~6 index points of high-main, at similar cost. A second opinion from the same lab is not a second opinion. |

**Prefer the frontier points themselves.** When the frontier has three points
below the cliff, that *is* the ladder — take them in order rather than reaching
off-frontier to fill a tier. A tier filled with a dominated model is worse than a
tier that repeats the one above it.

Apply the alternative-lab tiebreak: when a big-lab model and an alternative are
close in a tier, take the alternative. Take the big-lab model only when it is
clearly ahead at comparable-or-lower cost — "clearly ahead" means ahead on the
ranking axes, not ahead on one axis while behind on the others.

Note each pick's **effort** column (`max`, `xhigh`, `high`). The benchmark number
is for that effort level, so the effort has to carry into the config — a model
recorded at `max` and run at `low` is not the model that was measured.

### Sanity-check the ladder before presenting it

Four checks. Each one has caught a real bad pick; run all four.

1. **Cost must not invert.** low ≤ mid ≤ high-main on `$/task`. A mid that costs
   more than high-main is not a step up, it is a dominated pick — re-pick it, or
   collapse the tier into high-main and say so.
2. **No dominated pick wins a tier.** If a candidate is behind another on *both*
   index and cost, it cannot take a tier on any other merit. (Under the old
   Agentic Index, GPT-5.6 Sol nearly took mid at $0.953 while GLM-5.3 sat above
   it on the index *and* below it on price.)
3. **Check input price and tokens/task, not just `$/task`.** `$/task` is a
   benchmark blend. Real agentic sessions re-send their whole context every turn,
   so the bill tracks `price_in` far more closely than `$/task` — a terse model
   can win `$/task` and still be the more expensive one to actually run.
4. **Name which tier holds the default, out loud.** It is whichever tier is
   cheapest per input token among those that can carry everyday work — not
   automatically low. See `references/targets.md`.

### Getting tokens per task

The printed table has no token columns, and `--json` does not carry them either.
They exist only on the `/models` records (a featured subset) — leaderboard-only
models have none. Pull them with the script's own parser:

```python
import importlib.util
spec = importlib.util.spec_from_file_location("rm", ".claude/skills/agent-models/scripts/rank_models.py")
rm = importlib.util.module_from_spec(spec); spec.loader.exec_module(rm)
for r in rm.aa_models():
    otp = r.get("intelligenceIndexOutputTokensPerTask") or {}   # reasoning / answer / output
    can = r.get("canonicalIntelligenceIndexTokenCount") or {}   # suite totals, for the in:out ratio
    # input tokens per task = otp["output"] * can["input"] / can["output"]
```

`otp["output"]` checks out exactly against `cost.output ÷ price_out`, so the
derivation is sound. Use it whenever a tier decision turns on verbosity.

### Pricing a real workload

To compare candidates on how *this machine* actually bills rather than on the
benchmark blend, replay a measured session's token split at each model's prices.
`rl usage session <id>` gives the split; cache reads bill at roughly 12% of the
input rate (measured on OpenRouter, muse-spark: $0.15/M read vs $1.25/M input):

```
cost ≈ input_M × price_in  +  cache_read_M × 0.12 × price_in  +  output_M × price_out
```

Validate it against the session's own recorded cost before trusting it — pi
records per-turn `usage.cost`, so the replay can be checked to the cent.

### Step 2a: Choose the session tiers

Follow the ranking and selection rules in [session tiers](references/session-tiers.md)
for Claude's Anthropic ladder and the shared Codex/pix OpenAI ladder. Carry those
picks into the combined proposal in Step 3.

### Step 2b: Benchmark extraction candidates

Skip this step only for `--coding`. Read `docs/tests/extraction-models/README.md` and use the
small suite through the existing runner. Refresh exact model IDs, prices and
provider availability from OpenRouter; include `bin/extraction.py`'s default and
fallback as controls plus at most four candidates. Exclude Grok/xAI. Never choose
extraction models from the Intelligence Index alone.

```bash
python3 benchmark/bench-models.py --small benchmark/extraction-candidates.txt benchmark/runs/YYYY-MM-DD-extraction --repeats 3 --workers 3 --budget 5 --timeout 60
```

Use a fresh output directory for every run. The defaults bound each refresh to
144 requests and a $5 conservative spend guard; do not expand either without the
user asking. No automatic fallback runs in the benchmark. Inspect `summary.json`,
`responses.jsonl`, and `metadata.json`: incomplete runs, API errors, unknown costs,
and missing provider data remain explicit. Review every quote failure against the
saved source before calling it fabrication. Reject fabricated quotes and answers
to the wrong subject; among reliable candidates compare observed latency and cost.
Prefer an independently served fallback when its speed and quality are acceptable.
If all candidates pass, report that this small suite did not distinguish quality.

### Step 3: Present the proposal, and stop

Include the five session tiers per lab, with native model IDs, efforts, scores,
task costs and cliffs, alongside the OpenRouter and extraction proposals below.

Show a table: tier, model name, creator, intelligence index, cost per task, effort,
and exact OpenRouter id. Say in one line per tier why it won, and name the
runner-up so an override is a one-word reply.

Call out anything surprising — a tier that did not move, a pick that is off the
frontier, a cliff that shifted, a current model that has dropped out of the
ranking entirely.

For extraction, show full model names, exact IDs, passed/attempted counts, API
errors, latency, known cost and unknown-cost count, and provider names. Recommend
a default and fallback with the limits of the evidence. Link the saved results.

Then show the binding table from `references/targets.md` and **wait for
confirmation**. Do not change model configuration before the user approves the
picks and bindings for the requested scope. Approval of a coding ladder does not
authorize an unmentioned extraction-model change.

### Step 4: Apply

Apply the approved session tiers using [their targets](references/session-tiers.md#apply-and-verify).

Read `references/targets.md` and update the targets in the approved scope. It carries the exact key
names, prefix rules (`openrouter/` on some keys, not others), the `[1m]` context
suffix on `clor`, and which files are loadout sources vs generated.

Preserve unrelated content in every file: pi's `openai/*` entries, the
`clor` continuation backslashes, `CLOR_MODEL` / `CLOR_EFFORT` overrides.

For extraction, update `DEFAULT_MODEL` and `FALLBACK_MODEL` in
`bin/extraction.py`, which both CLIs import, and the matching `MODELS` pair in
`mods/dev-phase/pi/extension.ts` (see the extraction targets). Preserve `--model` overrides and
`JINA_FETCH_MODEL`. Update the current selection in `docs/tests/extraction-models/README.md`
with the supporting run link. Do not rerun the paid benchmark just to apply a pick.

### Step 5: Sync and verify

Run the [session-tier checks](references/session-tiers.md#apply-and-verify) for
session changes; those shell sources and the project skill need no loadout sync.

For extraction changes run:

```bash
python3 -m unittest discover -s bin -p 'test_extraction.py'
python3 -m unittest discover -s bin -p 'test_distill.py'
python3 -m unittest discover -s benchmark -p 'test_extraction_benchmark.py'
claude plugin test mods/dev-phase
jina-fetch --help
distill --help
```

No loadout sync is needed for extraction constants. For coding changes:


```bash
loadout sync --global     # required for opencode; needs an unsandboxed shell
source ~/.airc            # reloads clor / ocs / occli
loadout check --global    # must be clean
prat --config pratfall/config.toml config validate
```

`pratfall/config.toml` is symlinked to `~/.config/pratfall/config.toml`, so it
needs no sync — but a malformed profile only surfaces at the next `/second-opinion`
or `occli`, hence the validate.

`loadout sync` writes outside `~/wrksp`, so it needs `claude-raw` or a shell the
sandbox does not cover. If it fails on a grant, report it — do not relocate files.

Then confirm what landed:

```bash
grep -n "MODEL=\|CLOR_MODEL:-" .airc.d/claude.zsh   # 4 model slots + the start alias
grep -n "openrouter/" .airc.d/opencode.zsh pratfall/config.toml publish/overrides/second-opinion/SKILL.md
python3 -c "import json;d=json.load(open('loadout/settings/pi.json'));print(d['defaultModel'],d['defaultThinkingLevel'],d['enabledModels'])"
```

Confirm the start alias resolves to the default tier — that is the one edit with
no model id in it, so a stale value survives every id-based check.
Confirm the `advisor-*` profiles match the approved advisor models and reasoning
levels, preserving the user-selected overrides in `references/targets.md`. The
local `/second-opinion` skill names no model, but the public override in
`publish/overrides/` does — check both stayed in step.

Report the diff summary and leave the commit to the user.

## Examples

### Example 1: Routine refresh

User says: "refresh the model tiers"

1. Run `rank_models.py`. Frontier comes back
   `GPT-5.6 Luna ($0.049, 46.9) → GLM-5.3-Flash ($0.087, 58.2) →
   GLM-5.3 ($0.683, 59.1) → Claude Opus 5 ($2.337, 59.2)`, with a cliff flagged
   after GLM-5.3 at $23.23 per extra index point.
2. Three frontier points sit below the cliff, so they *are* the ladder: low =
   GPT-5.6 Luna (max), mid = GLM-5.3-Flash (max), high-main = GLM-5.3 (max).
   Luna is a frontier lab but takes low cleanly — the best alternative near its
   price is MiniMax-M3 at 36.1 for $0.139, so Luna is 10.8 points
   stronger *and* 2.8x cheaper. high-fallback = Qwen3.8 2.4T A95B — different
   lab, 57.1 at $0.807, beating Kimi K3 on both axes.
3. Run the four sanity checks. Costs rise across the ladder ✓, no pick is
   dominated ✓. But check 3 fires: Luna costs **more per input token** than mid
   ($0.20/M vs $0.15/M) and only wins `$/task` on terseness (136k tokens/task vs
   700k). So **mid holds the default** — low keeps `occli` and the haiku rung.
4. Present the four with runners-up, the default-tier call, and the binding
   table. Wait.
5. On approval: the six targets, `loadout sync --global`, `source ~/.airc`,
   `loadout check --global`.

Result: pi, OpenCode and `clor` default to GLM-5.3-Flash; `clor` opens on
`sonnet` so it lands on that default; subagents take it too; `occli` and the
haiku rung keep Luna.

### Example 2: Read-only question

User says: "what is clor running right now?"

Read `.airc.d/claude.zsh` and report the three tier mappings. Do not run the
ranking or edit anything — this is a question, not an instruction to re-pick.

## Troubleshooting

### `rank_models.py` returns few or no models

**Cause:** Artificial Analysis changed its payload shape. In November 2026 the
leaderboard dropped its top-level `slug`; model/effort record IDs still joined it
to the detail page. A release slug alone cannot distinguish effort variants.
**Solution:** Re-run with `--no-cache` first — a truncated cached fetch looks the
same. If still empty, inspect the cached HTML under `$TMPDIR/agent-models/` for
`self.__next_f`, `"intelligenceIndex"` (the leaderboard record key) and
`intelligenceIndexOutputTokensPerTask` (the `/models` record key). Report that the parser needs updating rather
than falling back to reading the rendered chart, which has no numbers in it.

### A model is excluded as "no OpenRouter id"

**Cause:** Either it genuinely is not on OpenRouter, or the AA and OpenRouter
names differ by more than the resolver's fallbacks handle.
**Solution:** Check by hand against `https://openrouter.ai/api/v1/models`. If it
exists, extend `resolve()` in the script. Never paste an id you have not seen in
that API response — `deepseek/deepseek-v4-pro` and
`deepseek/deepseek-v4-pro-0813` are different builds, and the plain-looking one
is the older one.

### Pi warns `Model "…" not found for provider "openrouter"`

**Cause:** Pi's cached model catalog predates the model. It still runs, using the
id verbatim.
**Solution:** `pi update` refreshes the catalog. Do not hand-edit
`~/.pi/agent/models-store.json` — Pi overwrites it.

### OpenCode still runs the old model after editing

**Cause:** `loadout/settings/opencode.json` is a loadout *source*. The file
OpenCode reads is `~/.config/opencode/opencode.json`, which is generated.
**Solution:** Run `loadout sync --global` from an unsandboxed shell. Never edit
the generated file — `loadout sync` refuses to overwrite a hand-edited output.
