---
name: inference-optimisation
description: >-
  Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
title: inference-optimisation
canonical_url: https://skilld.dev/gh/sybil-solutions/inference-optimisation-skill
last_updated: 2026-11-17T10:45:35.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [examples/ceiling-ornith-b390.json](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/examples/ceiling-ornith-b390.json), [examples/config-existing-server.json](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/examples/config-existing-server.json), [examples/config-sglang-docker.json](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/examples/config-sglang-docker.json), [references/ceilings.md](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/references/ceilings.md), [references/gotchas.md](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/references/gotchas.md), [references/levers.md](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/references/levers.md), [references/methodology.md](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/references/methodology.md), [references/publishing.md](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/references/publishing.md), [scripts/bench.py](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/scripts/bench.py), [scripts/ceiling.py](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/scripts/ceiling.py), [scripts/mem.sh](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/scripts/mem.sh), [scripts/queue.sh](https://skilld.dev/api/skills-raw/sybil-solutions/inference-optimisation-skill/inference-optimisation/scripts/queue.sh).
>
> If the user asked to install this Skill, run `npx skilld install sybil-solutions/inference-optimisation-skill/inference-optimisation`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/inference-optimisation`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/sybil-solutions/inference-optimisation-skill/inference-optimisation) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/inference-optimisation`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/inference-optimisation --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/inference-optimisation --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Inference optimisation (hill-climbing a model server)

You improve how fast a model server runs on one machine, with evidence.
The loop is:

1. Measure the baseline.
2. Calculate the ceiling.
3. Change one thing.
4. Measure it against controls.
5. Keep it or reject it, and write down why.
6. Repeat until the remaining gap needs work out of scope, then qualify and publish.

Detailed references, read when needed:

- `references/ceilings.md`: how to compute theoretical maxima. It includes a worked example.
- `references/methodology.md`: measurement pitfalls and statistics.
- `references/levers.md`: levers by area, ordered by typical payoff, with measured evidence.
- `references/gotchas.md`: engine- and hardware-specific traps.
- `references/publishing.md`: qualification gates, pinning, poster and report.
- `scripts/`: an engine-agnostic harness. It has a bench, ceiling calculator, memory watchdog, serialized queue and ledger.

## 0. Before touching anything

- **Write down the targets and constraints** the user gave. Ask only if one is missing and it changes what you do.
  - Metrics that matter: c1 decode, cN decode, prefill at several lengths, TTFT, startup.
  - Memory cap.
  - Context length that must stay stable.
  - Which quality changes are allowed: lossless only, or labelled lossy trades such as int8 prefill.
- **Inventory the machine:**
  - accelerator, memory type and size, and whether memory is unified;
  - driver, engine version, model, quantization, current launch command.
- **Measure, don't trust spec sheets:**
  - real memory bandwidth (device copy);
  - real matmul throughput (large GEMM);
  - clocks under load.
- **Read any existing experiment log for this machine and model.** Never rerun a rejected experiment without a new reason.
- **Create the working directory:**
  - `PLAN.md`: a timestamped log of every experiment, with numbers, verdict and reason.
  - `THEORY.md`: the ceilings.
  - `ledger.jsonl`: one line per run, from the harness.
  - `configs/`: one JSON per run.
  - `best-config.json`.

## 1. Ceilings first

Compute the theoretical maximum for each metric before optimising (`scripts/ceiling.py`, `references/ceilings.md`):

- **Decode is memory-bandwidth bound.** Count the bytes read per emitted token:
  - dense weights;
  - active experts: top-k ÷ experts × expert bytes;
  - LM head;
  - KV read at the working context.
  - Divide by measured bandwidth.
  - Speculative decoding divides the target's bytes by the accepted length. It also adds the draft's bytes, and the draft's LM head is often the biggest part.
- **Prefill is compute bound.** FLOPs per token ≈ 2 × active params, plus attention at about 4 × layers × hidden × position. Divide by ~50% of measured peak for a practical ceiling.
- **TTFT floor:** one forward pass plus tokenize and schedule.
- **Startup floor:** weight read from disk or page cache plus allocation, assuming JIT and graph caches persist.
- **KV:** bytes per token per layer × the layers that actually have KV (hybrid models have few). That gives how many full-length sequences fit under the memory cap.

Report progress as a percentage of the practical ceiling. If the user's target is above the ceiling, say so early, with the arithmetic.

## 2. Ground rules (non-negotiable)

1. **One change per experiment.** Give each its own config file. Derive new configs from the current best.
2. **Control for noise.** Run-to-run noise is typically ±5–10%.
   - Take ≥6 decode samples per run and use the median.
   - Run controls on both sides of each candidate: control, candidate, control.
   - Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%.
   - Repeat any single surprising win.
3. **Keep the environment fixed.** Stay on AC power, on a fixed performance profile, with an idle host and no other GPU jobs.
   - Make the harness refuse to run otherwise.
   - Record the state at the end of each run, and void the run if it changed.
   - Restore any profile you changed when you finish, and tell the user.
4. **Run one GPU job at a time.** Serialize them with a lock (`scripts/queue.sh`).
   - Never use `pkill -f`/`pgrep -f` patterns that can match your own shell.
   - Detach long queues (`setsid nohup … &`) so they survive agent restarts.
5. **Enforce the memory cap with a watchdog** that kills the run.
   - On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g. `drm-resident-*` in `/proc/<pid>/fdinfo`, `nvidia-smi`, or `ioreg`/`vm_stat` on Apple) plus process anonymous memory.
6. **Stability gates.** A config is only "best" after it passes:
   - a long-context retrieval run, with needles in a ~50% then ~100% context document, found exactly;
   - a soak with concurrent full-length requests, if the KV pool is sized for N users;
   - a greedy output equivalence check, if the change claims to be lossless.
7. **Never force-kill a server mid-batch.** Abort requests, wait for the drain, then stop with a timeout. A forced stop mid-prefill has caused GPU driver faults.
8. **Verify outputs, not just speed.** Flag degenerate or looping output, failed requests and an idle GPU. A suspiciously fast result is usually broken output.
9. **Log everything in `PLAN.md`,** including failures and why they failed. Negative results are the most reusable output.
10. **Secrets:** never ask twice for a credential the user already saved. Never print it, and never copy it or its path into anything published.

## 3. The loop

```
baseline → profile → pick the lever with the largest (ceiling gap × likelihood) → config → A/B with controls
   ↑                                                                                         ↓
   └──────── update best-config + PLAN.md + ledger ←──── accept / reject with reason ←───────┘
```

### Lever order

Cheapest first; see `references/levers.md` for evidence.

1. **Persistent JIT, compile and graph caches** (startup −50–70%). Also use eager safetensors loading instead of mmap where it is slow.
2. **Decode graphs** (CUDA/XPU graphs): often 2× c1 on their own.
3. **Speculative decoding:**
   - Choose the method (built-in MTP, EAGLE, a draft model, n-gram).
   - Sweep draft length per concurrency and context; the best length differs.
   - Prune the draft LM head's vocabulary.
   - Keep sampling on the GPU.
   - Make sure verify runs with decode-style attention.
4. **Concurrency and KV sizing:**
   - Set max running requests and the KV pool to N × context.
   - For hybrid SSM/GDN models, size the state slots. They, not KV, often cap concurrency.
   - Explicitly enable prefix caching and confirm `cached_tokens` > 0.
5. **Prefill:**
   - Tune chunk size; it trades throughput against decode stalls.
   - Use a faster attention library.
   - Use int8/fp8 prefill GEMMs as a labelled quality trade.
   - Avoid padding cliffs in grouped MoE GEMMs.
6. **Kernel thread and tile sizes:** defaults are often tuned for a bigger card.
7. **Kernel work:** profile, read the ISA, fix the dominant kernel. Do this only after config levers plateau.

### When a lever fails

- **If kernel-count reductions (fusion) give nothing** under graphs, the time is inside the main kernels. Stop fusing.
- **If higher acceptance doesn't give higher tok/s,** the draft costs too much. Measure per-step time.
- **If a weird result appears,** suspect the measurement first: tokens counted per streamed chunk, prompt length not measured with the real tokenizer, cache hits, thermal state, or the battery.

### Plateau

Declare a plateau when three consecutive levers land inside noise. Write a STATUS section in `PLAN.md` with:

- best numbers vs baseline vs ceiling;
- what limits further gains, with evidence;
- a ranked list of the remaining levers and their estimated cost.

Then keep the loop alive at a slower cadence if the user said "don't stop", or hand back.

## 4. Qualify and publish

See `references/publishing.md`.

- Pin weights by revision, engine by image digest or source commit, and kernels by commit. Build the published artifact from those pins in a clean checkout, never from a dirty tree.
- Run the acceptance gates on the clean build: load, chat, reasoning, tools round-trip, a long-context needle, and speed.
- Then A/B the clean build against your best image to prove they match.
- Report which numbers come from which harness. Gate numbers (e.g. thinking on, first 30 s) and hill-climb numbers (thinking off, medians) are not comparable.

## 5. Reporting to the user

- **Start with a table:** metric, baseline, now, ceiling, how it was measured (median of N, cold or warm, which harness).
- **Then:** what got the gains, why it stops here, the known limitations, and the system state you changed (power profile, running servers, uncommitted code, branches that must not be deleted).
- **Say plainly** what was not verified.
- **When asked to free the GPU:** drain, stop the server, confirm no processes and an idle clock, and end any loop.
