---
name: readme-benchmarks
description: "Running the Kokoro-FastAPI benchmark + transcription-roundtrip suites and regenerating the README plots. Use when asked to run/refresh benchmarks, RTF/first-token plots, transcription sanity checks, or the long-form baseline."
title: readme-benchmarks
canonical_url: https://skilld.dev/gh/remsky/kokoro-fastapi/readme-benchmarks
last_updated: 2026-11-12T03:45:43.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> If the user asked to install this Skill, run `npx skilld install remsky/kokoro-fastapi/readme-benchmarks`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/readme-benchmarks`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/remsky/kokoro-fastapi/readme-benchmarks) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/readme-benchmarks`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/readme-benchmarks --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/readme-benchmarks --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

Read [AGENTS.md](../../../AGENTS.md) and [CONTRIBUTING.md](../../../CONTRIBUTING.md) first, house rules live there.

# Benchmarks + README plots

Three suites under `examples/assorted_checks/` feed the README:

- `test_transcription/` - synth with a running server, transcribe with faster-whisper, report WER/CER. Short, multilingual, and long-form.
- `benchmarks/` - RTF (processing time vs tokens) and first-token latency/timeline plots. These feed the README performance images.
- `test_dialogue/` - multi-speaker functional checks plus the turn-length / text-length throughput sweeps. Run commands and flags live in `test_dialogue/README.md`; bench then plot with `plot_dialogue_bench.py`.

## Prereqs

- A Kokoro server on `:8880`. GPU and CPU docker images both bind 8880, so swap images, never run both. Set `KOKORO_DEVICE` / `BENCH_PREFIX` to match whichever is up.
- **Warm the server first.** cuDNN autotune cold-start inflates the first GPU run (heavy voices ~1.2s vs ~0.3s warm). Hit a couple voices (e.g. `af_heart`, `zf_xiaobei`) before capturing, or discard the first pass.
- `examples/` has its own uv venv. Run everything from **inside `examples/`** (`cd examples`), not the root `.venv`. The first-token script also needs `examples/` as cwd for its audio path.

```bash
cd examples
uv sync --extra transcription --extra transcription-gpu --extra benchmarks
```

Drop `--extra transcription-gpu` to skip the ~1.2 GB cuDNN/cuBLAS download and transcribe on CPU.

## Run (from `examples/`)

```bash
# short English per-voice sanity (base.en, WER)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription.py

# multilingual (small model, CER for ja/zh)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription_multilingual.py

# long-form book roundtrip (the baseline). see BASELINE.md
LONGFORM_CHARS=65000 WHISPER_DEVICE=cuda KOKORO_DEVICE=gpu \
  uv run python assorted_checks/test_transcription/test_long_form.py

# RTF plots
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_tts_rtf.py

# first-token latency/timeline plots
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_first_token_stream_unified.py
```

For a CPU capture: swap to the CPU image, then rerun with `KOKORO_DEVICE=cpu` / `BENCH_PREFIX=cpu` (and `WHISPER_DEVICE=cpu` for long-form). Long-form Windows wrapper: `run_long_form.bat [short|full|synth|transcribe]`.

## Env vars

| Var | Applies to | Notes |
| --- | --- | --- |
| `KOKORO_DEVICE` | transcription | Label only (server doesn't self-report device). Sets report filename `report_{device}.json` + `meta` header so a cpu capture can't overwrite a gpu baseline. Default `gpu`. |
| `BENCH_PREFIX` | benchmarks | Prefixes RTF output files and picks the token sweep (gpu: dense to 1000; cpu: dense to 500). RTF default `gpu`, first-token default `cpu`. |
| `WHISPER_DEVICE` / `WHISPER_COMPUTE` | transcription | Transcribe device is independent of synth device. Scripts default `cpu`/`int8`; long-form `.bat` uses `cuda`/`float16`. |
| `WHISPER_MODEL` | transcription | `base.en` short, `small` multilingual. |
| `LONGFORM_CHARS` / `LONGFORM_VOICE` / `LONGFORM_INPUT` | long-form | Char cap, voice, input book. |

## Gotchas

- **First-token plots save under GENERIC names**, not `BENCH_PREFIX`. `BENCH_PREFIX` only changes the (misleading) "Results saved to" print, neither file names nor plot titles carry it. Actual files are `first_token_{timeline,latency}_stream{,_openai}.png`. So a GPU run overwrites the CPU generic file. To keep both, copy the generic file to its device-named asset **between** runs (CPU run to copy to GPU run).
- Reports are split `_gpu` / `_cpu` with a `meta` header. Don't let a cpu run clobber a gpu file. If you see a bare `report.json` / `long_form_report.json`, it predates the split.
- `transcribe_seconds` is constant across kokoro cpu/gpu unless you change `WHISPER_DEVICE`. Only `synth_seconds` tracks the kokoro device.
- Regression bands live in `test_transcription/BASELINE.md` (WER < 0.07, synth >= 25x rt warm, transcribe >= 40x CUDA / 13-17x CPU int8).
- **Replotting doesn't need a server.** `{prefix}_benchmark_results_rtf.json` keeps `processing_time` + `output_length` at full precision, so you can rebuild any RTF plot from the saved data (recompute `rtf = processing_time / output_length`, feed `plot_correlation`). No re-synth. Note the stored `rtf` field is rounded to 2 decimals for the stats text, too coarse to plot, recompute it.
- **RTF gets a mean line, not a regression.** `plot_correlation(..., show_trend=False)` for the RTF plot: rtf is ~constant vs input size, so a fitted slope over-reads single-run noise (each token count is n=1). Processing-time keeps `show_trend=True`, it's genuinely linear (corr ~0.99). First-token plots use `plot_timeline` and impose no trend at all.

## Output paths

| Suite | Files |
| --- | --- |
| short | `test_transcription/output/report_{device}.json` + wavs |
| multilingual | `test_transcription/output_multilingual/report_{device}.json` |
| long-form | `test_transcription/output_long_form/long_form_report_{device}.json` + wav + transcript + `*.synth_meta.json` sidecar |
| RTF | `benchmarks/output_plots/{prefix}_{processing_time,realtime_factor,system_usage}_rtf.png`; data in `benchmarks/output_data/{prefix}_benchmark_{results,stats}_rtf.*` |
| first-token | `benchmarks/output_plots/first_token_{timeline,latency}_stream{,_openai}.png`, `total_time_latency_stream{,_openai}.png` |

## README asset mapping

Copy plot outputs to `assets/` under the README's names (No stamp):

| README asset | Source plot |
| --- | --- |
| `assets/gpu_processing_time.png` | `output_plots/gpu_processing_time_rtf.png` |
| `assets/gpu_realtime_factor.png` | `output_plots/gpu_realtime_factor_rtf.png` |
| `assets/gpu_first_token_timeline_openai.png` | `output_plots/first_token_timeline_stream_openai.png` (GPU run; name drops "stream") |
| `assets/cpu_first_token_timeline_stream_openai.png` | `output_plots/first_token_timeline_stream_openai.png` (CPU run) |
| `assets/gpu_dialogue_turn_length.png` | `test_dialogue/output/dialogue_turn_length.png` (GPU run; `KOKORO_DEVICE` only labels the caption) |
| `assets/gpu_dialogue_text_length.png` | `test_dialogue/output/dialogue_text_length.png` (same run as above) |

The README shows all six across three blocks (grep the asset name to find each), plus `gpu_model_unload_{short,longform}.png` from `benchmarks/benchmark_model_unload.py`. Other `gpu_first_token_*` assets exist but aren't displayed.
