Read AGENTS.md and CONTRIBUTING.md first, house rules live there.
Benchmarks + README plots
Three suites under examples/assorted_checks/ feed the README:
test_transcription/- synth with a running server, transcribe with faster-whisper, report WER/CER. Short, multilingual, and long-form.benchmarks/- RTF (processing time vs tokens) and first-token latency/timeline plots. These feed the README performance images.test_dialogue/- multi-speaker functional checks plus the turn-length / text-length throughput sweeps. Run commands and flags live intest_dialogue/README.md; bench then plot withplot_dialogue_bench.py.
Prereqs
- A Kokoro server on
:8880. GPU and CPU docker images both bind 8880, so swap images, never run both. SetKOKORO_DEVICE/BENCH_PREFIXto match whichever is up. - Warm the server first. cuDNN autotune cold-start inflates the first GPU run (heavy voices ~1.2s vs ~0.3s warm). Hit a couple voices (e.g.
af_heart,zf_xiaobei) before capturing, or discard the first pass. examples/has its own uv venv. Run everything from insideexamples/(cd examples), not the root.venv. The first-token script also needsexamples/as cwd for its audio path.
cd examples
uv sync --extra transcription --extra transcription-gpu --extra benchmarksDrop --extra transcription-gpu to skip the ~1.2 GB cuDNN/cuBLAS download and transcribe on CPU.
Run (from examples/)
# short English per-voice sanity (base.en, WER)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription.py
# multilingual (small model, CER for ja/zh)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription_multilingual.py
# long-form book roundtrip (the baseline). see BASELINE.md
LONGFORM_CHARS=65000 WHISPER_DEVICE=cuda KOKORO_DEVICE=gpu \
uv run python assorted_checks/test_transcription/test_long_form.py
# RTF plots
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_tts_rtf.py
# first-token latency/timeline plots
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_first_token_stream_unified.pyFor a CPU capture: swap to the CPU image, then rerun with KOKORO_DEVICE=cpu / BENCH_PREFIX=cpu (and WHISPER_DEVICE=cpu for long-form). Long-form Windows wrapper: run_long_form.bat [short|full|synth|transcribe].
Env vars
| Var | Applies to | Notes |
|---|---|---|
KOKORO_DEVICE |
transcription | Label only (server doesn't self-report device). Sets report filename report_{device}.json + meta header so a cpu capture can't overwrite a gpu baseline. Default gpu. |
BENCH_PREFIX |
benchmarks | Prefixes RTF output files and picks the token sweep (gpu: dense to 1000; cpu: dense to 500). RTF default gpu, first-token default cpu. |
WHISPER_DEVICE / WHISPER_COMPUTE |
transcription | Transcribe device is independent of synth device. Scripts default cpu/int8; long-form .bat uses cuda/float16. |
WHISPER_MODEL |
transcription | base.en short, small multilingual. |
LONGFORM_CHARS / LONGFORM_VOICE / LONGFORM_INPUT |
long-form | Char cap, voice, input book. |
Gotchas
- First-token plots save under GENERIC names, not
BENCH_PREFIX.BENCH_PREFIXonly changes the (misleading) "Results saved to" print, neither file names nor plot titles carry it. Actual files arefirst_token_{timeline,latency}_stream{,_openai}.png. So a GPU run overwrites the CPU generic file. To keep both, copy the generic file to its device-named asset between runs (CPU run to copy to GPU run). - Reports are split
_gpu/_cpuwith ametaheader. Don't let a cpu run clobber a gpu file. If you see a barereport.json/long_form_report.json, it predates the split. transcribe_secondsis constant across kokoro cpu/gpu unless you changeWHISPER_DEVICE. Onlysynth_secondstracks the kokoro device.- Regression bands live in
test_transcription/BASELINE.md(WER < 0.07, synth >= 25x rt warm, transcribe >= 40x CUDA / 13-17x CPU int8). - Replotting doesn't need a server.
{prefix}_benchmark_results_rtf.jsonkeepsprocessing_time+output_lengthat full precision, so you can rebuild any RTF plot from the saved data (recomputertf = processing_time / output_length, feedplot_correlation). No re-synth. Note the storedrtffield is rounded to 2 decimals for the stats text, too coarse to plot, recompute it. - RTF gets a mean line, not a regression.
plot_correlation(..., show_trend=False)for the RTF plot: rtf is ~constant vs input size, so a fitted slope over-reads single-run noise (each token count is n=1). Processing-time keepsshow_trend=True, it's genuinely linear (corr ~0.99). First-token plots useplot_timelineand impose no trend at all.
Output paths
| Suite | Files |
|---|---|
| short | test_transcription/output/report_{device}.json + wavs |
| multilingual | test_transcription/output_multilingual/report_{device}.json |
| long-form | test_transcription/output_long_form/long_form_report_{device}.json + wav + transcript + *.synth_meta.json sidecar |
| RTF | benchmarks/output_plots/{prefix}_{processing_time,realtime_factor,system_usage}_rtf.png; data in benchmarks/output_data/{prefix}_benchmark_{results,stats}_rtf.* |
| first-token | benchmarks/output_plots/first_token_{timeline,latency}_stream{,_openai}.png, total_time_latency_stream{,_openai}.png |
README asset mapping
Copy plot outputs to assets/ under the README's names (No stamp):
| README asset | Source plot |
|---|---|
assets/gpu_processing_time.png |
output_plots/gpu_processing_time_rtf.png |
assets/gpu_realtime_factor.png |
output_plots/gpu_realtime_factor_rtf.png |
assets/gpu_first_token_timeline_openai.png |
output_plots/first_token_timeline_stream_openai.png (GPU run; name drops "stream") |
assets/cpu_first_token_timeline_stream_openai.png |
output_plots/first_token_timeline_stream_openai.png (CPU run) |
assets/gpu_dialogue_turn_length.png |
test_dialogue/output/dialogue_turn_length.png (GPU run; KOKORO_DEVICE only labels the caption) |
assets/gpu_dialogue_text_length.png |
test_dialogue/output/dialogue_text_length.png (same run as above) |
The README shows all six across three blocks (grep the asset name to find each), plus gpu_model_unload_{short,longform}.png from benchmarks/benchmark_model_unload.py. Other gpu_first_token_* assets exist but aren't displayed.