Index of every page on this site: llms.txt
All skills
remsky avatar

/readme-benchmarks

@ffa0691
by Jeremyremsky/kokoro-fastapi5.7k stars
951
Run it yourself

Running the Kokoro-FastAPI benchmark + transcription-roundtrip suites and regenerating the README plots. Use when asked to run/refresh benchmarks, RTF/first-token plots, transcription sanity checks, or the long-form baseline.

  • 1 file
  • 6.7 KB
  • Updated 2 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/remsky/kokoro-fastapi/readme-benchmarks

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈61 tokens for metadata: the name and description. ≈1.6k when used: this file.

Description uses 3.1% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈245 characters, or 3.1%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Read AGENTS.md and CONTRIBUTING.md first, house rules live there.

Benchmarks + README plots

Three suites under examples/assorted_checks/ feed the README:

  • test_transcription/ - synth with a running server, transcribe with faster-whisper, report WER/CER. Short, multilingual, and long-form.
  • benchmarks/ - RTF (processing time vs tokens) and first-token latency/timeline plots. These feed the README performance images.
  • test_dialogue/ - multi-speaker functional checks plus the turn-length / text-length throughput sweeps. Run commands and flags live in test_dialogue/README.md; bench then plot with plot_dialogue_bench.py.

Prereqs

  • A Kokoro server on :8880. GPU and CPU docker images both bind 8880, so swap images, never run both. Set KOKORO_DEVICE / BENCH_PREFIX to match whichever is up.
  • Warm the server first. cuDNN autotune cold-start inflates the first GPU run (heavy voices ~1.2s vs ~0.3s warm). Hit a couple voices (e.g. af_heart, zf_xiaobei) before capturing, or discard the first pass.
  • examples/ has its own uv venv. Run everything from inside examples/ (cd examples), not the root .venv. The first-token script also needs examples/ as cwd for its audio path.
cd examples
uv sync --extra transcription --extra transcription-gpu --extra benchmarks

Drop --extra transcription-gpu to skip the ~1.2 GB cuDNN/cuBLAS download and transcribe on CPU.

Run (from examples/)

# short English per-voice sanity (base.en, WER)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription.py

# multilingual (small model, CER for ja/zh)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription_multilingual.py

# long-form book roundtrip (the baseline). see BASELINE.md
LONGFORM_CHARS=65000 WHISPER_DEVICE=cuda KOKORO_DEVICE=gpu \
  uv run python assorted_checks/test_transcription/test_long_form.py

# RTF plots
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_tts_rtf.py

# first-token latency/timeline plots
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_first_token_stream_unified.py

For a CPU capture: swap to the CPU image, then rerun with KOKORO_DEVICE=cpu / BENCH_PREFIX=cpu (and WHISPER_DEVICE=cpu for long-form). Long-form Windows wrapper: run_long_form.bat [short|full|synth|transcribe].

Env vars

Var Applies to Notes
KOKORO_DEVICE transcription Label only (server doesn't self-report device). Sets report filename report_{device}.json + meta header so a cpu capture can't overwrite a gpu baseline. Default gpu.
BENCH_PREFIX benchmarks Prefixes RTF output files and picks the token sweep (gpu: dense to 1000; cpu: dense to 500). RTF default gpu, first-token default cpu.
WHISPER_DEVICE / WHISPER_COMPUTE transcription Transcribe device is independent of synth device. Scripts default cpu/int8; long-form .bat uses cuda/float16.
WHISPER_MODEL transcription base.en short, small multilingual.
LONGFORM_CHARS / LONGFORM_VOICE / LONGFORM_INPUT long-form Char cap, voice, input book.

Gotchas

  • First-token plots save under GENERIC names, not BENCH_PREFIX. BENCH_PREFIX only changes the (misleading) "Results saved to" print, neither file names nor plot titles carry it. Actual files are first_token_{timeline,latency}_stream{,_openai}.png. So a GPU run overwrites the CPU generic file. To keep both, copy the generic file to its device-named asset between runs (CPU run to copy to GPU run).
  • Reports are split _gpu / _cpu with a meta header. Don't let a cpu run clobber a gpu file. If you see a bare report.json / long_form_report.json, it predates the split.
  • transcribe_seconds is constant across kokoro cpu/gpu unless you change WHISPER_DEVICE. Only synth_seconds tracks the kokoro device.
  • Regression bands live in test_transcription/BASELINE.md (WER < 0.07, synth >= 25x rt warm, transcribe >= 40x CUDA / 13-17x CPU int8).
  • Replotting doesn't need a server. {prefix}_benchmark_results_rtf.json keeps processing_time + output_length at full precision, so you can rebuild any RTF plot from the saved data (recompute rtf = processing_time / output_length, feed plot_correlation). No re-synth. Note the stored rtf field is rounded to 2 decimals for the stats text, too coarse to plot, recompute it.
  • RTF gets a mean line, not a regression. plot_correlation(..., show_trend=False) for the RTF plot: rtf is ~constant vs input size, so a fitted slope over-reads single-run noise (each token count is n=1). Processing-time keeps show_trend=True, it's genuinely linear (corr ~0.99). First-token plots use plot_timeline and impose no trend at all.

Output paths

Suite Files
short test_transcription/output/report_{device}.json + wavs
multilingual test_transcription/output_multilingual/report_{device}.json
long-form test_transcription/output_long_form/long_form_report_{device}.json + wav + transcript + *.synth_meta.json sidecar
RTF benchmarks/output_plots/{prefix}_{processing_time,realtime_factor,system_usage}_rtf.png; data in benchmarks/output_data/{prefix}_benchmark_{results,stats}_rtf.*
first-token benchmarks/output_plots/first_token_{timeline,latency}_stream{,_openai}.png, total_time_latency_stream{,_openai}.png

README asset mapping

Copy plot outputs to assets/ under the README's names (No stamp):

README asset Source plot
assets/gpu_processing_time.png output_plots/gpu_processing_time_rtf.png
assets/gpu_realtime_factor.png output_plots/gpu_realtime_factor_rtf.png
assets/gpu_first_token_timeline_openai.png output_plots/first_token_timeline_stream_openai.png (GPU run; name drops "stream")
assets/cpu_first_token_timeline_stream_openai.png output_plots/first_token_timeline_stream_openai.png (CPU run)
assets/gpu_dialogue_turn_length.png test_dialogue/output/dialogue_turn_length.png (GPU run; KOKORO_DEVICE only labels the caption)
assets/gpu_dialogue_text_length.png test_dialogue/output/dialogue_text_length.png (same run as above)

The README shows all six across three blocks (grep the asset name to find each), plus gpu_model_unload_{short,longform}.png from benchmarks/benchmark_model_unload.py. Other gpu_first_token_* assets exist but aren't displayed.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at ffa0691. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago

README badge

README badge for remsky/kokoro-fastapi/readme-benchmarks