Benchmark
Run the benchmarks and update benchmark/README.md (and wasm-size/README.md).
Prerequisites
mise run on-task-startedvendor/wasmtimesubmodule must exist (the SessionStart hook handles it; otherwisegit submodule update --init --recommend-shallow vendor/wasmtime).- http-routing needs
oha(cargo install oha);bunis mise-managed. - gale-gen's and sqlite-parse's ANTLR4 references need
javaandjavac, whichbenchmark/mise.tomlprovides; the jar is fetched to~/.cache/gale. Those rows are skipped if the tool is absent. - wasm-size needs
rustup target add wasm32-wasip1and Moonbit (curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash, thenmoon updatein eachwasm-size/*dir).
Procedure
- Record the PGO profiles again first:
scripts/dev-bin.sh wado test --pgo benchmark package-gale-highlight-html, and commit the profiles it rewrites. - Run
mise run benchmark-allthree times, each to its own log inscratchpad/, then pick per row withmise exec -- node benchmark/pick.ts scratchpad/run{1,2,3}.log(throttling only ever slows things down). Use that tool rather than reading the logs by eye: it keys rows by (task, implementation, phase) and selects on ms/iter, so a rate that rounds to a tie across runs cannot pair with the wrong ms/iter. Which benchmarks run, and in what order, isbenchmark/mise.toml'salltask. - Run http-routing separately (needs
oha+ pinned cores):SLICE=10 ROUNDS=3 SHAPES="1 4" mise run benchmark-http-routing. It keeps the per-(server, request) max internally, so one invocation suffices, and it measures one worker shape per entry inSHAPES. AddHEADROOM_CHECK=1whenCONNECTIONS_PER_WORKER,OHA_CORE_COUNTorSHAPESchanged, to confirmohawas not the ceiling. - Refresh the README Environment line versions:
mise exec -- node --version,mise -C benchmark exec -- bun --version,rustc --version,cc --version | head -1.mise -C benchmark lsnames wasi-sdk, and wasmtime's version is thevendor/wasmtimeworkspace version. - Update the tables, following README.md's existing layout. http-routing is
req/s (higher is better), one table per worker shape. Keep measured figures
inside tables — prose around them is not re-measured and drifts. An input
size in the prose comes from the program's own header line, such as
gale-gen (Rust): 42786 bytes, so compare each one with the log. - wasm-size:
mise run report-wasm-size, then updatewasm-size/README.md.mise -C wasm-size lsnames Zig, andmoon versionnames Moonbit. - Compare every Wado row with the README's previous figure, and attribute each one that moved (see PGO). Report those to the user.
PGO
A Wado row in the procedure is a PGO build: wado run and wado compile apply
the profile in the benchmark's pgo/ by default. benchmark-all runs each one
again as its (no-pgo) row, which wado_rows in benchmark/wado.sh prints, and
the tables list both. http-routing has no such row, and the README says so.
wasm-size has no profile, so a size change is the compiler's or the program's.
The README's PGO section compares the two builds by wall clock. It records no
fuel: benchmark/fuel.json is the one record of that.
Compare them with
mise exec -- node benchmark/ab.ts --base <no-pgo logs> --head <PGO logs>. Its
verdict gives the table's noise. It keys rows by implementation, so split a
benchmark-all log into one log per build first.
A profile can move a row through what the program allocates, because a change of
allocation moves where collections land. wado run --gc-log shows that, and
microgpt needs a new heap sweep when it happens.
Two figures from different sessions can differ by noise alone, so attribute a
Wado row that moved by an A/B in one session: the compiler that measured the
README against this one, with WADO_BIN and ab.ts (see the wado-performance
skill). If the verdict is noise, the row did not move. Otherwise read the
verdict on its (no-pgo) row. If that moved too, the compiler moved it apart
from PGO. If not, PGO moved it: either the row's pgo/ file, or how the compiler
uses a profile, such as the thresholds in pgo.rs or the inliner's hot price.
git log on each shows what changed, and when both did, isolate them before
naming one. http-routing has no (no-pgo) row, so its A/B, which
benchmark/http_routing/README.md states, ends at the first verdict.
mise run update-baselines records the profiles again, so a branch's
profiles match its code once it goes up for review.
Sweeping a compiler knob
WADO_BENCH_FLAGS is appended to every wado compile / wado run the harness
issues, so an arm costs a benchmark run rather than a release rebuild:
set -e # a failed arm would leave pick.ts choosing among the rest
for t in 13 20 32; do
WADO_BENCH_FLAGS="--optimize-inline-threshold $t" mise run benchmark-all > scratchpad/thr$t.log 2>&1
done
mise exec -- node benchmark/pick.ts scratchpad/thr{13,20,32}.logRead the sweep with pick.ts the same way as a best-of-three: it keys rows by
(task, implementation, phase), so the "best" column names the winning arm per
row. Only a knob both wado compile and wado run accept can be swept this
way, since the harness passes the flags to both. A knob only the engine reads,
such as --function-alignment, goes in WADO_RUN_FLAGS, which the harness
appends to wado run and wado serve alone. The knob a sweep settles on is
a default in optimize.rs, not a flag the README's numbers were taken under —
re-run the suite unflagged before updating the tables.
Comparing the settled default against origin/main is a different measurement,
and WADO_BIN plus ab.ts is how: see the wado-performance skill.
Reading output
Each program prints a throughput line — <rate> <unit>/s (<ms> ms/iter, <n> iter) — per phase (zlib prints two phases, Compress:/Decompress:).
Read the rate and the ms/iter straight off; the iteration count auto-calibrates
to ~1s, so there is no total to report. The unit is the benchmark's own
(numbers/s, px/s, conversions/s, MB/s, req/s); vs best = fastest rate / this
rate.
Report a change between two measurements with improvement as positive: +9%
means faster, -13% slower. This is the sign ab.ts prints.
Each benchmark prints the implementations it compared, so read them off the run
rather than from a list here. Two are conditional: the ANTLR4 rows for
sqlite-parse and gale-gen are skipped when java is absent, and the run
says SKIP: when it drops one.
Workload sizing
Benchmarks auto-calibrate their iteration count to run ~1s, so no manual denomination is needed. A workload whose single iteration approaches that reports one iteration and stops averaging — shrink its problem size, across every language implementation of that benchmark, until it calibrates again.
Notes
- Tool versions come from mise, not the system.
- Cloud VMs are noisy; best-of-three absorbs the drift.