Corpus Oracles, Performance Budgets, Scorecard & CI Reference
This reference provides step-by-step instructions for working with an engine-first egui application's real-file corpora, performance benchmark suite, generated scorecard, and CI pipeline.
1. Real-File Test Corpora (cargo xtask corpus & test-corpus)
Real-world test files are gitignored under corpus/, pinned in xtask/src/corpus_pins.rs, and verified against SHA-256 manifests (xtask/*.sha256).
Corpus Design Pattern
| Component | Location | Purpose |
|---|---|---|
| Pin Definitions | xtask/src/corpus_pins.rs |
Pins external repository commits or archive URLs for each format suite (e.g., --<suite>). |
| SHA-256 Manifests | xtask/<suite>-corpus.sha256 |
Verifies every downloaded file's cryptographic hash before running tests. |
| Corpus Test Runner | crates/io/tests/corpus.rs |
Defines Source structs with pass_floor and roundtrip_floor ratchets. |
| Diagnostic Diff Tool | crates/io/examples/oracle_diff.rs |
Inspects differences between the application's output/render and the reference oracle for a single file. |
Running & Debugging Corpus Tests
# Check status of all corpora
cargo xtask corpus
# Fetch all standard corpora
cargo xtask corpus --all
# Run corpus tests faster by disabling thin LTO (matches CI's build speedup)
CARGO_PROFILE_RELEASE_LTO=false CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16 \
cargo xtask test-corpus -p <app>-io -- --nocapture
# Inspect a single file's diff against its reference oracle
cargo run --release -p <app>-io --example oracle_diff -- \
corpus/<suite>/<file>.extMulti-Level Oracle Verification
- Import & Reference Oracle (
crates/io/tests/corpus.rs): Imports the external file into adoc::Document, evaluates or renders it, and compares against the reference oracle within a defined tolerance. Also verifiesexport -> re-importround-trip fidelity and structural invariants. - Full Engine Re-Evaluation Oracle (
crates/engine/tests/*_oracles.rs): Discards any pre-cached previews or baked outputs stored in the source file, re-evaluates all procedural/computed elements throughcrates/engine, and compares against the reference composite/output.
Testing Local Corpus Changes & Bumping Pins
- Clone the external corpus repository as
../<app>-corpus(or set the corresponding*_CORPUS_REPOenvironment variable). - Test uncommitted/local corpus files with
cargo xtask test-corpus --local. - Once pushed to the corpus repository, update the pinned commit constant in
xtask/src/corpus_pins.rs. - Run
cargo xtask corpus --<suite> --update-manifest, runcargo xtask test-corpus, raise any improved floors incrates/io/tests/corpus.rs, and regeneratecargo xtask scorecard.
2. Performance Benchmarks (cargo xtask perf)
xtask perf runs release benchmark examples with --json, merges their outputs into target/perf/results.json and target/perf/summary.md, and checks them against perf/budgets.toml and perf/baseline.json.
How a Benchmark Example Reports Metrics (testkit::perf)
Every benchmark registered in perf/budgets.toml uses crates/testkit/src/perf.rs:
RssSampler::start(Duration::from_millis(2))polls resident memory on a background thread; callsampler.reset()before each scenario andsampler.peak()after.perf::row(name, &samples_ms, peak_rss, gpu_bytes)computes nearest-rankp50_ms,p95_ms, andmax_ms.perf::report(bench_name, context_json, rows, Some(&sampler))wraps the rows withprocess_peak_rss_bytes(VmHWMon Linux).perf::write_report(path, &report)writes the JSON file expected byxtask perf.
Adding or Updating a Performance Scenario
- Add the measurement row to the benchmark example (e.g.,
crates/ui-egui/examples/perf_scenarios.rsorinteractive_bench.rs). Keep the row'snamestable. - Add a
[[scenario]]entry inperf/budgets.tomllinkingbenchandrowto itsid(P1..Pn),metric(p50|p95|max),budget_ms, andenforce = true|false.enforce = false: reported asover budget (target not met yet)in the scorecard; only p50 regressions > 15% (and > 1.0 ms) against the same machine class fail the run.- Flip
enforce = truein the PR that meets the budget.
- Run
cargo xtask perf --quicklocally to verify the harness, then regeneratecargo xtask scorecard. - Only update
perf/baseline.jsonviacargo xtask perf --update-baselineon an idle machine (check the 1/5/15-min load average recorded in the baseline header—noisy background load invalidates latency numbers).
3. The Self-Auditing Scorecard (cargo xtask scorecard)
docs/scorecard.md is generated deterministically from committed files (xtask/src/scorecard.rs). Never edit docs/scorecard.md by hand; CI enforces cargo xtask scorecard --check.
What xtask scorecard Reads
perf/budgets.toml+perf/baseline.json: The performance scenarios table.crates/io/tests/corpus.rs: Parsesconst *: Source = Source { .. }blocks and their doc comments to extract corpus file counts,pass_floor, androundtrip_floor.scorecard/*.toml: Per-area TOML checklists (e.g.,file_compat.toml,tools.toml,ui.toml,automation.toml,reliability.toml,distribution.toml,bugs.toml).- Note:
targetandnotefields must not contain|(breaks Markdown tables), andids must be globally unique.
- Note:
- Dead Preferences Audit: Automatically scans
crates/engine/src/prefs.rsforPreferencesstruct fields that are never read outsideprefs.rsandprefs_ui.rs(target: 0 dead settings). - Source Tree Audits: Counts
#![deny(clippy::unwrap_used, ...)]coverage across all workspace crates, checks forpanic_hunt.rs, and reads live menu parity fromdocs/parity-checklist.md.
4. CI Pipeline Architecture (.github/workflows/ci.yml)
Understanding how CI runs tests helps diagnose failures and avoid slowing down builds:
- Dynamic Platform Selection (
platformsjob):ubuntu-latestruns on every PR.windows-latestsplits into two parallel legs (unit:--lib --bins;integration:--test '*') and runs onmain,full-ci-labeled PRs, or when platform-sensitive paths (apps/,crates/{ui-egui,gpu}/,Cargo.lock,build.rs) change.web(WebAssembly bundle + per-file size gate) runs when web or crate sources change.macoscan be reserved for release workflows or platform-specific changes.
- Test Variants (
test-variantsjob):- Optional feature matrix: Verifies crates compile and return clear errors when optional native/codec features are disabled.
- External asset/font fixtures: Checks out pinned test assets (e.g., CJK/Unicode fonts or domain symbol libraries) and sets
*_REQUIRED=1so tests that skip locally when optional assets are absent are strictly enforced in CI.
- Cache Discipline:
- Only
refs/heads/mainsavesrust-cacheandsccacheentries; PRs restoremain's cache in read-only mode so PR caches do not evictmainwithin GitHub's 10 GB cache limit. - Whenever running follow-up
nexteststeps (likepanic_hunt), pass the exact same--workspace --locked --features <features> --lib --bins --testsflags and filter with-Eso Cargo does not invalidate feature metadata and recompile the workspace.
- Only