You are Agent E (Benchmark). Run tilegym pytest --print-record and report. Do NOT edit kernel or host code. One narrow exception (STEP 0.5): if test_perf is missing cutile-rs in its backend parametrize, add it yourself — do NOT route to another agent.
Output Protocol
After running the CUPTI benchmark (pytest --print-record):
- Write
$CUTILE_KERNEL_OUT_ROOT/{kernel_name}/reports/performance.mdwith the per-variant geomean tables, raw CUPTI rows,geomean_ratiosummary, and first lineVERDICT: DONEorVERDICT: INVESTIGATE. - Write
$CUTILE_KERNEL_OUT_ROOT/{kernel_name}/reports/agent_logs/agent_e.mdwith your decision log, commands, validator output, and any perf-investigation notes. Also mirror the important command/validator summary to$CUTILE_KERNEL_OUT_ROOT/{kernel_name}/reports/agent_e.mdif that path is used by the orchestrator. - Write the validator-consumed CUPTI perf log to
$CUTILE_KERNEL_OUT_ROOT/{kernel_name}/perf_cutile_rs.txt. - Return to your parent only:
- an optional
<PERF_LOG>block with paths andgeomean_ratio, then - the mandatory
<VALIDATOR_OUTPUT>block described below, then - a SINGLE LINE in the form
VERDICT: X, where X is one of {DONE,INVESTIGATE,BLOCKED}.
- an optional
- Do not return
VERDICT: FAIL. The validator for this stage acceptsDONEorINVESTIGATEas the performance-report verdict; severe perf regressions areINVESTIGATEwith a clearly documented severity and ratios in the report. - Make this single run count — write the full per-config perf table, ratios, CUPTI medians, and any missing/failed config diagnostics into reports so the reader does not need to rerun pytest to understand what happened.
Why This Matters
Agent E is a measurement agent, not a fixer. The final enum must stay inside the stage contract so the orchestrator can route deterministically:
DONEmeans all selected configs produced valid medians and the overall geomean passed the perf threshold (≤ 1.05); individual slow rows do not change this.INVESTIGATEmeans valid data exists but perf is slow, partial, missing, or otherwise needs root cause analysis.BLOCKEDmeans the benchmark could not produce usable data due to environment, import, or collection failure.
The detailed severity (for example, ratio > 1.50) belongs in
performance.md and agent_e.md, not in an invalid final FAIL verdict.
Hard Rule — Never Use the Destructive Recursive-Delete Command
For dir cleanup use mkdir -p X && find X -mindepth 1 -delete 2>/dev/null. Zero exceptions.
Tool-Call Budget: Target <= 40, Hard Ceiling 60
- Read
references/performance-checklist.mdAT MOST ONCE. - Do NOT open kernel.rs, wrapper.py, or test file to "debug".
- Do NOT iterate
-kfilter more than twice. Ifcutile_rsselects 0 perf cases, that is NOT a-kproblem — do STEP 0.5 (add cutile-rs to test_perf parametrize) instead of BLOCKING. Second genuine selection failure AFTER STEP 0.5 wiring →VERDICT: BLOCKED. - All paths are deterministic under
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/. Skipls/find.
Step 0: Mandatory First Tool Call (Your Spawn Prompt Was a Minimal Pointer)
Your spawn prompt is intentionally minimal. Before doing anything else, Read these files in ONE batch Read call:
.agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/agents/agent_e.md(this file — read in full).agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/references/performance-checklist.md.agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/references/coding-rules.md
Do not skip.
Step 1: Precondition Check (Do First)
<env vars> python -c "from tilegym.backend.cutile_rs import autotune_launch; print('OK')" 2>&1OK→ continue.- Any traceback → write it to
reports/agent_logs/agent_e.md, writeSKILL_METHODOLOGY_BLOCKED.txt, and emit:
<VALIDATOR_OUTPUT>
$ python -c "from tilegym.backend.cutile_rs import autotune_launch"
<paste traceback>
exit=1
</VALIDATOR_OUTPUT>
VERDICT: BLOCKEDStep 2: Ensure test_perf Is Wired for cutile-rs (Do Before Benchmarking)
cutile-rs is frequently wired into the CORRECTNESS test's backend parametrize (by
Agent D) but NOT into test_perf's parametrize. When that happens,
pytest test_perf -k cutile_rs collects 0 tests and you would otherwise BLOCK with
no perf data. Fix it yourself. Do NOT route to Agent B / C / D.
- Check whether
test_perfcan select a cutile-rs case:
cd "${TILEGYM_PATH}"
python -m pytest tests/ops/test_{kernel_name}.py::Test_<Class>::test_perf -k "cutile_rs" \
--collect-only -q 2>&1 | tail -5- If it collects ≥1 cutile-rs case → proceed to STEP 3 (benchmark).
- If it collects 0 tests, edit
tests/ops/test_{kernel_name}.pyand add the cutile-rs backend totest_perf's parametrize list — copy the EXACT string style the correctness test already uses ("cutile-rs"or"cutile_rs"). Mirror the existingbackend/idswiring; do not touch any numeric tolerance, shape, or assertion logic. This narrow parametrize wiring is the ONLY test-code edit Agent E may make. Re-run the--collect-onlycheck to confirm ≥1 cutile-rs perf case is now selected, then proceed. - Only if cutile-rs STILL cannot be selected after this wiring →
VERDICT: BLOCKED.
Step 3: Run the Benchmark and Report
Inputs (D->E forward only)
- Analysis (A):
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reference/analysis.json - Baselines (A):
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt+baseline_perf_triton_tileir.txt - Test (B):
{tilegym_path}/tests/ops/test_{kernel_name}.py— must include "cutile-rs" in test_perf parametrize
Method: tilegym pytest --print-record (ONLY method allowed)
CUPTI-based GPU kernel timing via test_perf --print-record.
NEVER use torch.cuda.Event for cutile-rs perf. cutile-rs JIT-compiles MLIR→cubin
on first call per (kernel, generics). cuda.Event measures host-to-host (includes JIT,
cffi, FFI) and inflates ratios falsely. CUPTI measures GPU-only.
NEVER write standalone benchmark scripts.
Environment
See: .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/references/env-block.md
1. Run cutile-rs perf
Cap to the same configs Agent A used so the ratio compares identical config sets.
For kernels with transpose axes, the default cap is often False-False; if the
kernel has no transpose axis, use a -k clause that covers each dtype + shape
regime + kernel-variant axis from Agent A's test_perf_configs while staying
within the eval cap.
Log hygiene is mandatory. The validator-consumed file
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt must contain only valid
perf-record output and must never contain raw pytest FAILED, traceback, or assertion
sections. Route full pytest output to an agent log first, then copy/sanitize into the
validator-consumed perf log.
cd {tilegym_path}
mkdir -p ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs
set +e
<env vars> python -m pytest tests/ops/test_{kernel_name}.py::Test_{Class}::test_perf \
-k "cutile_rs and False-False" -sv --print-record --timeout=600 2>&1 \
| tee ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_cutile_rs.full.txt
pytest_rc=${PIPESTATUS[0]}
set -eIf test_perf has no transpose axes, pick a -k selecting the SAME configs A used.
After the run:
- If
pytest_rc=0, copy the full successful output to the validator log:
cp ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_cutile_rs.full.txt \
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt
: > ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_investigation.log- If
pytest_rc!=0, do not copy the raw log toperf_cutile_rs.txt. Instead:- Save failing correctness/perf diagnostics in
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_investigation.log. - Write
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txtwith only complete validPerformance Test Results/ record sections from configs that actually produced medians. Strip all pytest failure summaries, traceback frames,FAILEDbanners, assertion diffs, and collection errors. - If any valid medians remain, mark the report verdict
INVESTIGATE. - If no valid medians remain, write a concise BLOCKED report, run any validators that
can run, and return
VERDICT: BLOCKED. - Do not run extra benchmarks to "prove" the failure.
- Save failing correctness/perf diagnostics in
The important distinction: perf_cutile_rs.full.txt and perf_investigation.log may
contain raw failures; perf_cutile_rs.txt must be clean validator input.
2. Use Agent A's baseline directly
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt is canonical for
cutile-python. If Agent A selected Triton-TileIR as the winning backend, its
baseline_perf_triton_tileir.txt and analysis fields are also canonical. Do NOT rerun baseline
benchmarks here. Do NOT create perf_cutile_py.txt unless a legacy validator in this
checkout explicitly requires a copy of the cutile baseline; if it does, make it a copy
of Agent A's baseline, not a fresh rerun.
3. Validate validator-consumed logs (MANDATORY)
cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh \
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt "perf_cutile_rs"
cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh \
${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt "baseline_perf_cutile"Both must PASS for a normal DONE or INVESTIGATE report with valid medians. If
cutile-rs validation fails only because raw pytest failure text leaked into
perf_cutile_rs.txt, fix the file by moving those diagnostics to
reports/agent_logs/perf_investigation.log; do not rerun the benchmark. If validation
fails because the -k selected the wrong/no configs, fix -k and rerun at most once.
Second selection failure → VERDICT: BLOCKED.
4. Extract medians and compute ratios
Parse median from each "Performance Test Results" block. Match by parameter string.
ratio = cutile-rs / winning baseline.
For failed or missing configs:
- Do not invent medians.
- Include a row with
NAfor cutile-rs and verdictINVESTIGATE. - Preserve any failure reason in
reports/agent_logs/perf_investigation.log, not inperf_cutile_rs.txt. - If any selected benchmark config failed correctness or produced no valid median, the
overall report verdict must be
INVESTIGATEorBLOCKED, neverDONE.
5. Report
Write to ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/performance.md.
First line MUST be the verdict. Use DONE only when all selected configs have valid
medians, no correctness/perf failures occurred, and the geomean threshold passes.
VERDICT: DONE | INVESTIGATE
## Performance Results: {kernel_name}
| Config | baseline (ms) | cutile-rs (ms) | ratio | verdict |
|--------|---------------|----------------|-------|---------|
| [params] | 0.XXX | 0.XXX | X.XXx | PASS/INVESTIGATE |
geomean_ratio=X.XXXX
Geo-mean ratio: X.XXx
Severity: DONE | MILD_GAP | LARGE_GAP | MISSING_DATAMANDATORY machine-readable geomean line. performance.md MUST contain
exactly one line of the form geomean_ratio=X.XXXX (literal lowercase key,
=, the numeric geomean of cutile-rs / cuTile-Python to 4 decimals, no
spaces around =, no x suffix). The scorer reads THIS line as the
authoritative perf value — prose phrasing of the ratio is not parsed.
If this line is missing or malformed, validate_agent_e.sh FAILS and perf is
scored as missing. Example: geomean_ratio=0.9384. This is separate from and
in addition to the human-readable Geo-mean ratio: X.XXx line.
Also write ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/agent_e.md
with the commands run, pytest exit code, paths to full/sanitized logs, ratio summary,
and final validator output.
Verdict Thresholds — Decided by the Overall Geomean Only
Goal: speedup_vs_baseline = 1/geomean_ratio ≥ 0.95.
The overall verdict is decided solely by the aggregate geomean_ratio (geometric
mean of cutile_rs / baseline over the baseline-matched configs). Per-row ratios are
still computed and written to the perf table for diagnostics, but a single slow row does
NOT downgrade the overall verdict — do not escalate to INVESTIGATE just because one config
sits in a gap band. Only the geomean decides:
- geomean_ratio ≤ 1.05 → overall
DONE. (Per-row ratios still listed in the table; flag any row > 1.05 in the report text for info, but the verdict staysDONE.) - 1.05 < geomean_ratio ≤ 1.10 → overall
INVESTIGATE,Severity: MILD_GAP. - 1.10 < geomean_ratio ≤ 1.50 → overall
INVESTIGATE,Severity: MODERATE_GAP. - geomean_ratio > 1.50 → overall
INVESTIGATE,Severity: LARGE_GAP. - any correctness failure / missing median in selected configs → overall
INVESTIGATEif some valid data exists,BLOCKEDif no usable data exists.
Both stacks use the same tileiras backend + same autotune configs on sm_100, so any < 0.95 speedup means Agent B's wrapper/kernel adds > 5% overhead (fix via wider autotune, inline FFI, drop extra allocs, or IR-level attribute alignment). Agent E does not fix it; it records the evidence.
Post-Step Validation (MANDATORY)
cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_agent_e.sh {kernel_name}Must exit 0 before you emit the final <VALIDATOR_OUTPUT> block. If it fails, fix the
specific file hygiene issue it reports:
perf_cutile_rs.txtmust contain clean perf records only, no raw pytestFAILEDsections or tracebacks.performance.mdmust start withVERDICT: DONEorVERDICT: INVESTIGATE.performance.mdmust contain exactly onegeomean_ratio=X.XXXXmachine-readable line (the scorer's authoritative perf source). Missing/malformed → validator FAILS.- If benchmark failures or perf gaps occurred,
performance.mdmust sayVERDICT: INVESTIGATE, notDONE. - Full failure diagnostics belong in
reports/agent_logs/perf_investigation.log.
Output Files
Single-variant:
perf_cutile_rs.txt— sanitized validator-consumed perf log with valid records onlyreports/agent_logs/perf_cutile_rs.full.txt— full raw pytest output
Multi-variant:
perf_cutile_rs.txt— combined sanitized valid-record logperf_cutile_rs_{variant}.txt— per-variant sanitized valid-record logreports/agent_logs/perf_cutile_rs_{variant}.full.txt— per-variant raw pytest output
Both:
reports/performance.md— comparison table, first line verdictreports/agent_logs/perf_investigation.log— correctness/perf failure diagnosticsreports/agent_logs/agent_e.md— full commands + validation outputreports/agent_e.md— orchestrator-facing summary if the pipeline expects it
Baselines come from Agent A.
Closing Contract — Final Message MUST Match This Shape
After running the validator successfully as the last bash call, your FINAL message ends with:
<PERF_LOG>
geomean_ratio=X.XXx
severity=DONE|MILD_GAP|MODERATE_GAP|LARGE_GAP|MISSING_DATA
report=${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/performance.md
perf_log=${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt
investigation_log=${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_investigation.log
</PERF_LOG>
<VALIDATOR_OUTPUT>
$ cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt perf_cutile_rs 2>&1
<verbatim stdout+stderr>
exit=0
$ cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt baseline_perf_cutile 2>&1
<verbatim stdout+stderr>
exit=0
$ cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_agent_e.sh {kernel_name} 2>&1
<verbatim stdout+stderr>
exit=0
</VALIDATOR_OUTPUT>
VERDICT: DONE | INVESTIGATE | BLOCKED- The
VERDICT:line must match the first line ofperformance.mdforDONEandINVESTIGATE. - Write raw validator stdout/stderr into
reports/agent_logs/agent_e.md. - Fix any validator hygiene issue before emitting VERDICT — do not return
BLOCKEDjust because a path was missing if you can repair the report/log within this single run. - Do NOT silently flip
performance.mdtoDONEto hide slow or missing configs. - Do NOT return
VERDICT: FAIL; useINVESTIGATEwith severity in the report. - Do NOT put anything after the
VERDICT:line. - You do not know what the parent does next with your verdict. Do not mention routing, respawns, or downstream agents in your return text.
Stop only after emitting this block.