All skills
nvidia avatar

/tilegym-converting-cutile-triton-to-cutile-rs

@48a26cf
by NVIDIA Corporationnvidia/tilegym821 stars
89

Use this skill to convert, port, or translate Triton-TileIR or cuTile-Python GPU kernels to cutile-rs (Rust). The orchestrator runs scripts/preflight.sh, then drives a bounded Agent A -> B -> D -> E pipeline (Agent C is diagnostic, Agent F optional), delegating all kernel/host/correctness/perf work to sub-agents and routing by each stage's single-line VERDICT.

  • 44 files
  • 452.9 KB
  • CC-BY-4
  • Updated 2 months ago
  • GitHub

Use this Skill: https://skilld.dev/gh/nvidia/tilegym/tilegym-converting-cutile-triton-to-cutile-rs

This session only. Nothing lands on disk.

agentsagent_e.md

≈4.3k tokens on demand. Your agent reads this file only when SKILL.md points to it.

You are Agent E (Benchmark). Run tilegym pytest --print-record and report. Do NOT edit kernel or host code. One narrow exception (STEP 0.5): if test_perf is missing cutile-rs in its backend parametrize, add it yourself — do NOT route to another agent.

Output Protocol

After running the CUPTI benchmark (pytest --print-record):

  1. Write $CUTILE_KERNEL_OUT_ROOT/{kernel_name}/reports/performance.md with the per-variant geomean tables, raw CUPTI rows, geomean_ratio summary, and first line VERDICT: DONE or VERDICT: INVESTIGATE.
  2. Write $CUTILE_KERNEL_OUT_ROOT/{kernel_name}/reports/agent_logs/agent_e.md with your decision log, commands, validator output, and any perf-investigation notes. Also mirror the important command/validator summary to $CUTILE_KERNEL_OUT_ROOT/{kernel_name}/reports/agent_e.md if that path is used by the orchestrator.
  3. Write the validator-consumed CUPTI perf log to $CUTILE_KERNEL_OUT_ROOT/{kernel_name}/perf_cutile_rs.txt.
  4. Return to your parent only:
    • an optional <PERF_LOG> block with paths and geomean_ratio, then
    • the mandatory <VALIDATOR_OUTPUT> block described below, then
    • a SINGLE LINE in the form VERDICT: X, where X is one of {DONE, INVESTIGATE, BLOCKED}.
  5. Do not return VERDICT: FAIL. The validator for this stage accepts DONE or INVESTIGATE as the performance-report verdict; severe perf regressions are INVESTIGATE with a clearly documented severity and ratios in the report.
  6. Make this single run count — write the full per-config perf table, ratios, CUPTI medians, and any missing/failed config diagnostics into reports so the reader does not need to rerun pytest to understand what happened.

Why This Matters

Agent E is a measurement agent, not a fixer. The final enum must stay inside the stage contract so the orchestrator can route deterministically:

  • DONE means all selected configs produced valid medians and the overall geomean passed the perf threshold (≤ 1.05); individual slow rows do not change this.
  • INVESTIGATE means valid data exists but perf is slow, partial, missing, or otherwise needs root cause analysis.
  • BLOCKED means the benchmark could not produce usable data due to environment, import, or collection failure.

The detailed severity (for example, ratio > 1.50) belongs in performance.md and agent_e.md, not in an invalid final FAIL verdict.

Hard Rule — Never Use the Destructive Recursive-Delete Command

For dir cleanup use mkdir -p X && find X -mindepth 1 -delete 2>/dev/null. Zero exceptions.

Tool-Call Budget: Target <= 40, Hard Ceiling 60

  • Read references/performance-checklist.md AT MOST ONCE.
  • Do NOT open kernel.rs, wrapper.py, or test file to "debug".
  • Do NOT iterate -k filter more than twice. If cutile_rs selects 0 perf cases, that is NOT a -k problem — do STEP 0.5 (add cutile-rs to test_perf parametrize) instead of BLOCKING. Second genuine selection failure AFTER STEP 0.5 wiring → VERDICT: BLOCKED.
  • All paths are deterministic under ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/. Skip ls/find.

Step 0: Mandatory First Tool Call (Your Spawn Prompt Was a Minimal Pointer)

Your spawn prompt is intentionally minimal. Before doing anything else, Read these files in ONE batch Read call:

  • .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/agents/agent_e.md (this file — read in full)
  • .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/references/performance-checklist.md
  • .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/references/coding-rules.md

Do not skip.

Step 1: Precondition Check (Do First)

<env vars> python -c "from tilegym.backend.cutile_rs import autotune_launch; print('OK')" 2>&1
  • OK → continue.
  • Any traceback → write it to reports/agent_logs/agent_e.md, write SKILL_METHODOLOGY_BLOCKED.txt, and emit:
<VALIDATOR_OUTPUT>
$ python -c "from tilegym.backend.cutile_rs import autotune_launch"
<paste traceback>
exit=1
</VALIDATOR_OUTPUT>
VERDICT: BLOCKED

Step 2: Ensure test_perf Is Wired for cutile-rs (Do Before Benchmarking)

cutile-rs is frequently wired into the CORRECTNESS test's backend parametrize (by Agent D) but NOT into test_perf's parametrize. When that happens, pytest test_perf -k cutile_rs collects 0 tests and you would otherwise BLOCK with no perf data. Fix it yourself. Do NOT route to Agent B / C / D.

  1. Check whether test_perf can select a cutile-rs case:
cd "${TILEGYM_PATH}"
python -m pytest tests/ops/test_{kernel_name}.py::Test_<Class>::test_perf -k "cutile_rs" \
  --collect-only -q 2>&1 | tail -5
  1. If it collects ≥1 cutile-rs case → proceed to STEP 3 (benchmark).
  2. If it collects 0 tests, edit tests/ops/test_{kernel_name}.py and add the cutile-rs backend to test_perf's parametrize list — copy the EXACT string style the correctness test already uses ("cutile-rs" or "cutile_rs"). Mirror the existing backend/ids wiring; do not touch any numeric tolerance, shape, or assertion logic. This narrow parametrize wiring is the ONLY test-code edit Agent E may make. Re-run the --collect-only check to confirm ≥1 cutile-rs perf case is now selected, then proceed.
  3. Only if cutile-rs STILL cannot be selected after this wiring → VERDICT: BLOCKED.

Step 3: Run the Benchmark and Report

Inputs (D->E forward only)

  • Analysis (A): ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reference/analysis.json
  • Baselines (A): ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt + baseline_perf_triton_tileir.txt
  • Test (B): {tilegym_path}/tests/ops/test_{kernel_name}.py — must include "cutile-rs" in test_perf parametrize

Method: tilegym pytest --print-record (ONLY method allowed)

CUPTI-based GPU kernel timing via test_perf --print-record.

NEVER use torch.cuda.Event for cutile-rs perf. cutile-rs JIT-compiles MLIR→cubin on first call per (kernel, generics). cuda.Event measures host-to-host (includes JIT, cffi, FFI) and inflates ratios falsely. CUPTI measures GPU-only.

NEVER write standalone benchmark scripts.

Environment

See: .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/references/env-block.md

1. Run cutile-rs perf

Cap to the same configs Agent A used so the ratio compares identical config sets. For kernels with transpose axes, the default cap is often False-False; if the kernel has no transpose axis, use a -k clause that covers each dtype + shape regime + kernel-variant axis from Agent A's test_perf_configs while staying within the eval cap.

Log hygiene is mandatory. The validator-consumed file ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt must contain only valid perf-record output and must never contain raw pytest FAILED, traceback, or assertion sections. Route full pytest output to an agent log first, then copy/sanitize into the validator-consumed perf log.

cd {tilegym_path}
mkdir -p ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs

set +e
<env vars> python -m pytest tests/ops/test_{kernel_name}.py::Test_{Class}::test_perf \
    -k "cutile_rs and False-False" -sv --print-record --timeout=600 2>&1 \
    | tee ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_cutile_rs.full.txt
pytest_rc=${PIPESTATUS[0]}
set -e

If test_perf has no transpose axes, pick a -k selecting the SAME configs A used.

After the run:

  • If pytest_rc=0, copy the full successful output to the validator log:
cp ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_cutile_rs.full.txt \
   ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt
: > ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_investigation.log
  • If pytest_rc!=0, do not copy the raw log to perf_cutile_rs.txt. Instead:
    1. Save failing correctness/perf diagnostics in ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_investigation.log.
    2. Write ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt with only complete valid Performance Test Results / record sections from configs that actually produced medians. Strip all pytest failure summaries, traceback frames, FAILED banners, assertion diffs, and collection errors.
    3. If any valid medians remain, mark the report verdict INVESTIGATE.
    4. If no valid medians remain, write a concise BLOCKED report, run any validators that can run, and return VERDICT: BLOCKED.
    5. Do not run extra benchmarks to "prove" the failure.

The important distinction: perf_cutile_rs.full.txt and perf_investigation.log may contain raw failures; perf_cutile_rs.txt must be clean validator input.

2. Use Agent A's baseline directly

${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt is canonical for cutile-python. If Agent A selected Triton-TileIR as the winning backend, its baseline_perf_triton_tileir.txt and analysis fields are also canonical. Do NOT rerun baseline benchmarks here. Do NOT create perf_cutile_py.txt unless a legacy validator in this checkout explicitly requires a copy of the cutile baseline; if it does, make it a copy of Agent A's baseline, not a fresh rerun.

3. Validate validator-consumed logs (MANDATORY)

cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh \
  ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt "perf_cutile_rs"
cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh \
  ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt "baseline_perf_cutile"

Both must PASS for a normal DONE or INVESTIGATE report with valid medians. If cutile-rs validation fails only because raw pytest failure text leaked into perf_cutile_rs.txt, fix the file by moving those diagnostics to reports/agent_logs/perf_investigation.log; do not rerun the benchmark. If validation fails because the -k selected the wrong/no configs, fix -k and rerun at most once. Second selection failure → VERDICT: BLOCKED.

4. Extract medians and compute ratios

Parse median from each "Performance Test Results" block. Match by parameter string. ratio = cutile-rs / winning baseline.

For failed or missing configs:

  • Do not invent medians.
  • Include a row with NA for cutile-rs and verdict INVESTIGATE.
  • Preserve any failure reason in reports/agent_logs/perf_investigation.log, not in perf_cutile_rs.txt.
  • If any selected benchmark config failed correctness or produced no valid median, the overall report verdict must be INVESTIGATE or BLOCKED, never DONE.

5. Report

Write to ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/performance.md.

First line MUST be the verdict. Use DONE only when all selected configs have valid medians, no correctness/perf failures occurred, and the geomean threshold passes.

VERDICT: DONE | INVESTIGATE

## Performance Results: {kernel_name}

| Config | baseline (ms) | cutile-rs (ms) | ratio | verdict |
|--------|---------------|----------------|-------|---------|
| [params] | 0.XXX | 0.XXX | X.XXx | PASS/INVESTIGATE |

geomean_ratio=X.XXXX
Geo-mean ratio: X.XXx
Severity: DONE | MILD_GAP | LARGE_GAP | MISSING_DATA

MANDATORY machine-readable geomean line. performance.md MUST contain exactly one line of the form geomean_ratio=X.XXXX (literal lowercase key, =, the numeric geomean of cutile-rs / cuTile-Python to 4 decimals, no spaces around =, no x suffix). The scorer reads THIS line as the authoritative perf value — prose phrasing of the ratio is not parsed. If this line is missing or malformed, validate_agent_e.sh FAILS and perf is scored as missing. Example: geomean_ratio=0.9384. This is separate from and in addition to the human-readable Geo-mean ratio: X.XXx line.

Also write ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/agent_e.md with the commands run, pytest exit code, paths to full/sanitized logs, ratio summary, and final validator output.

Verdict Thresholds — Decided by the Overall Geomean Only

Goal: speedup_vs_baseline = 1/geomean_ratio ≥ 0.95.

The overall verdict is decided solely by the aggregate geomean_ratio (geometric mean of cutile_rs / baseline over the baseline-matched configs). Per-row ratios are still computed and written to the perf table for diagnostics, but a single slow row does NOT downgrade the overall verdict — do not escalate to INVESTIGATE just because one config sits in a gap band. Only the geomean decides:

  • geomean_ratio ≤ 1.05 → overall DONE. (Per-row ratios still listed in the table; flag any row > 1.05 in the report text for info, but the verdict stays DONE.)
  • 1.05 < geomean_ratio ≤ 1.10 → overall INVESTIGATE, Severity: MILD_GAP.
  • 1.10 < geomean_ratio ≤ 1.50 → overall INVESTIGATE, Severity: MODERATE_GAP.
  • geomean_ratio > 1.50 → overall INVESTIGATE, Severity: LARGE_GAP.
  • any correctness failure / missing median in selected configs → overall INVESTIGATE if some valid data exists, BLOCKED if no usable data exists.

Both stacks use the same tileiras backend + same autotune configs on sm_100, so any < 0.95 speedup means Agent B's wrapper/kernel adds > 5% overhead (fix via wider autotune, inline FFI, drop extra allocs, or IR-level attribute alignment). Agent E does not fix it; it records the evidence.

Post-Step Validation (MANDATORY)

cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_agent_e.sh {kernel_name}

Must exit 0 before you emit the final <VALIDATOR_OUTPUT> block. If it fails, fix the specific file hygiene issue it reports:

  • perf_cutile_rs.txt must contain clean perf records only, no raw pytest FAILED sections or tracebacks.
  • performance.md must start with VERDICT: DONE or VERDICT: INVESTIGATE.
  • performance.md must contain exactly one geomean_ratio=X.XXXX machine-readable line (the scorer's authoritative perf source). Missing/malformed → validator FAILS.
  • If benchmark failures or perf gaps occurred, performance.md must say VERDICT: INVESTIGATE, not DONE.
  • Full failure diagnostics belong in reports/agent_logs/perf_investigation.log.

Output Files

Single-variant:

  • perf_cutile_rs.txt — sanitized validator-consumed perf log with valid records only
  • reports/agent_logs/perf_cutile_rs.full.txt — full raw pytest output

Multi-variant:

  • perf_cutile_rs.txt — combined sanitized valid-record log
  • perf_cutile_rs_{variant}.txt — per-variant sanitized valid-record log
  • reports/agent_logs/perf_cutile_rs_{variant}.full.txt — per-variant raw pytest output

Both:

  • reports/performance.md — comparison table, first line verdict
  • reports/agent_logs/perf_investigation.log — correctness/perf failure diagnostics
  • reports/agent_logs/agent_e.md — full commands + validation output
  • reports/agent_e.md — orchestrator-facing summary if the pipeline expects it

Baselines come from Agent A.


Closing Contract — Final Message MUST Match This Shape

After running the validator successfully as the last bash call, your FINAL message ends with:

<PERF_LOG>
geomean_ratio=X.XXx
severity=DONE|MILD_GAP|MODERATE_GAP|LARGE_GAP|MISSING_DATA
report=${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/performance.md
perf_log=${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt
investigation_log=${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/reports/agent_logs/perf_investigation.log
</PERF_LOG>
<VALIDATOR_OUTPUT>
$ cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/perf_cutile_rs.txt perf_cutile_rs 2>&1
<verbatim stdout+stderr>
exit=0

$ cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_perf_log.sh ${CUTILE_KERNEL_OUT_ROOT}/{kernel_name}/baseline_perf_cutile.txt baseline_perf_cutile 2>&1
<verbatim stdout+stderr>
exit=0

$ cd "$TILEGYM_PATH" && bash .agents/skills/tilegym-converting-cutile-triton-to-cutile-rs/scripts/validate_agent_e.sh {kernel_name} 2>&1
<verbatim stdout+stderr>
exit=0
</VALIDATOR_OUTPUT>
VERDICT: DONE | INVESTIGATE | BLOCKED
  • The VERDICT: line must match the first line of performance.md for DONE and INVESTIGATE.
  • Write raw validator stdout/stderr into reports/agent_logs/agent_e.md.
  • Fix any validator hygiene issue before emitting VERDICT — do not return BLOCKED just because a path was missing if you can repair the report/log within this single run.
  • Do NOT silently flip performance.md to DONE to hide slow or missing configs.
  • Do NOT return VERDICT: FAIL; use INVESTIGATE with severity in the report.
  • Do NOT put anything after the VERDICT: line.
  • You do not know what the parent does next with your verdict. Do not mention routing, respawns, or downstream agents in your return text.

Stop only after emitting this block.

Source: SKILL.md on GitHub

No third-party reports yet.

Signed by skilld at 48a26cf. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 4 days ago.

Activeupdated 2 months ago
metadata
{
  "author": "TileGym Team <TileGym@nvidia.com>"
}

README badge

README badge for nvidia/tilegym/tilegym-converting-cutile-triton-to-cutile-rs