/clickhouse-js-node-rowbinary

Eval result (Composer) — RowBinary skill (skill-bench orders, with-skill vs no-skill)

Date: 2026-06-21 Model: Composer 2.5 Fast — composer-2.5-fast (executors). No separate grader — correctness scored against live ClickHouse server output (same fixture procedure as skill-bench). Harness: Cursor Task subagents (isolated work dirs under /tmp/rb-skill-bench/orders/) Method: The skill-bench orders contract — one fixed schema (UInt8, UUID, Decimal64(2), Enum8) — run as a 2×1 matrix (Composer with-skill vs no-skill). Fixture bytes and expected fields built from a live ClickHouse server (64 rows); throughput measured on a 64×400 = 25,600-row concatenated buffer (equivalence-checked, best of 5 rounds). 1 run per cell. This is server-truth scoring, not the 24 LLM-graded evals in evals/evals.json (see Sonnet / Opus for those).

Headline

Both cells decoded every field correctly against what ClickHouse itself produced — including the three classic traps this schema is chosen for (UUID byte order, Decimal64 fixed scale, Enum8 underlying int). The skill's lift on Composer is throughput and optimization patterns, not correctness on this run: the no-skill parser got UUID/decimal/enum right without peeking at the skill.

With-skill vs without-skill by trap (orders schema)

The orders skill-bench task exercises the same gotchas several evals target individually:

Composer vs Sonnet (skill-bench orders, same fixture shape)

Cross-run comparison uses the same orders contract and scoring rules; Sonnet numbers are from the prior claude-CLI skill-bench cell in .claude/skills/skill-bench/results/orders/.

Composer-no-skill is slower than Sonnet-no-skill on this one run (5.8M vs 8.0M rows/s) despite equal correctness — the Sonnet baseline used a tighter hand-rolled loop, while Composer-no-skill used readUInt32LE/readInt32LE for decimal and string-built UUID hex. Composer-with-skill more than closes that gap and beats Sonnet-no-skill by 1.76× on generated-code speed.

Where the skill clearly earned its keep (Composer)

Correctness gaps the skill closes on weaker models (see Sonnet eval-6 at 0.20) did not appear here — Composer-unaided passed server truth. What the skill did deliver:

• formatUUIDTable lookup-table path — adapted from the orders example (src/examples/orders.ts / skill UUID guidance) instead of per-byte string concatenation in the no-skill cell.
• Bigint + DataView.getBigInt64 decimal path — faithful signed Int64 unscaled units with scale-2 padding; no-skill used JS number arithmetic on 32-bit limbs.
• Flattened 26-byte fixed row — single stride (1 + 16 + 8 + 1), pre-sized new Array(rowCount), column comments — the "flatten the assembled row reader" tier from SKILL.md.

Isolation audit (no-skill): clean — hand-rolled formatUUID with per-half byte reversal, no formatUUIDTable / UUID_HEX16 / skill module names; verified the agent did not read paths under skills/clickhouse-js-node-rowbinary-parser-generator.

Findings specific to this Composer run

1. Composer-unaided correctness on orders is strong. One run, but both UUID and decimal formatting matched ClickHouse toString() output — unlike Sonnet-no-skill on eval-6 (0.20 pass rate across the 24-eval suite). Skill-bench still recommends multiple no-skill trials before claiming stability; this run is a point estimate only.
2. Skill value here is performance, not rescue. The 2.40× throughput gap is the headline; the skill cell is also ~2× the line count because it inlines the optimized UUID table and bigint formatters.
3. Agent cost not measured. Cursor Task subagents do not emit the stream-json transcript the claude-CLI skill-bench procedure uses for turns/tokens/USD — only generated-code metrics are reported.

Findings that align with the Opus / Sonnet eval runs

These skill-bench observations are consistent with themes from the 24-eval A/B runs, even though Composer did not execute that suite:

• Gotcha types are where no-skill breaks on weaker models — Composer passed this small schema; Sonnet's 24-eval no-skill pass rate was 60.4% with UUID/JSON/AggregateFunction as the big holes.
• Optimization tier is not automatic without the skill — no-skill Composer reached for readable Buffer.read*LE helpers; with-skill reached for the benchmarked hot path. Same pattern as eval-23 (monomorphized / inlined) and eval-20 (zero-copy / packed arrays).
• The orders schema is a weak correctness discriminator for strong models — both Composer cells correct; the schema discriminates code quality and speed instead (as skill-bench expects for Sonnet-no-skill UUID flakiness, not for every model).

Caveats

• Server-truth scoring on one schema, not the 24 LLM-graded evals — for assertion-level coverage across all type families, see eval_result_sonnet.md and eval_result.md.
• 1 run per (cell) — per-cell results are point estimates; Sonnet-no-skill UUID failure is non-deterministic across runs.
• Agent cost metrics unavailable in the Cursor Task harness (wall time, turns, tokens, USD all N/A).
• Cross-model throughput comparison is provisional — Sonnet-no-skill is from a different fixture build (fresh INSERT each run); row bytes differ, but schema and scoring rules match.

Raw parsers, fixture, and machine-readable scores: .claude/skills/skill-bench/results/orders/ (composer-{noskill,skill}.parser.mjs, fixture.json, results.json, report.md). Work dirs: /tmp/rb-skill-bench/orders/composer-{noskill,skill}/.