/clickhouse-js-node-rowbinary

RowBinary reader (decode) for Node.js

Decoding a RowBinary / RowBinaryWithNames / RowBinaryWithNamesAndTypes response from ClickHouse into JS values. Read SKILL.md first for the format gate ("is RowBinary even the right format?") and the principles that apply to both directions; this file covers the decisions and the per-type reference specific to reading. Writing? See writer.md.

First: complete buffer, or incremental stream?

Decide this before writing the reader — it changes the shape of the code and is a real performance fork.

• Incremental / streaming (the default here). You consume the HTTP response chunk by chunk as it arrives — low latency to the first row, bounded memory. It is generally the best choice for large results, but slower per-row.

• Whole buffer in memory (faster, when it fits). If you already hold the entire response as one Buffer, the bounds check never fires — so you can drop advance() entirely and read at a running offset in one monolithic loop. This is 2-3x faster but introduces latency and unbounded memory use.

The exposed API is streaming by default and requires an optimisation pass.

Second: row objects, or columnar (typed arrays)?

The default output is one object per row (array-of-structs). For a numeric, fixed-width result that the consumer reads column-wise, decode instead into one typed array per column (struct-of-arrays) — it is ~4x faster and several times smaller because it removes the per-row object, Date, and number-boxing allocations that dominate a numeric decode (the byte reads are already at memory bandwidth). Measured in tests/iot.columnar.bench.ts; rationale in case-studies/wasm-vs-js.md.

• Use columnar when columns are numeric/fixed-width and the consumer aggregates / filters / scans / plots them, or hands the buffers to a Worker or WASM kernel (typed-array ArrayBuffers are transferable — zero-copy).
• The preallocation trick: if EVERY column is fixed-width the row stride is known, so the exact count is buf.length / stride — allocate each column once, write at [i], no growth, no per-row bounds check.
• Streaming columnar is just that arithmetic per chunk. Fixed width means honoring a partial buffer needs no advance()/NeedMoreData/restart: the complete-row count is (chunk.length / stride) | 0, and the leftover bytes carry to the next chunk. Yield one typed-array batch per chunk, each owning a fresh transferable ArrayBuffer (see streamSensorColumns in src/readers/columnar.ts).
• Stay row-oriented when downstream code is row-shaped, the row is string-dominated (columnar's win is numeric — a JS string allocates either way), or the schema is nested/heterogeneous (Array/Map/Tuple).
• Hybrid: store columnar, expose a lazy rowAt(i) accessor that builds an object only for rows actually touched (see iotRowAt in src/examples/iot.ts).

Third: are the column types known ahead of time?

• Known (the default). Generate a straight-line reader specialized to those types — everything below.
• Only at runtime (the schema varies, or you just want to decode an arbitrary RowBinaryWithNamesAndTypes stream). Call compileRowBinaryWithNamesAndTypes(cursor) (src/readers/rowBinaryWithNamesAndTypes.ts): it reads the header, folds each column type's AST into a Reader (astToReader, src/readers/compile.ts; type strings parsed by @clickhouse/datatype-parser), and returns a readRows driver for the rest of the stream. Generic and unoptimized (no codegen), so prefer the specialized path whenever the types are fixed.

Reader guidance

On top of the shared principles in SKILL.md, the read path has its own:

• Streaming: throw + restart, not generators. To signal "need more bytes", a synchronous reader that throws a sentinel (NeedMoreData) and restarts the row beats generators for realistic chunk sizes.

• Keep an eye on chunk sizes. Partial trailing rows, small chunks are a silent throughput killer: streamRowBatches warns once when rows-per-chunk falls too low, and coalesceChunks(source, { minSize, timeoutMs }) merges small chunks in front of it when the source size isn't yours to raise.

• Hoist the cursor into locals. Prefer the working buffer and view declared once at the top of the generated reader, and keep the read offset in a local variable, operating on it directly instead of re-reading from an object.

• Coalesce advance() across adjacent fixed-width columns. A run of neighbouring fixed-width columns has a known combined size, so bounds-check it ONCE.

• Pre-allocate small result arrays. RowBinary gives every array/map its element count up front (the LEB128 prefix), so DEFAULT is to new Array(n). NOTE: for large arrays the application will iterate or compute over repeatedly, prefer [] + push (faster to traverse in V8) — or a typed array (Float64Array…) for numeric elements.

Reader type family references

The readers live as real code under src/readers/, split by type family.