/basin

Basin Pipelines

Streaming ingest: receive events over HTTP/Workers/Logpush, transform with SQL, write to R2 as Iceberg tables or Parquet/JSON files.

Documentation

This reference is a fast-start with verified code and gotchas. For limits, settings, full SQL syntax, and pricing, retrieve the live docs — use the Cloudflare MCP docs tool if available, otherwise webfetch the URL. Docs are source of truth over this file.

Three Components

Sources → Stream → Pipeline (SQL) → Sink → R2
          ↑          ↓                 ↓
   HTTP / Workers / Transform     Iceberg (Basin Catalog)
   Logpush          (row-level)   or Parquet/JSON files

Status: The linked docs PR proposes Basin general availability. Verify current availability, limits, and pricing in the live docs before making claims or sizing workloads.

Quick Start

# Interactive — creates stream + sink + pipeline, optionally bucket + catalog
npx wrangler basin pipelines setup

Minimal Worker producer:

interface Env { MY_STREAM: Pipeline; }

export default {
  async fetch(req: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    ctx.waitUntil(env.MY_STREAM.send([{ event_id: crypto.randomUUID(), amount: 29.99 }]));
    return new Response("OK");
  }
} satisfies ExportedHandler<Env>;

Which Sink Type?

Need SQL queries / ACID / time-travel on the data?
  → Basin Catalog (Iceberg)   ✅ Basin SQL, schema evolution   ❌ more setup

Just archival / external tools (Spark, Athena)?
  → R2 raw files (Parquet/JSON) ✅ simple, partitioned files  ❌ no built-in SQL

Critical Behaviors (read before building)

These are non-obvious and prevent most failures — see gotchas.md for detail.

• Everything is immutable after creation — stream schema, pipeline SQL, sink config. To change, delete and recreate.
• Sinks create their own table — they cannot target an existing Iceberg table.
• __ingest_ts is added automatically (TIMESTAMP, partitioned by day). Don't define it in your schema.
• Data isn't queryable immediately — first flush takes 3–7 minutes (warm-up + table creation) even with a short roll interval.
• Schema validation is deferred — invalid events are accepted then silently dropped. Monitor via GraphQL error metrics.
• Binding field renamed pipeline → stream (June 2026); old field still accepted.

Reading Order

1. configuration.md — schema, streams, sinks, pipelines (CLI + REST + Terraform), bindings
2. api.md — send(), HTTP ingest, REST API, pipeline SQL, lifecycle states
3. patterns.md — fire-and-forget, validation, Logpush, observability, end-to-end
4. gotchas.md — silent drops, immutability, REST≠CLI field names

See Also

• Basin Catalog — Iceberg sink destination
• Basin SQL — query the ingested data
• r2 · queues · workers