Choose an ingestion strategy based on throughput, latency, and producer shape
Principle
Do not recommend a single ingestion pattern for every workload. The right approach depends on:
- events per second
- rows per insert
- acceptable buffering latency
- whether producers can batch
- whether decoupling is required
Decision framework
| Condition | Recommended path | Category |
|---|---|---|
| Producers can batch to 10K-100K rows and latency tolerance is moderate | Direct inserts | official |
| Producers send many small inserts and cannot batch effectively | Async inserts | official |
| Producers are bursty, many independent writers exist, or decoupling is needed | Kafka engine + materialized view | derived |
| Reliability, replay, and ingestion fan-out are primary concerns | Upstream queue or log broker before ClickHouse | field |
Guidance
Recommendation: direct batched inserts
Use when the application can naturally batch inserts into healthy sizes.
Why The existing best-practices guidance already favors appropriately sized insert batches.
Official sources
insert-batch-size- https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy
Recommendation: async inserts
Use when producers emit many small writes and the application cannot easily batch.
Why Async inserts let ClickHouse buffer small writes server-side to reduce part pressure.
Official sources
- https://clickhouse.com/docs/en/operations/settings/settings#async_insert
- https://clickhouse.com/docs/optimize/asynchronous-inserts
Recommendation: Kafka engine + materialized view
Use when a queue-based, decoupled ingest path is needed.
Why This is typically the right design when multiple producers, burst handling, or replayability matter.
Category derived
Sources
- https://clickhouse.com/docs/engines/table-engines/integrations/kafka
- https://clickhouse.com/docs/materialized-view/incremental-materialized-view
Validation
- Check average rows per insert
- Check part creation rate
- Check whether insert latency spikes correlate with small batch behavior