---
name: distributed-logging
description: This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collect → buffer → ship → index → store → retain pipeline, sampling, ordering, and cold-storage tiering. Use it whenever many services emit logs that must be searched in one place under load, even if the user doesn't say "logging pipeline".
title: distributed-logging
canonical_url: https://skilld.dev/gh/proyecto26/system-design-skills/distributed-logging
last_updated: 2026-10-29T10:01:24.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [references/deep-dive.md](https://skilld.dev/api/skills-raw/proyecto26/system-design-skills/distributed-logging/references/deep-dive.md), [references/providers/aws.md](https://skilld.dev/api/skills-raw/proyecto26/system-design-skills/distributed-logging/references/providers/aws.md), [references/providers/azure.md](https://skilld.dev/api/skills-raw/proyecto26/system-design-skills/distributed-logging/references/providers/azure.md), [references/providers/gcp.md](https://skilld.dev/api/skills-raw/proyecto26/system-design-skills/distributed-logging/references/providers/gcp.md), [references/providers/generic.md](https://skilld.dev/api/skills-raw/proyecto26/system-design-skills/distributed-logging/references/providers/generic.md).
>
> If the user asked to install this Skill, run `npx skilld install proyecto26/system-design-skills/distributed-logging`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/distributed-logging`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/proyecto26/system-design-skills/distributed-logging) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/distributed-logging`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/distributed-logging --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/distributed-logging --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Distributed logging

Move logs from thousands of processes into one searchable place, fast enough to
debug a live incident and cheap enough to keep for months. Getting it wrong is a
classic "ignore failure" miss: the logging pipeline is itself a distributed system
that buckles under the exact traffic spike you most need it during, and a naive
design either drops the evidence or takes down the app it instruments.

## When to reach for this
More than one process emits logs and someone needs to search them together; an
incident requires correlating a request across services; log volume has outgrown
`grep` on a box; or compliance demands retention. The pipeline buys central search,
cross-service correlation, and a durable record decoupled from any single host.

## When NOT to
A single service on one host where `journald` + log rotation is enough — a full
pipeline is pure operational overhead (YAGNI). Numeric time-series questions ("what
is p99 latency", "is error rate up") belong to metrics, not log scans — that is
`observability`'s job; logs answer "what exactly happened to *this* request". Don't
ship every debug line at full volume before a number shows the volume justifies the
cost; sample first.

## Clarify first
- **Volume and peak** — lines/sec and bytes/sec, average and peak (→ `back-of-the-envelope`). This sizes every stage.
- **Structured or free-text** — can producers emit JSON now, or is there legacy text to parse?
- **Query latency need** — interactive search in seconds (hot index) vs. occasional forensic/audit reads (cold archive)?
- **Retention + compliance** — how long hot, how long cold, any legal hold or PII redaction requirement?
- **Loss tolerance** — may logs be dropped/sampled under overload, or is every line evidence (audit/financial)?

## The options

**Collection (agent on the host)**
- **Sidecar/node agent (Fluentd, Fluent Bit, Vector, Filebeat):** tails files or
  reads stdout, adds metadata, ships out. Use when apps log to files/stdout and you
  want app code untouched — the default.
- **Direct-to-bus SDK:** the app writes structured events straight to a transport.
  Use when you control the code and want exact structure, accepting tighter coupling.

**Transport / buffer (the shock absorber)**
- **Agent-side buffer + direct ship to indexer:** simplest; agent disk-buffers and
  retries. Use at low-to-moderate volume with one consumer.
- **Durable log bus (`messaging-streaming`, e.g. Kafka):** producers write to a
  partitioned bus; indexers consume at their own pace. Use at high volume or when
  multiple sinks (search, archive, analytics) read the same stream.

**Index / store (the search backend)**
- **Full-text index (Elasticsearch/OpenSearch — the "E" in ELK/EFK):** rich queries,
  expensive RAM/disk. Use when interactive field search matters.
- **Label-indexed store (Loki):** indexes only labels, stores log bodies compressed;
  much cheaper, grep-style queries. Use for high volume where you mostly filter by
  service/label then scan.

**Retention / tiering**
- **Hot index → cold object store (`blob-store`):** keep days of searchable data
  hot, roll older data to compressed objects. Use whenever retention exceeds the
  hot window economically (almost always).

## Trade-offs

| Option | What it solves | What it worsens | Change it when |
|---|---|---|---|
| Node agent (Fluent Bit/Vector) | No app changes; central metadata + routing | One more daemon per host; parsing CPU; agent can lag | You need exact structure → emit structured events from the app |
| Direct-to-bus SDK | Clean structured events, no file parse | Couples app to transport; app blocks/loses logs if bus is down | Coupling/availability hurts → go back to agent + local buffer |
| Agent buffer → direct indexer | Fewest moving parts | Indexer backpressure hits producers; no replay; one sink | Volume spikes or you need >1 sink → add a log bus |
| Durable log bus (Kafka) | Absorbs spikes, decouples, replay, fan-out | Extra system to run; ordering only per-partition; cost | Volume is low and single-sink → drop the bus |
| Full-text index (ES/OpenSearch) | Fast rich field search | RAM/disk hungry; mapping explosions; costly at scale | Cost dominates and queries are label-filtered → Loki |
| Label-indexed (Loki) | Cheap storage at high volume | Weak full-text; slow on high-cardinality scans | You truly need arbitrary field search → full-text index |
| Hot index + cold archive | Cheap long retention | Cold reads are slow/manual to rehydrate | Forensic reads on old data must be fast → widen hot window |

## Behavior under stress
The pipeline's failure mode is that incidents *generate* log spikes — an outage
emits floods of errors and stack traces exactly when the pipeline is busiest, so it
must degrade without amplifying the outage.

- **Producer backpressure:** if the indexer slows and agents ship synchronously,
  log calls can block the app. *Mitigate:* bounded local buffer with **drop-newest /
  sample on overflow**, never block the request path. A durable bus moves the
  backlog off the hosts (backpressure + DLQ semantics are owned by `messaging-streaming`).
- **Volume amplification:** one bad deploy logging at debug can 100× volume and blow
  the index. *Mitigate:* per-service rate caps at the agent (rate limiting is owned by
  `resilience-failure`), dynamic sampling, alerts on ingest bytes/sec.
- **Index hot-shard / mapping explosion:** high-cardinality fields (raw user IDs as
  index fields) or one fat shard wrecks the cluster. *Mitigate:* time-based indices,
  bounded field mappings, label discipline.
- **Ordering:** the bus only orders *within a partition*; merged multi-host streams
  interleave. *Mitigate:* sort by event timestamp at query time; carry a monotonic
  sequence or trace span order, don't trust arrival order.
- **Losing the evidence:** dropping logs silently hides the very failure under
  investigation. *Mitigate:* meter and alert on drop/sample rate so loss is visible.

**Monitor:** ingest bytes/sec and lines/sec, end-to-end ship lag, bus consumer lag,
agent buffer fullness + drop rate, index queue/rejection rate, query latency.

## How to apply
1. **Clarify the inputs** — pin volume (avg + peak bytes/sec), structured-vs-text,
   query-latency need, retention, and loss tolerance (see *Clarify first*). If volume
   is tiny and single-host, stop — host-local logs suffice (YAGNI).
2. **Pick stages from the trade-off table** — choose a collector, decide whether a
   durable bus is warranted (volume/spike/multi-sink), pick the index backend by the
   query need, and choose the hot window + cold tier.
3. **Set the key knobs** — mandate a structured schema with a **correlation/trace ID**
   on every line, set agent buffer size + overflow policy, sampling rates, index
   rollover (time-based) and shard sizing, and the retention/tiering policy.
4. **Stress-test the choice** — walk each item in *Behavior under stress* (producer
   backpressure, a debug-flood deploy, hot shard, ordering, silent drops) and confirm
   the pipeline degrades by sampling/buffering, never by blocking the app.
5. **Size it with numbers** — multiply lines/sec × bytes/line × peak factor × retention
   to get hot index GB and cold archive GB/month; confirm the bus partition count and
   indexer fleet cover peak ingest (→ *Numbers that matter*).
6. **Pick a provider** — default to the generic recipe; open a provider file only if
   the user named a cloud (see *Choosing a provider*).
7. **Show the trade-off in the answer** — give a `Decision | Solves | Worsens |
   Change it when…` table with one row per choice you made here (e.g. the
   option, its store, its failure policy); the last column is a concrete trigger
   (a number or observable signal). Then name what breaks first.

## Dos and don'ts
**Do**
- Emit **structured** logs (JSON) with a correlation/trace ID stamped at the edge and propagated, so a request is one query.
- Put a **bounded buffer** between producers and the indexer and drop/sample on overflow rather than block the app.
- Use a durable log bus once volume spikes or more than one sink reads the stream.
- Tier aggressively: short hot window for search, compressed cold archive for retention.
- Meter ingest rate, ship lag, and drop rate, and alert on them — invisible loss is the trap.

**Don't**
- Don't let a logging call block or crash the request path; logging is best-effort by default.
- Don't index high-cardinality free-form fields — it explodes the cluster; keep labels bounded.
- Don't trust arrival order for causality; sort by event time / sequence (→ `sequencer`).
- Don't re-teach metrics/alerting here or set SLOs in logs — that's `observability`.
- Don't keep everything hot forever to avoid cold reads; size the hot window to the debugging window.

## Numbers that matter
Size the pipeline from volume: lines/sec × bytes/line gives ingest bytes/sec; a few
KB/line at tens of thousands of lines/sec is already 100s of MB/s and TBs/day. Hot
index storage ≈ daily bytes × hot-days × (1 + replica + overhead); cold archive ≈
daily bytes × retention-days, compressed ~5–10×. Apply a peak multiplier for incident
floods. Don't restate latency/QPS/storage rules of thumb here — pull them from
`back-of-the-envelope`.

## Interface sketch
A log line is a contract: a **structured event**, not a string. Minimum fields:

```
{ "ts": "2026-05-29T12:00:00.123Z", "level": "ERROR", "service": "checkout",
  "host": "pod-7", "trace_id": "abc123", "span_id": "f9", "msg": "charge failed",
  "user_id": "u42", "err": "timeout", "latency_ms": 812 }
```

`ts` is the event time (sort key at query); `trace_id`/`span_id` correlate across
services; `level` and `service` are bounded **index labels**; high-cardinality values
(`user_id`) stay as searchable body fields, not index keys. The bus partition key is
usually `service` or `trace_id` to keep a request's lines ordered together.

## Choosing a provider
Default to the generic recipe above. If the user names a cloud, read
`references/providers/<provider>.md` for the managed-service mapping, quotas/limits,
and provider-specific trade-offs. If no file exists for that provider, the generic
recipe is the answer.

## Diagram
To visualize the pipeline (producers → agents → bus → indexer/search + archive sink)
or the overflow/degradation path, use the in-plugin `architecture-diagram` skill —
draw the durable bus as the buffer between producers and sinks, the cold-archive sink
with a `blob-store` color, and the drop-on-overflow path as a dashed arrow.

## Related building blocks
- `observability` — *owned-concept lives in* the three-pillars view: metrics, traces, alerting, and SLO/SLIs are taught there; logs are one pillar, and *what* to alert on is its job, not this skill's.
- `messaging-streaming` — *depends on* it for the durable log bus: delivery guarantees, ordering, backpressure, and DLQ semantics are owned there; this skill just uses the bus as transport.
- `blob-store` — *feeds into* it for cold log archival: durability, tiering, and lifecycle of the compressed cold objects live there.
- `sequencer` — *pairs with* this when ordering across hosts matters; monotonic IDs and clock-skew handling are owned there.
- `system-design` — *owned-concept lives in* the orchestrator: the reasoning loop, the trade-off method, and the ten failure modes. *Feeds into* the wider design.

## References
- **`references/deep-dive.md`** — buffering and backpressure mechanics, sampling strategies, structured-logging + correlation-ID propagation, index lifecycle/rollover, ordering, and cold tiering. Read when designing the pipeline in detail.
- **`references/providers/{generic,aws,azure,gcp}.md`** — service mappings, limits, and pitfalls per environment.
