Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/distributed-logging

@228d57d
Run it yourself

This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collect → buffer → ship → index → store → retain pipeline, sampling, ordering, and cold-storage tiering. Use it whenever many services emit logs that must be searched in one place under load, even if the user doesn't say "logging pipeline".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/distributed-logging

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersgeneric.md

≈661 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Distributed logging — generic (self-host / open source)

The default answer when no cloud is named. Vendor-neutral mechanics and the open-source building blocks for each pipeline stage.

Stage → tool mapping

  • Collect: Fluent Bit or Vector (lightweight node agents), Fluentd (heavier, plugin-rich), Filebeat (Elastic's shipper). Read files/stdout, parse, enrich, route.
  • Transport / buffer: the agent's own disk buffer for simple cases; Kafka (or Redpanda/Pulsar) as a durable, partitioned log bus for high volume / fan-out. Treat delivery, ordering, and backpressure as owned by messaging-streaming.
  • Index / search: Elasticsearch / OpenSearch for full-text field search (the "E" in ELK/EFK); Loki for cheap label-indexed storage with grep-style queries.
  • Visualize / query: Kibana (ES/OpenSearch) or Grafana (Loki).
  • Cold archive: object storage (MinIO self-host, or any S3-compatible bucket) via blob-store; query cold data with an in-place engine if needed.

Common assemblies: ELK = Elasticsearch + Logstash + Kibana; EFK swaps Logstash for Fluentd/Fluent Bit; PLG = Promtail/Loki/Grafana.

When to pick which

  • Need rich, ad-hoc full-text search across fields → Elasticsearch/OpenSearch.
  • Volume is large and you mostly filter by label then scan → Loki (far cheaper).
  • One sink, moderate volume → agent disk buffer straight to the indexer (skip Kafka).
  • Spiky volume or multiple sinks (search + archive + analytics) → put Kafka in front.

Limits / things that bite (verify against current docs)

  • Elasticsearch shard pressure: too many shards exhausts heap; keep shards moderate and even, use rollover. High-cardinality index fields cause mapping explosions.
  • Heap/JVM: ES nodes are RAM-bound; under-provisioned heap → GC pauses → ingest rejection (HTTP 429). Watch the bulk/index queue.
  • Kafka: ordering is per-partition only; partition count caps consumer parallelism; retention is bytes/time per topic — size it for the longest sink outage.
  • Loki: weak at high-cardinality and arbitrary full-text; label cardinality is the cost driver.

Pitfalls

  • Synchronous ship from the app/agent that blocks the request path on a slow indexer.
  • Unbounded buffers (memory or disk) that OOM/fill during the spike they should absorb.
  • One ever-growing index with no time-based rollover → retention becomes a crisis.
  • Regex-parsing free text at the agent at high volume (CPU sink) instead of emitting structured logs at the source.
  • Running ES "because it's standard" when label-indexed Loki would cost a fraction.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    The skill provides architectural guidance for designing distributed logging systems. No security issues were detected.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/distributed-logging