Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/distributed-logging

@228d57d
Run it yourself

This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collect → buffer → ship → index → store → retain pipeline, sampling, ordering, and cold-storage tiering. Use it whenever many services emit logs that must be searched in one place under load, even if the user doesn't say "logging pipeline".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/distributed-logging

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersgcp.md

≈612 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Distributed logging — GCP

Service mapping (generic stage → GCP)

  • Collect: the Ops Agent on GCE; on GKE, logging is built in (a Fluent Bit DaemonSet ships container logs to Cloud Logging automatically). Apps can also write structured entries directly via the Cloud Logging API/SDK.
  • Buffer / transport: Pub/Sub as the durable log bus — Cloud Logging sinks route matching entries to Pub/Sub for fan-out / stream processing.
  • Index / search: Cloud Logging (Logs Explorer) is the managed hot search tier with its own query language; for analytics, route to BigQuery via a log sink.
  • Cold archive: Cloud Storage bucket via a log sink, with lifecycle rules to Nearline/Coldline/Archive classes (→ blob-store).

Common recipe: agents/resources → Cloud Logging → log sinks fan out to: a logging bucket (hot, searchable for N days), BigQuery (analytics), Cloud Storage (cold), and/or Pub/Sub (stream to anywhere).

Limits / things that bite (verify against current docs)

  • Log buckets have a configurable retention period; the default is a fixed window — extend it (and pay) or sink to GCS for long retention.
  • Ingestion is billed per GB after a free allotment; verbose unsampled logging gets expensive fast.
  • Exclusion filters drop matching entries before ingest billing — the primary cost lever (e.g. drop health-check and 200-OK access logs).
  • Pub/Sub ordering is only guaranteed with an ordering key within a region; without it, messages may arrive out of order.
  • BigQuery sink streaming has its own quotas and per-GB cost.

Provider trade-offs

  • Cloud Logging is deeply integrated (auto-collection on GKE, one query surface) but long retention and high volume push you to GCS/BigQuery sinks for cost.
  • BigQuery is excellent for SQL analytics over logs but is not a low-latency tail/search tool — use Logs Explorer for that.
  • Exclusion filters + sinks are the cost/queryability control plane; design them up front. Lock-in: sink config, the query language, and BigQuery schemas don't port.

Pitfalls

  • Not setting exclusion filters → paying to ingest health checks and noise.
  • Treating BigQuery as interactive log search (it's analytics, not tailing).
  • Assuming Pub/Sub preserves order without an ordering key.
  • Leaving default log-bucket retention and expecting cheap long-term storage there instead of GCS.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    The skill provides architectural guidance for designing distributed logging systems. No security issues were detected.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/distributed-logging