Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/distributed-logging

@228d57d
Run it yourself

This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collect → buffer → ship → index → store → retain pipeline, sampling, ordering, and cold-storage tiering. Use it whenever many services emit logs that must be searched in one place under load, even if the user doesn't say "logging pipeline".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/distributed-logging

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersazure.md

≈582 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Distributed logging — Azure

Service mapping (generic stage → Azure)

  • Collect: Azure Monitor Agent (AMA) on VMs/VMSS; for AKS, Container Insights or a Fluent Bit DaemonSet. Apps/services emit diagnostic logs to Azure Monitor.
  • Buffer / transport: Event Hubs as the durable log bus (Kafka-protocol compatible). Diagnostic settings can stream resource logs to Event Hubs for fan-out.
  • Index / search: Azure Monitor Logs / Log Analytics workspace, queried with KQL (Kusto Query Language) — the managed hot search/index tier.
  • Cold archive: Azure Blob Storage via diagnostic settings or Event Hubs capture, with lifecycle tiering to Cool/Archive (→ blob-store).

Common recipe: agents/resources → diagnostic settings → Log Analytics (hot, KQL) and/or Event Hubs → Blob Storage (cold) for cheap long retention.

Limits / things that bite (verify against current docs)

  • Log Analytics ingestion is billed per GB ingested and per GB retained; the default retention window is limited and longer retention costs more — set it per table.
  • Table plans: an Analytics tier for interactive KQL vs. a cheaper Basic/Auxiliary tier for high-volume logs you rarely query interactively — choosing wrong is a cost/queryability trap.
  • Event Hubs throughput is sized in throughput units / processing units; partition count is fixed at creation and caps consumer parallelism; ordering is per-partition.
  • Ingestion has per-workspace rate caps that throttle during spikes.

Provider trade-offs

  • Log Analytics + KQL is powerful and tightly integrated with Azure Monitor metrics and alerts (the alerting/SLO side is observability), but ingest cost scales with volume — sample and tier.
  • Basic/Auxiliary tables are far cheaper for verbose logs but restrict query features and retention; route high-value logs to Analytics, bulk logs to the cheap tier or straight to Blob.
  • Lock-in: KQL queries, diagnostic-setting routing, and workspace structure don't port.

Pitfalls

  • Sending all verbose logs to the Analytics tier (expensive) instead of the Basic tier or Blob archive.
  • Leaving default retention and ingesting unsampled debug floods → bill shock.
  • Assuming Event Hubs preserves global order — it's per-partition only.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    The skill provides architectural guidance for designing distributed logging systems. No security issues were detected.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/distributed-logging