Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/sharded-counters

@228d57d
Run it yourself

This skill should be used when the user needs a "sharded counter", "distributed counter", to "count likes / views at scale", handles a "high-write counter" or "hot counter contention", asks about "approximate counting", "real-time counts", or "HyperLogLog". It gives the recipe for absorbing write-heavy counting without a single hot row. Use it whenever one row/key takes concurrent increments faster than it can serialize them, even if the user doesn't say "sharded counter".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/sharded-counters

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesdeep-dive.md

≈1.4k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Sharded counters deep-dive

Mechanics that don't belong in the lean SKILL.md. Read when designing the counter in detail.

Why one counter goes hot

A single logical count is one row/key. Every increment must serialize against every other — a row lock (SQL UPDATE … SET c = c + 1), a compare-and-swap retry loop, or a single-threaded shard handling one atomic op at a time. Under contention, writers queue: throughput plateaus at the serialization rate of that one record, and added concurrency only deepens the queue (latency climbs, CAS retries multiply). Scaling the box up doesn't help — the limit is one record's write path, not the machine.

Write-sharded (striped) counter

Split the logical count into N physical sub-counters and sum on read:

write:  shard = rand(0, N-1);  INCR counter:{id}:shard:{shard}
read:   total = sum(GET counter:{id}:shard:{i} for i in 0..N-1)
  • Pick the shard at random per write, not by hash(actorId). Hashing by the actor reconcentrates load when one actor (a celebrity, a bot, a retry storm) is responsible for the spike — the exact case sharding exists to fix. Random spreads evenly regardless of who writes.
  • Choosing N: N ≥ ceil(peak_increments_per_sec / per_shard_write_ceiling), then round up for headroom and skew. Bigger N spreads writes better but makes every read sum more keys. N is a knob between write headroom and read cost; pick it from the peak on the hottest single count, not the average across all counters.
  • Eventual consistency: a read may catch some shards mid-increment, so the total can lag the true value by the in-flight writes. This is the core trade: throughput for exact-now. State the window; if it's unacceptable, you don't want a sharded counter (→ consistency-coordination).

Aggregation on read and roll-up

Summing N shards per read is fine at low read rates. When reads dominate or the count goes viral:

  • Cached aggregate: a background job (or a periodic task) sums the shards and writes count:{id}; reads serve that. Refresh interval = freshness budget. Pairs with caching.
  • Two-tier roll-up: shards → periodic roll-up into a durable lifetime total (e.g. flush hourly buckets into a total column). Keeps the hot path in fast storage and the source-of-truth total in durable storage.
  • Read-time vs. write-time aggregation: aggregate on read when writes are the hot path (most counters); aggregate on write (maintain a materialized total) only when reads are hotter than writes and you can afford the write coupling.

HyperLogLog (approximate distinct count)

Counting unique items (unique visitors, distinct terms) exactly needs storing every member — memory grows with cardinality. HyperLogLog is a probabilistic sketch that estimates cardinality in fixed memory:

  • Hash each member; use the leading bits to pick one of m registers, and record the position of the leftmost 1-bit in the rest. Cardinality is estimated from the harmonic mean of register values (a long run of leading zeros implies many distinct items were seen).
  • ~12 KB holds the standard m = 16384 registers and estimates up to billions of uniques at ~0.81% / √m ≈ 2% standard error — independent of how large the true count is.
  • Mergeable: sketches union losslessly (take the max per register), so per-shard or per-window sketches combine into a global unique count. This is what makes HLL scale across nodes and time buckets.
  • Limits: you cannot list members, remove a member, or get an exact count. If any of those is required, use a stored set (and accept the memory cost) or a different structure.

Time-windowed (bucketed) counters

For "views in the last hour / 5 minutes":

write:  bucket = floor(now / granularity);  INCR views:{id}:{bucket}  (+ TTL)
read:   sum the buckets covering the window
  • Granularity trades resolution for key count: 1-minute buckets give a tight sliding window but more keys to sum; 1-hour buckets are cheap but coarse.
  • TTL each bucket to just beyond the longest window read, so old data self-expires — no cleanup job.
  • Stagger TTLs / pre-create buckets so an entire counter's buckets don't expire or cold-start at the same instant (window-boundary stampede). HLL sketches can be bucketed the same way for windowed unique counts.

Reconciliation and durability

Increments to an in-memory store (e.g. Redis) are fast but not durable by default; a crash can lose recent increments. Options:

  • Enable persistence (AOF) and/or replication for the counter store.
  • Treat the fast counter as a cache of a durable source of truth (events in a log/DB) and reconcile periodically — exact when it matters, fast on the hot path. The event log is owned by messaging-streaming / data-storage.
  • For money or inventory, do not use an eventual sharded counter — use a transactional decrement with the right isolation (→ consistency-coordination).

Common mistakes

  • Sharding a counter a single INCR already handled (read cost for no gain).
  • Hashing the shard by actor ID, so a hot actor lands on one shard anyway.
  • Summing N shards on every read of a viral count instead of caching the total.
  • One TTL for all buckets → synchronized mass expiry.
  • Using HLL where the member list or an exact count is later required.
  • Assuming the in-memory counter is durable without persistence/reconciliation.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    This skill consists purely of architectural documentation and design patterns for distributed counters. It contains no executable code, network operations, or data-handling scripts, and presents no security risks.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/sharded-counters