Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/sharded-counters

@228d57d
Run it yourself

This skill should be used when the user needs a "sharded counter", "distributed counter", to "count likes / views at scale", handles a "high-write counter" or "hot counter contention", asks about "approximate counting", "real-time counts", or "HyperLogLog". It gives the recipe for absorbing write-heavy counting without a single hot row. Use it whenever one row/key takes concurrent increments faster than it can serialize them, even if the user doesn't say "sharded counter".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/sharded-counters

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈124 tokens for metadata: the name and description. ≈2.8k when used: this file. ≈3.8k more on demand in 5 files.

Description uses 6.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈496 characters, or 6.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Sharded counters

Count a thing that is incremented far faster than a single row, key, or partition can serialize writes — likes, views, votes, rate tallies, inventory decrements. The trap is the hot counter: every writer contends on one record, so latency climbs and throughput plateaus no matter how big the box is. Getting it wrong turns a trivial +1 into the bottleneck of the whole feature.

When to reach for this

Concurrent increments to a single logical count exceed what one row/key can absorb — a viral post's like count, a live-event view counter, a global rate tally. The symptom is write contention (lock waits, CAS retries, partition hot-spotting) on one record while the rest of the store is idle. Reaching for this means the write side is the problem, and an exact-to-the-millisecond total is not required.

When NOT to

Low write rate (a single atomic INCR handles thousands/sec — don't shard a counter nobody is hammering; YAGNI). Counts that must be transactionally exact and read-after-write consistent at every instant (bank balances, seat inventory at sell-out) — that's a transactional decrement, see consistency-coordination, not a fan-out tally. Counting distinct items exactly (unique visitors) where you also need the member list — that's a set in the store, not a counter. If reads dominate and writes are cheap, you need a cached aggregate, not sharding.

Clarify first

  • Write rate to the hottest single count — peak increments/sec on one logical counter, not the aggregate (→ back-of-the-envelope).
  • Exact or approximate — is an off-by-a-few total acceptable, and for how long may shards disagree (eventual)? Drives shard count and read path.
  • Counting occurrences or distinct items — a running total vs. unique-count (likes vs. unique viewers) decides plain shards vs. HyperLogLog.
  • Read rate and freshness — how often is the total read, and how stale may the served number be (sub-second? minutes?).
  • Time-windowed or lifetime — "views in the last hour" needs bucketed keys and expiry; a lifetime total does not.

The options

  • Single atomic counter — one row/key with atomic INCR/UPDATE +1. Use when peak write rate on the hottest count is well within one node's serialized write throughput. The default; don't outgrow it prematurely.
  • Write-sharded (striped) counter — split one logical count into N physical shards (counter:{id}:shard:{0..N-1}); each write increments a random/hashed shard, reads sum all N. Use when single-key contention is the bottleneck and the total may be eventually consistent.
  • Approximate distinct count (HyperLogLog) — a fixed-size probabilistic sketch (~12 KB) that counts unique items with ~2% error. Use for uniques at scale where exact membership isn't needed (unique visitors, distinct search terms).
  • Time-windowed (bucketed) counters — key the counter by time bucket (views:{id}:2026-06-01T14), increment the current bucket, sum recent buckets on read, expire old ones. Use for "last N minutes/hours" rate-style counts.
  • Aggregate-on-read + cached total — sum shards (or roll up) periodically and serve the cached number. Use when reads vastly outnumber writes and a slightly stale total is fine (pairs with caching).

Trade-offs

Option What it solves What it worsens Change it when
Single atomic counter Simplest; exact; read-after-write trivial One hot record caps write throughput; contention under spikes Increments on one count exceed one node → shard the writes
Write-sharded counter Spreads write load N-way; removes the hot spot Reads cost N lookups + sum; total is eventually consistent; pick N up front Read cost of summing N grows painful → cache the aggregate / roll up
HyperLogLog Counts uniques in fixed tiny memory at huge scale ~2% error; can't list members or do exact counts Exact uniques or the member set is required → use a stored set
Time-windowed buckets Cheap rolling/rate counts; old data self-expires More keys; window boundaries need care; cross-bucket reads sum many keys You need an exact lifetime total → keep a separate lifetime counter
Aggregate-on-read + cache Cheap reads of a heavy-write count Served total lags writes by the refresh interval Reads must be fresh-to-the-write → read shards live (eat the N-sum)

Behavior under stress

A counter is a tiny thing that punches above its weight in an outage.

  • Hot-shard skew: if writes pick shards by hash(userId) instead of random, one viral actor or a bad hash can still pile onto one shard. Mitigate: pick the shard at random per write; size N to peak contention, not average.
  • Read amplification on spikes: when a count goes viral, reads of the total multiply the N-shard sum across the read fan-out and can overload the store. Mitigate: cache the aggregate and refresh on an interval, not per read (→ caching).
  • Lost increments: fire-and-forget increments (or a crash before flush in a buffered/write-back path) silently undercount. Mitigate: use the store's atomic increment, accept the eventual-consistency window explicitly, and reconcile from a source of truth if exactness later matters.
  • Window-boundary stampede: time-bucketed counters all roll to a new key at the top of the hour — a synchronized cold bucket plus a flood of reads. Mitigate: pre-create buckets and jitter rollups.
  • Mass expiry: bucketed counters expiring together can spike the store. Mitigate: stagger TTLs.

Monitor: per-shard write distribution (skew), increments/sec vs. node ceiling, read-path latency for the N-sum, sketch error budget (HLL), and under/over-count drift against any source of truth.

How to apply

  1. Clarify the inputs — peak increments/sec on the hottest single count, exact-vs-approximate tolerance, occurrence-vs-distinct, read rate, and freshness budget (see Clarify first). If no number shows one count is too hot, stay on a single atomic counter (YAGNI).
  2. Pick from the trade-off table — single atomic if it fits one node; write-sharded if single-key contention is the wall; HyperLogLog for uniques; time-buckets for rolling windows. Combine (e.g. sharded + cached aggregate).
  3. Set the key knobs — choose N (shard count) from peak contention, the shard-selection rule (random, not user-hashed), the read aggregation method, bucket granularity + TTL for windows, and the cache refresh interval.
  4. Stress-test the choice — walk Behavior under stress: confirm shard skew, read amplification, lost increments, and window boundaries each have a mitigation the traffic profile actually needs.
  5. Size it with numbers — N ≥ peak increments/sec ÷ per-shard write ceiling; confirm the N-sum read cost and any HLL error fit the budget (→ back-of-the-envelope).
  6. Pick a provider — default to the generic recipe; open a provider file only if the user named a cloud (see Choosing a provider).
  7. Show the trade-off in the answer — give a Decision | Solves | Worsens | Change it when… table with one row per choice you made here (e.g. the option, its store, its failure policy); the last column is a concrete trigger (a number or observable signal). Then name what breaks first.

Dos and don'ts

Do

  • Start with a single atomic counter; shard only when a number shows one count is the bottleneck.
  • Pick the write shard at random so load spreads evenly regardless of the actor.
  • Cache the summed aggregate and refresh on an interval when reads dominate.
  • Use HyperLogLog for uniques-at-scale and state the ~2% error as a known cost.
  • Expire time-bucket keys with staggered TTLs and pre-create the next bucket.
  • State the eventual-consistency window out loud — sharding trades exact-now for throughput.

Don't

  • Don't shard a counter that a single INCR already handles (premature sharding adds read cost for nothing).
  • Don't hash the shard by user/entity ID — a hot actor reconcentrates the load.
  • Don't sum N shards on every read of a viral count — cache the aggregate.
  • Don't use a sharded/eventual counter where the number must be transactionally exact (money, sell-out inventory) → consistency-coordination.
  • Don't reach for HyperLogLog when you also need the member list or an exact count.

Numbers that matter

The deciding figure is peak increments/sec on the single hottest count vs. a node's serialized write ceiling — that ratio sets N. A HyperLogLog sketch is ~12 KB for billions of uniques at ~2% standard error, regardless of cardinality — the reason it beats a stored set at scale. Reading a sharded total costs N lookups, so N trades write headroom for read cost. For QPS rates, per-node write ceilings, and storage sizing, don't restate them here — see back-of-the-envelope.

Interface sketch

A sharded counter is a key contract, not one value:

  • Write: INCR counter:{id}:shard:{rand(0..N-1)} (atomic, fire to one shard).
  • Read: sum(GET counter:{id}:shard:{0..N-1}) — or read the cached aggregate count:{id} refreshed every T seconds.
  • Distinct: PFADD uniq:{id} {member} then PFCOUNT uniq:{id} (HLL sketch).
  • Windowed: INCR views:{id}:{bucket} with a TTL; read sums the recent buckets.

Decide N, the shard-selection rule, the aggregation/refresh policy, and the window granularity up front — they are the contract, not implementation details.

Choosing a provider

Default to the generic recipe above. If the user names a cloud, read references/providers/<provider>.md for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

Diagram

To visualize the fan-out write path (writer → random shard) and the aggregate-on-read sum (read → N shards → cached total), use the in-plugin architecture-diagram skill — shards share the store color, the read fan-out is a dashed sum arrow, and the cached aggregate sits in the cache color.

Related building blocks

  • data-storage — depends on this for where the shards physically live; sharding/partitioning theory and key design are owned there.
  • caching — pairs with this to serve the cached aggregate so a viral count's reads don't re-sum N shards every time.
  • consistency-coordination — depends on this for the exact-vs-eventual count decision; transactional/atomic semantics and quorum are owned there.
  • back-of-the-envelope — feeds into this: it supplies the per-count write rate and per-node ceiling that justify sharding and set N.
  • system-design — owned-concept lives in the orchestrator: the reasoning loop, the trade-off method, and the ten failure modes.

References

  • references/deep-dive.md — shard-count math, random vs. hashed selection, HyperLogLog mechanics and error, time-bucket layout, roll-up/aggregation patterns, and reconciliation. Read when designing the counter in detail.
  • references/providers/{generic,aws,azure,gcp}.md — service mappings, atomicity/contention limits, and pitfalls per environment.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    This skill consists purely of architectural documentation and design patterns for distributed counters. It contains no executable code, network operations, or data-handling scripts, and presents no security risks.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/sharded-counters