Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/sharded-counters

@228d57d
Run it yourself

This skill should be used when the user needs a "sharded counter", "distributed counter", to "count likes / views at scale", handles a "high-write counter" or "hot counter contention", asks about "approximate counting", "real-time counts", or "HyperLogLog". It gives the recipe for absorbing write-heavy counting without a single hot row. Use it whenever one row/key takes concurrent increments faster than it can serialize them, even if the user doesn't say "sharded counter".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/sharded-counters

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersgeneric.md

≈683 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Sharded counters — generic (vendor-neutral)

The default answer when no cloud is named. Self-hosted / open-source primitives.

Service mapping (generic recipe → tools)

  • Single atomic counter → Redis INCR/INCRBY; SQL UPDATE … SET c = c + 1 (one row); any store with an atomic increment.
  • Write-sharded counter → N Redis keys counter:{id}:shard:{i}, increment a random shard, sum on read (MGET the N shards). In SQL, N rows keyed by (id, shard) with SELECT SUM(c) … GROUP BY id.
  • Approximate distinct (uniques) → Redis HyperLogLog (PFADD / PFCOUNT / PFMERGE) — ~12 KB per sketch, ~2% error, mergeable across shards/windows.
  • Time-windowed → Redis keys per time bucket with EXPIRE, or a wide-column store (Cassandra) with a counter column per (id, bucket).
  • Eventual aggregate at very high write scale → Cassandra counter columns (the cluster handles distribution; counters are eventually consistent and non-idempotent on retry — see pitfalls).

When to pick which

  • Redis for the hot path: atomic ops, HLL built in, microsecond increments, trivial key sharding. The common choice for likes/views/rate tallies.
  • SQL single-row counter only at low write rates; shard to N rows when the row lock is the wall.
  • Cassandra counters when writes are distributed-scale and eventual consistency is acceptable; avoid where retries must not double-count.

Limits / things that bite (verify against current docs)

  • A single Redis instance is single-threaded for command execution — one hot key is capped by one core's op rate; sharding the key (not just the cluster) is what removes the hot spot.
  • Redis Cluster shards by hash slot: counter:{id}:shard:{i} keys spread across the cluster only if the slots differ — don't wrap the whole key in a hash tag.
  • Redis HLL: ~12 KB max per key, ~0.81%/√m (~2%) standard error; PFCOUNT on a union of large sketches is comparatively expensive — cache it.
  • Cassandra counter columns are not idempotent: a timed-out write that is retried may apply twice (over-count). They can't be part of a row with non-counter columns, and deletes-then-reuse is unsafe.
  • SQL single-row counters serialize on a row lock; contention shows as lock waits and deadlocks long before the box is CPU-bound.

Pitfalls

  • Treating a Redis counter as durable without AOF/replication — a crash drops recent increments.
  • Choosing N once for the average and getting crushed when one count goes viral — size N to peak on the hottest count.
  • Summing shards on every read instead of caching the aggregate.
  • Using Cassandra counters where exactness matters (retry double-count).

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    This skill consists purely of architectural documentation and design patterns for distributed counters. It contains no executable code, network operations, or data-handling scripts, and presents no security risks.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/sharded-counters