Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/messaging-streaming

@228d57d
Run it yourself

This skill should be used when the user designs a "message queue", reaches for "Kafka", "RabbitMQ", "SQS", "Kinesis", "pub/sub", or "event-driven" architecture, asks about "async processing", "background jobs", "stream processing", or wrestles with "exactly-once vs at-least-once", "delivery guarantees", "message ordering", "duplicate handling / dedup", "dead letter queue", "backpressure", "saga orchestration", or "durable workflow". Use it whenever a slow or spiky operation should move off the request path, or two services must be decoupled, even if the user doesn't say "queue".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/messaging-streaming

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersaws.md

≈637 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Messaging & streaming — AWS

Service mapping → the generic options

  • SQS (Standard) — managed work queue; at-least-once, best-effort ordering, near-infinite throughput. The default queue. Built-in DLQ via redrive policy.
  • SQS (FIFO) — exactly-once processing (within a dedup window) and ordering per MessageGroupId; lower throughput than Standard. Use when order/dedup matter.
  • SNS — pub/sub fan-out; push to SQS, Lambda, HTTP. SNS→SQS fan-out is the canonical AWS decouple-and-fan-out pattern.
  • EventBridge — event bus with content-based routing/filtering and SaaS integrations; pub/sub with rules. Use for cross-service event routing.
  • Kinesis Data Streams — partitioned, replayable log (the Kafka analog); ordering per shard, consumers track position. Use for streams/replay.
  • MSK — managed Apache Kafka; use when you specifically need Kafka APIs/ecosystem.

Decision-changing limits (verify against current docs)

  • SQS message size ~256 KB (larger → S3 + claim-check pointer).
  • SQS visibility timeout governs redelivery; shorter than job time → duplicate processing. Max in-flight messages is bounded.
  • SQS FIFO throughput is far lower than Standard (raised by batching / high-throughput mode, still a ceiling); ordering is per MessageGroupId.
  • Kinesis ~1 MB/s or 1000 records/s write per shard, ~2 MB/s read; parallelism = shard count; resharding is an operation. A hot partition key saturates one shard.
  • SNS/EventBridge are at-least-once → subscribers must be idempotent.

Provider-specific trade-offs

  • SQS Standard can deliver duplicates and out of order — consumers must be idempotent (the AWS "delivered twice" gotcha). FIFO removes most of this at a throughput cost.
  • Managed DLQ is first-class (redrive), but you must set max receives and watch the DLQ — it's silent by default.
  • Kinesis vs. MSK: Kinesis is simpler/serverless-ish but AWS-specific (lock-in); MSK is portable Kafka but you manage more.
  • For durable workflows AWS offers Step Functions (state-machine orchestration, AWS-native) — compare with Temporal in temporal.md.

Pitfalls

  • Using SQS Standard where order matters, then bolting on ordering logic — use FIFO.
  • Forgetting SNS/SQS/Kinesis are at-least-once → no idempotency → double side effects.
  • Under-sharding Kinesis (throttling) or ignoring a hot shard from a skewed key.
  • No redrive/DLQ → poison messages redeliver until retention expires, then vanish.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    The skill provides architectural guidance and reference material for messaging and streaming systems. It contains no executable scripts, network operations, or security risks. All content is strictly educational and design-oriented.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/messaging-streaming