Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/task-scheduling

@228d57d
Run it yourself

This skill should be used when the user designs a "task scheduler", "job scheduler", "job queue", "cron at scale", "distributed cron", "delayed / scheduled / recurring tasks", a "worker pool", reaches for "Celery / Sidekiq / Airflow", or wrestles with "task leasing", visibility timeouts, job priorities, fairness, or duplicate task execution. Use it whenever work must run later, on a schedule, or be reliably leased to a pool of workers, even if the user doesn't say "scheduler". (For plain queue transport and delivery guarantees, that is `messaging-streaming`.)

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/task-scheduling

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersgeneric.md

≈638 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Task scheduling — generic / self-hosted

The vendor-neutral default. When no cloud is named, this is the answer. The queue transport underneath is messaging-streaming; this layer adds scheduling and leasing.

What to run

  • Celery (Python) / Sidekiq (Ruby) / BullMQ (Node) — task-queue frameworks over Redis or RabbitMQ. Worker pools, retries+backoff, priority queues, scheduled/delayed tasks. Maps the pull leasing + priority options. The default for app-level background jobs.
  • Quartz (JVM) — scheduler library; clustered mode (DB-backed) gives HA cron with misfire handling. Maps the distributed scheduler option.
  • Airflow / Dagster / Prefect — DAG orchestrators; dependencies, backfill, run history. Maps the workflow DAG option for batch pipelines.
  • Redis sorted set (ZADD score=run_at + poller) or RabbitMQ delayed exchange — maps the delay-queue / timer option.
  • Leader lock in etcd / ZooKeeper / Consul / a DB row — makes a single-instance scheduler (Celery beat, Sidekiq-cron) HA without double-firing.

Topology

  • One scheduler/beat process (leader-elected if HA) enqueues due jobs.
  • A queue (Redis/RabbitMQ) holds ready and delayed jobs.
  • A horizontally-scaled worker pool leases and drains; scale workers on backlog.
  • A DLQ (or failed-set) holds poison tasks after the retry cap.

Limits / things that bite

  • Beat/cron is usually single-instance — running two without a leader lock double-fires every recurring job. Make it HA yourself.
  • Visibility-timeout semantics differ by broker: Redis-backed task queues often re-deliver on worker loss only if configured (acks_late, visibility timeout); the default may ack-on-receive and lose a job on crash. Set it deliberately.
  • Redis as broker can lose messages (no fsync per message) — fine for recompute, risky for must-run jobs; RabbitMQ persists but you operate the nodes.
  • Priority queues in some brokers are coarse (a fixed set of levels), not arbitrary integer priority.

Pitfalls

  • Using acks_early/ack-on-receive for jobs that must survive a crash.
  • No leader lock on the scheduler → split-brain double-fire.
  • Reaching for Airflow for low-latency or high-rate tiny jobs (scheduler tick latency makes it the wrong tool).
  • Treating Redis-broker durability as guaranteed for money/must-run work.

Operationally heavier than a managed service, but zero lock-in and full control. Use when running your own infra or for portability across clouds.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    The skill provides architectural guidance for designing task scheduling systems. No security issues were detected.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/task-scheduling