Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/task-scheduling

@228d57d
Run it yourself

This skill should be used when the user designs a "task scheduler", "job scheduler", "job queue", "cron at scale", "distributed cron", "delayed / scheduled / recurring tasks", a "worker pool", reaches for "Celery / Sidekiq / Airflow", or wrestles with "task leasing", visibility timeouts, job priorities, fairness, or duplicate task execution. Use it whenever work must run later, on a schedule, or be reliably leased to a pool of workers, even if the user doesn't say "scheduler". (For plain queue transport and delivery guarantees, that is `messaging-streaming`.)

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/task-scheduling

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈146 tokens for metadata: the name and description. ≈3.4k when used: this file. ≈4.6k more on demand in 6 files.

Description uses 7.3% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈583 characters, or 7.3%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Task Scheduling

Decide when work runs and which worker runs it: fire jobs on a schedule (cron/delayed/recurring), hand each job to exactly one worker via a lease, and make sure it completes once despite crashes and retries. This sits on top of messaging-streaming queues — the queue is the transport; this skill adds the scheduling, leasing, priorities, and task-level idempotency. Getting it wrong shows up as jobs that never run, run twice (double charge, double email), or pile up until a worker fleet falls permanently behind.

When to reach for this

Work must run later (send a reminder in 24h), on a schedule (nightly rollups, hourly cron), or repeatedly (poll every 5 min); a slow operation is already off the request path (→ messaging-streaming) and now needs reliable allocation to a pool of workers; jobs need priorities (paid before free) or fairness (no single tenant starves others); or a job must complete exactly once even though the worker holding it can crash mid-flight.

When NOT to

The caller needs the result inline — that's a synchronous call, not a scheduled job. A single fire-and-forget async step with no schedule, priority, or exactly-once need — a plain queue + idempotent consumer (messaging-streaming) is simpler; don't add a scheduler on top. One periodic job on one box — OS cron is fine until you have multiple schedulers or need history and retries. A long-running multi-step saga with rollback — reach for a durable workflow engine instead of hand-rolling state across jobs. Don't stand up Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a stateful control plane to operate and monitor.

Clarify first

  • Trigger type — scheduled (cron/at a time), delayed (run after N seconds), recurring (every N), or event-driven (a queue message arrives)? This decides whether a scheduler is even in scope.
  • Exactly-once vs at-least-once — is a duplicate run harmful (money, email) or harmless (idempotent recompute)? Drives the leasing + dedup design.
  • Latency budget vs throughput — must a delayed job fire within seconds of its time, or is "within a few minutes" fine? Tight timing is more expensive.
  • Priority / fairness — do some jobs jump the line, and must one tenant or job class be prevented from starving the rest? (→ back-of-the-envelope for arrival vs. service rate.)
  • Job duration & variance — seconds or hours? Sets the visibility-timeout / lease length and whether long jobs need heartbeats.
  • Idempotency key — what identifies a task as the same task on retry? (The key contract is owned by api-design.)

The options

Scheduling trigger

  • OS cron / single scheduler — one process fires jobs on a crontab. Use when one node, a handful of jobs, no HA requirement.
  • Distributed scheduler (HA cron) — a leader-elected scheduler enqueues due jobs into a queue; followers stand by. Use when the schedule must survive a node loss and must not double-fire.
  • Delay queue / timer — jobs carry a "not before" time; the queue holds them until due (delivery delay, sorted-set scoring, or a timer wheel). Use when per-job delays vary and you don't want a cron tick.
  • Workflow/orchestration DAG — declared task dependencies with backfill and history (Airflow-style). Use when batch pipelines have dependencies and you need a run history and reruns.

Worker allocation

  • Pull (worker leasing) — workers poll the queue, lease a job for a visibility timeout, and ack/delete on success. Use when you want back-pressure for free and elastic, self-balancing workers. The default.
  • Push (dispatcher assigns) — a coordinator routes jobs to specific workers. Use when affinity/locality matters (a job must run where its data is).

Priority & fairness

  • Priority queues — separate high/low queues drained in order. Use when some classes must run first.
  • Weighted / fair scheduling — round-robin or weighted draw across per-tenant queues. Use when one tenant's burst must not starve others.

Trade-offs

Option What it solves What it worsens Change it when
OS cron / single scheduler Trivial; zero infra SPOF — node dies, schedule stops; no retry/history You need HA or missed-run recovery → distributed scheduler
Distributed scheduler (HA cron) Survives node loss; no double-fire (leader-elected) Needs leader election (→ consistency-coordination); more moving parts One box and one job is enough → OS cron
Delay queue / timer Per-job delays without a cron tick; precise-ish timing Far-future jobs sit in the queue; timer accuracy bounded by poll interval Delays are uniform/periodic → cron; dependencies exist → DAG
Workflow DAG (Airflow-style) Dependencies, backfill, run history, reruns Heavy control plane; scheduler latency; overkill for single jobs Jobs are independent one-shots → plain queue + scheduler
Pull (worker leasing) Self-balancing, elastic, natural back-pressure At-least-once: lease expiry on a slow job re-runs it (need idempotency) A job must run on a specific node (data locality) → push
Push (dispatcher) Affinity/locality; central control Dispatcher is a bottleneck/SPOF; must track worker health No locality need → pull is simpler
Priority queues Important work runs first Low-priority starvation under sustained load Fairness across tenants matters → weighted/fair
Weighted / fair scheduling No tenant starves another More complex; per-tenant accounting Only one workload class exists → single queue

Behavior under stress

A scheduler can quietly fall behind, or it can amplify an outage by re-dispatching work a struggling fleet can't finish.

  • Backlog growth: arrival rate exceeds worker throughput; due jobs queue up and "scheduled for 09:00" runs at 09:40. End-to-end delay climbs while CPU looks fine. Mitigate: alarm on oldest-due-job age and queue depth, not just rate; scale workers; shed or defer low-priority jobs.
  • Lease expiry / re-run storm (the classic): a job runs longer than its visibility timeout, the lease expires, the queue redelivers it to a second worker, and now two workers run it — wasting capacity and, without idempotency, double-applying side effects. Mitigate: set the timeout above p99 job duration, heartbeat to extend the lease on long jobs, and make tasks idempotent/deduped.
  • Poison task: a job that always fails is retried forever, burning workers and re-loading a sick downstream. Mitigate: cap retries with backoff+jitter, then route to a dead-letter queue (retries/backoff/DLQ are owned by resilience-failure).
  • Thundering herd at the tick: thousands of cron jobs all scheduled at the top of the hour fire at once and stampede a downstream. Mitigate: jitter the schedule, spread triggers, or rate-limit dispatch.
  • Scheduler split-brain: two schedulers both think they're leader and double-enqueue every recurring job. Mitigate: a single leader via leader election + fencing (→ consistency-coordination); idempotent enqueue keyed by (job, scheduled_time).
  • Starvation: a flood of high-priority or one noisy tenant's jobs starves everyone else. Mitigate: fair/weighted scheduling and per-tenant concurrency caps.

Monitor: oldest-due-job age (the best lateness signal), queue depth per priority, lease-expiry / redelivery rate, retry and DLQ rate, worker utilization, and per-tenant share.

How to apply

  1. Clarify the inputs. Settle trigger type, exactly-once vs at-least-once, latency budget, priority/fairness, and job duration (see Clarify first). If the work is a single async step with no schedule or priority, stop — a plain queue + idempotent consumer (messaging-streaming) is enough.
  2. Pick from the trade-off table. Choose a trigger (cron → distributed scheduler → delay queue → DAG, cheapest that fits), an allocation model (pull leasing is the default; push only for locality), and a priority/fairness model only if more than one class exists.
  3. Set the key knobs. Visibility timeout above p99 job duration, retry cap + backoff before the DLQ, the dedup/idempotency key per task, the lease heartbeat interval for long jobs, and per-tenant concurrency limits.
  4. Stress-test the choice. Walk backlog growth, lease-expiry re-runs, poison tasks, tick stampede, scheduler split-brain, and starvation. Confirm a mitigation exists for each one the workload can trigger.
  5. Size it with numbers. Workers needed = arrival rate × avg job seconds / target concurrency (Little's law); confirm sustained throughput drains peak arrival, and that far-future delayed jobs fit storage (→ Numbers that matter).
  6. Pick a provider. Default to the generic recipe; only open a provider file if the user named a cloud (see Choosing a provider).
  7. Show the trade-off in the answer — give a Decision | Solves | Worsens | Change it when… table with one row per choice you made here (e.g. the option, its store, its failure policy); the last column is a concrete trigger (a number or observable signal). Then name what breaks first.

Dos and don'ts

Do

  • Default to pull-based worker leasing with a visibility timeout; it self-balances.
  • Set the visibility timeout above p99 job duration and heartbeat-extend long jobs.
  • Make every task idempotent (dedup on a task key) so a re-run is harmless.
  • Elect a single leader for the scheduler and key recurring enqueues by (job, time).
  • Cap retries with backoff+jitter, then dead-letter; alarm on oldest-due-job age.

Don't

  • Don't assume a leased job ran once — at-least-once means design for duplicates.
  • Don't schedule every cron job at :00 — jitter so the tick doesn't stampede.
  • Don't run two schedulers without leader election (split-brain double-fires).
  • Don't retry a poison task forever; cap it and dead-letter it.
  • Don't reach for Airflow/Celery before a schedule, priority, or HA need is real (YAGNI).

Numbers that matter

Size the worker pool with Little's law: in-flight jobs = arrival rate × average job duration, so workers ≈ peak arrival × avg seconds-per-job / per-worker concurrency. Sustained drain must exceed peak arrival or the backlog never clears. Set the visibility timeout above the p99 job duration (a too-short timeout is the #1 cause of duplicate runs); set far-future delay storage = delayed-job rate × max delay × job size. Don't restate the latency/QPS tables — pull the rates and durations from back-of-the-envelope.

Interface sketch

A scheduled task is a contract. Define it, not "a job":

  • Task envelope: stable task_id / idempotency key (dedup on retry — key contract owned by api-design), task_type/version, payload, priority, scheduled_for (run-not-before), attempt, and a trace_id.
  • Schedule spec: for recurring jobs, the cron/interval expression plus a dedup_key = (job_name, scheduled_time) so a double-enqueue is a no-op.
  • Lease contract: on dequeue a worker gets the task invisible for visibility_timeout; it must ack/delete on success or heartbeat to extend. Lease expiry → automatic redelivery. Failure after the retry cap → DLQ with attempt count and last error.

Choosing a provider

Default to the generic recipe above. If the user names a cloud, read references/providers/<provider>.md for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

Diagram

To visualize the scheduler → queue → leased workers path, or the lease-expiry redelivery loop, use the in-plugin architecture-diagram skill. Quick inline sketch: [scheduler] → [delay/priority queue] → workers (lease) ─expire→ requeue ─fail×N→ [DLQ]; main path solid, the expiry/DLQ branches dashed.

Related building blocks

  • messaging-streaming — depends on it: queues are the transport this skill schedules onto and leases from; it owns delivery guarantees, ordering, and DLQs and this skill does not reimplement them.
  • resilience-failure — pairs with it for the retry policy: backoff, jitter, and DLQ-as-containment are tuned there; this skill names them.
  • api-design — depends on its idempotency-key contract, the mechanism that makes a re-run after lease expiry safe.
  • consistency-coordination — depends on it for leader election (and fencing) so only one scheduler is active and recurring jobs don't double-fire.
  • back-of-the-envelope — feeds into sizing: arrival rate and job duration set the worker count and backlog drain.
  • system-design — feeds into the orchestrator's reasoning loop; it routes here when work must run later, on a schedule, or be reliably allocated to workers.

References

  • references/deep-dive.md — leasing/visibility-timeout mechanics and heartbeats, distributed-cron + leader election, delay-queue implementations (sorted set, timer wheel), priority/fairness algorithms, exactly-once-effect via dedup, and Celery/Sidekiq/Airflow internals. Read when designing the scheduler in detail.
  • references/providers/{generic,aws,azure,gcp,temporal}.md — service mappings, decision-changing limits, and pitfalls per environment.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    The skill provides architectural guidance for designing task scheduling systems. No security issues were detected.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/task-scheduling