/weave

Retry State Machine Reference

Purpose: Design retry state machines with exponential backoff, jitter, max-attempt cap, DLQ as terminal state, and idempotency contract. Separates retriable vs non-retriable failure modes and emits a contract that weave (schedule) and Beacon (alerts) can consume.

Scope Boundary

• weave retry: Retry state machine design (states, transitions, backoff, jitter, DLQ) — this document.
• weave timeout (elsewhere): TTL / deadline design. Retry and timeout are often paired; design timeout first, then retry inside the timeout envelope.
• weave compensation (elsewhere): Saga compensation. Retry handles transient failure; compensation handles logical undo.
• weave (elsewhere): Schedule / cron / backoff curve details. Retry state machine consumes weave's backoff rules.
• Beacon (elsewhere): SLO / retry-exhaustion alerts.
• siege concurrency (elsewhere): Race/deadlock analysis on retry implementations.

Workflow

CLASSIFY   →  split failures: retriable (transient) vs non-retriable (logical)
           →  5xx/network/timeout → retriable
           →  4xx/validation/auth → non-retriable (usually)

STATES     →  idle → attempting → waiting → attempting ... → success / DLQ
           →  terminal: success | permanent_failure | dlq

BACKOFF    →  base delay × 2^(attempt-1), capped at max_delay
           →  add jitter: full / equal / decorrelated

CAP        →  max_attempts = f(business SLA, error budget, DLQ capacity)
           →  typical: 3 for user-facing, 5-7 for background, 10+ for async

IDEMPOTENCY → every attempt uses the same idempotency key
           →  server must dedupe; retry state machine never generates a new key mid-retry

DLQ        →  terminal state for exhausted retries
           →  DLQ emits alert + preserves original message for human review

ALERT      →  hand off retry-exhaustion signal to Beacon
           →  hand off DLQ depth alert to Beacon

State Diagram

              ┌──────────┐
              │   idle   │
              └────┬─────┘
                   │ start
                   ▼
              ┌──────────┐       success       ┌──────────┐
              │attempting├─────────────────────►│ success  │ (terminal)
              └────┬─────┘                     └──────────┘
                   │ failure
       non-retri   │    retriable + attempts<max
       ┌───────────┤
       │           ▼
       │      ┌──────────┐
       │      │ waiting  │
       │      │ (backoff)│
       │      └────┬─────┘
       │           │ timer fires
       │           ▼
       │      ┌──────────┐
       │      │attempting│ (attempt++)
       │      └────┬─────┘
       │           │ attempts == max
       │           ▼
       │      ┌──────────┐
       │      │   dlq    │ (terminal, alert)
       │      └──────────┘
       ▼
  ┌──────────────┐
  │permanent_fail│ (terminal, no retry)
  └──────────────┘

Backoff + Jitter

Rule: always add jitter. No jitter → synchronized retries → thundering herd after a downstream recovery.

Typical Parameters

Retriable vs Non-retriable Classification

Idempotency Contract

Every retry must use the same idempotency key:

type RetryState = {
  idempotencyKey: string;      // generated ONCE at `idle → attempting`
  attempt: number;             // 1..max_attempts
  lastError: string | null;
  nextAttemptAt: Date | null;
  terminal: 'success' | 'permanent_failure' | 'dlq' | null;
};

Server-side contract:

• Dedupe by idempotency key within a TTL window (often 24h).
• Return the original response for duplicate key (same status, same body).
• If business rule conflicts (e.g., email already sent), return 409 with the original request-id.

DLQ Design

The DLQ terminal state must:

1. Preserve full payload (not just error).
2. Record failure cause (last error + attempt history).
3. Emit alert (Beacon signal — DLQ depth > threshold).
4. Allow manual replay with idempotency preserved.
5. Have TTL (typically 7-30 days before archival / purge).

DLQ Alert Thresholds (Beacon)

Output Template

## Retry State Machine: [Name]

### Context
- **Use case**: [HTTP client / queue consumer / webhook / batch]
- **SLA**: [end-to-end deadline]
- **Blast radius on failure**: [customer impact if DLQ]

### Failure Classification
- **Retriable**: [list of error codes / exceptions]
- **Non-retriable**: [list]
- **Ambiguous**: [list + default handling + rationale]

### State Machine
| State | Entry Actions | Exit Transitions |
|-------|--------------|------------------|
| idle | none | start → attempting |
| attempting | increment attempt counter | success → success; non-retriable → permanent_failure; retriable + attempts<max → waiting; retriable + attempts==max → dlq |
| waiting | compute next_delay with jitter | timer → attempting |
| success | emit success event | terminal |
| permanent_failure | emit permanent-failure event | terminal |
| dlq | enqueue to DLQ + alert | terminal |

### Backoff Parameters
- **Strategy**: [Exponential + Full Jitter / ...]
- **base**: [X ms]
- **max_delay**: [X s]
- **max_attempts**: [N]
- **total_time_budget**: [approx. Σ delays, worst case]

### Idempotency
- **Key generation**: [UUID v4 / ULID / hash of payload]
- **Server-side dedup TTL**: [hours]
- **Conflict handling**: [409 response pattern]

### DLQ
- **Storage**: [SQS FIFO / Kafka DLQ topic / DB table]
- **Retention**: [days]
- **Replay mechanism**: [manual / scheduled]

### Alert Integration (Beacon)
- **Retry-exhaustion rate > X%** over 5 min → HIGH
- **DLQ depth > N** → HIGH
- **DLQ growing > M/min** → CRITICAL

### Handoffs
- weave (backoff curve details)
- Beacon (alerts)
- Builder (implementation)
- Triage (DLQ review runbook)

Common Pitfalls

Deliverable Contract

When retry completes, emit:

• Failure classification table (retriable / non-retriable / ambiguous).
• State machine (states, transitions, terminal conditions).
• Backoff + jitter parameters (strategy, base, max_delay, max_attempts, total budget).
• Idempotency contract (key generation, server dedup TTL, conflict handling).
• DLQ design (storage, retention, replay).
• Alert thresholds for Beacon.
• Handoffs: weave, Beacon, Builder, Triage.

References

• Amazon Architecture Blog — "Exponential Backoff and Jitter"
• AWS SDK default retry policy documentation
• Google SRE Book — Handling Overload
• Stripe — Idempotency key design
• RFC 6585 — 429 Too Many Requests
• Polly (.NET), resilience4j (Java), tenacity (Python) — reference implementations