/launch

Canary Rollout Reference

Purpose: Design progressive traffic-shifting deployments with guardrails. Defines per-stage traffic share, observation windows, automatic halt triggers, and promote/rollback decision rules.

When to Use Canary

• New version carries non-trivial risk (user-facing change, cross-service protocol change, DB interaction change)
• SLO-protected service with measurable error/latency budget
• Traffic splitter available (Envoy, Istio, K8s service weights, ALB, CloudFront, LaunchDarkly percentage targeting)

Skip canary and use blue-green when:

• Traffic-splitting infra unavailable
• Change is all-or-nothing (schema migration)
• Rollback window tighter than smallest canary stage

Standard Traffic Progression

Default: 1% → 10% → 50% → 100%

Aggressive (trusted change / small service)

0% → 25% → 100%, 15 min per stage. Use only when historical confidence + rapid rollback both confirmed.

Conservative (high-risk / regulated)

0.1% → 1% → 5% → 25% → 50% → 100%, 30 min+ per stage. Use for payments, billing, GDPR-adjacent flows.

Guardrail Metrics

Every canary must monitor, per stage:

Define per-service baselines from the last 7-day rolling p50.

Automatic Halt vs Human Approval

Halt and Rollback Procedure

Halt (freeze at current stage)

# Example: K8s service weighting
kubectl patch svc myservice -p '{"spec":{"selector":{"version":"stable"}}}'  # fall back to stable
# Or: Istio VirtualService weight: canary 0, stable 100

Halt is a decision point, not a final state. Within 15 min, oncall must choose:

• Rollback (revert to stable)
• Roll forward (fix + re-canary)
• Hold (extend observation with reduced risk)

Rollback (full traffic back to stable)

# Route 100% → stable
# Stop canary deployment
# Run post-rollback verification (smoke tests)
# Notify stakeholders

Rollback SLO: ≤ 3 minutes from decision to traffic fully routed.

The rollback target is the last known-safe version, not the previous one. Where the release being reverted also carried a security or safety fix, "stable" and "one release back" are different artifacts, and reverting to the latter reintroduces the fix's vulnerability under the cover of an incident response. Name the target release explicitly in the plan, and confirm which fixes it contains before routing traffic. Where the rollback crosses a schema or permission change, also verify: the old version can read records the new one wrote, permissions granted by the new version are revoked, and commitments already made to users are handled rather than silently dropped.

Decision Matrix

For each stage transition:

Traffic Splitter Configuration Snippets

Kubernetes + Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: myapp
        subset: stable
      weight: 90
    - destination:
        host: myapp
        subset: canary
      weight: 10

AWS ALB Weighted Target Groups

TargetGroup stable: weight 900
TargetGroup canary: weight 100
(ratio = 90:10)

Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 1
      - pause: { duration: 10m }
      - setWeight: 10
      - pause: { duration: 30m }
      - setWeight: 50
      - pause: { duration: 60m }
      - setWeight: 100
      analysis:
        templates:
        - templateName: error-rate-check
        args:
        - name: service-name
          value: myapp

LaunchDarkly Percentage Rollout

Flag: new-checkout
Rule: percentage rollout
  canary: 1% → 10% → 50% → 100%
  (increment via scheduled releases)

Per-Change-Type Recommendations

Playbook Template

# Canary Rollout: v2.15.0

**Service**: checkout-api
**Change**: new discount engine (PR #1234)
**Risk**: medium (logic change, conversion metric sensitive)

## Stages
| Stage | Traffic | Window | Auto-Promote Conditions |
|-------|---------|--------|-------------------------|
| 1 | 1% | 10 min | error rate < 0.2%, p95 < 220ms |
| 2 | 10% | 30 min | conversion ≥ 98% of 7d baseline |
| 3 | 50% | 60 min | all SLOs green |
| 4 | 100% | 2 h soak | no regressions reported |

## Guardrails
- Error rate (5xx): baseline 0.1%, halt at > 0.2%, rollback at > 0.5%
- p95: baseline 150ms, halt at > 225ms, rollback at > 300ms
- Conversion: baseline 4.8%, halt at < 4.6%, rollback at < 4.4%
- CPU: halt at > 85%

## Dashboards
- [Canary analysis dashboard](link)
- [Conversion funnel](link)

## Rollback
- Command: `argo rollouts abort checkout-rollout -n prod`
- SLO: ≤ 3 min
- Communication: #incidents + status page + PM notification

## Post-Success
- Close canary deployment
- Merge release commit to main-stable tag
- Update CHANGELOG + release notes (handoff Launch `notes`)
- Remove feature flag (if applicable, handoff Launch `flag`)

Anti-Patterns

• Canary without guardrails (just traffic split, no automated halt)
• Observation window shorter than issue detection time (too fast to catch)
• All-or-nothing traffic jumps (0 → 100) disguised as canary
• Ignoring business metrics, only watching infra metrics
• Promoting on "no alerts" (absence of signal ≠ presence of success)
• Running canary during peak traffic (no baseline to compare to)
• No rollback rehearsal (first time using the rollback command is during an incident)
• Parallel canaries on unrelated changes (can't attribute cause when things break)
• Mutating shared state (DB rows) from canary version (corrupts data visible to stable)

Handoffs

• Beacon → Launch: provides SLO baselines + dashboards
• Gear → Launch: implements traffic-splitting config
• Guardian → Launch: PR size + risk classification informs canary aggressiveness
• Launch → Triage: halt / rollback events escalate to incident commander
• Launch reporting: include the rollout summary and guardrail outcome in the weekly report