/aws-resilience-lifecycle

Resilience Lifecycle: Define → Test → Operate

Overview

This SOP guides the integrated use of all three AWS resilience services (Resilience Hub v2, FIS, ARC) to achieve proven, operational resilience through a continuous Define → Test → Operate loop.

Parameters

service_name (required): Name of the service going through the lifecycle service_arn (optional): Existing service ARN if already onboarded to Resilience Hub lifecycle_phase (optional, default: "all"): Which phase to execute — define, test, operate, or all aws_region (optional, default: "us-east-1"): Primary AWS region

Steps

1. Verify Dependencies

Check for required tools and warn the user if any are missing.

Constraints:

• You MUST verify that either the AWS CLI or the AWS MCP server's call_aws tool is available (e.g., check the CLI is on PATH with command -v aws)
• You MUST NOT call AWS service APIs during verification — checking tool presence (such as command -v aws) is fine; do not make any AWS service calls
• You MUST inform the user about any missing tools with a clear message
• You MUST ask if the user wants to proceed anyway despite missing tools
• You SHOULD recommend the user authenticate with short-lived IAM role credentials (IAM role assumption, AWS SSO, an instance profile, or aws sts assume-role) rather than long-lived IAM user access keys, since this lifecycle performs privileged and potentially destructive operations.
• You MUST honor the lifecycle_phase parameter and execute ONLY the selected phase(s): "define" → Step 2 only; "test" → Step 3 only; "operate" → Step 4 only; "all" (default) → Steps 2–5 in order
• You MUST skip any phase not selected by lifecycle_phase

2. Phase 1 — DEFINE (Resilience Hub)

Establish resilience targets and identify gaps through assessment.

Constraints:

• You MUST guide the user through: create policy → create system → create user journey → create service → add input sources → run assessment
• You MUST delegate execution of each step's mutating AWS CLI commands to the resilience-hub-getting-started skill, which owns those commands and their validation — this lifecycle skill sequences and gates the phases; it does not execute the Define-phase mutations itself
• You MUST triage findings by action required:
  • Architecture gap (NOT_ACHIEVABLE) → Fix infrastructure first
  • Untested failure mode → Design FIS experiment (Phase 2)
  • No recovery mechanism → Set up ARC controls (Phase 3)
  • Missing observability → Add CloudWatch alarms
• You MUST NOT skip directly to testing if findings show NOT_ACHIEVABLE — architecture must be fixed first

3. Phase 2 — TEST (Fault Injection Service)

Validate resilience through controlled fault-injection experiments.

Constraints:

• You MUST map each "untested" finding from Phase 1 to an FIS experiment
• You MUST guide the user through: identify scenario → define stop conditions → scope blast radius → create template → execute → analyze results
• You MUST recommend scoping the FIS execution role to least privilege — only the specific actions and resource ARNs each experiment needs (e.g. ec2:TerminateInstances limited to tagged resources), never broad ec2:* or *, since FIS execution roles are high-privilege (they terminate instances, disrupt networking, etc.).
• You MUST delegate execution of each step's mutating AWS CLI commands to the fault-injection-experiment-design skill, which owns those commands and their validation — this lifecycle skill sequences and gates the phases; it does not create or start experiments itself
• You MUST enforce the pre-experiment checklist:
  • Stop condition alarms are in OK state
  • Rollback plan documented
  • Team notified (or scheduled GameDay)
  • Blast radius limited (start non-prod, then canary %)
• You MUST present the fully-rendered experiment template (actions, targets, stop conditions, role ARN) to the user and receive explicit confirmation BEFORE calling aws fis start-experiment — never auto-start a fault-injection experiment, since it is destructive (terminates instances, disrupts networking). This mirrors the confirm-before-proceeding gate in Step 1.
• You MUST gate the finding update on validation — mark a finding RESOLVED only after the experiment confirms recovery within objectives. The actual update-failure-mode-finding call is executed by the resilience-hub-getting-started skill (which owns resiliencehubv2 commands); this skill supplies the gate and the evidence comment, e.g. --status RESOLVED --comment "Validated via FIS experiment {experiment_id}. RTO: {actual_rto}s (target: {target_rto}s)"

4. Phase 3 — OPERATE (Application Recovery Controller)

Configure operational recovery mechanisms for production use.

Constraints:

• You MUST guide the user through: create routing controls → add safety rules → test failover → configure zonal autoshift
• You MUST delegate execution of each step's mutating AWS CLI commands to the recovery-controller-setup skill, which owns those commands and their validation — this lifecycle skill sequences and gates the phases; it does not create routing controls or configure autoshift itself
• You MUST NOT create routing controls without safety rules
• You MUST verify the application can handle N-1 AZ capacity before enabling zonal autoshift
• You MUST connect ARC back to NGRH findings:
  • "No cross-region failover mechanism" → Set up routing controls + safety rules
  • "Single-AZ dependency" → Enable zonal autoshift
  • "Recovery not validated" → Run FIS experiment, then enable ARC

5. Phase 4 — CONTINUOUS (Loop Back)

Establish ongoing resilience validation cadence.

Constraints:

• You MUST recommend the following cadence:
  • After architecture changes → Re-run NGRH assessment
  • After new deployments → Verify input sources capture new resources
  • Monthly → Run FIS experiments (single-service)
  • Quarterly → Cross-service GameDay
  • After incidents → Re-assess affected services, add new experiments
• You MUST inform the user that zonal autoshift practice runs provide continuous validation automatically
• You MUST recommend monitoring routing control state changes via CloudTrail
• You SHOULD suggest expanding experiments progressively: single-fault → multi-fault → cascading → surprise (unannounced to on-call)

Examples

Note: the CLI commands below illustrate what the delegated companion skills (resilience-hub-getting-started, fault-injection-experiment-design, recovery-controller-setup) execute on behalf of this lifecycle skill. This skill orchestrates the phases and gates progression — it does not run these commands directly.

# Phase 1: Define
aws resiliencehubv2 create-policy --name "tier-1" --availability-slo target=99.99 \
  --multi-az rtoInMinutes=5,rpoInMinutes=1,disasterRecoveryApproach=ACTIVE_ACTIVE \
  --multi-region rtoInMinutes=30,rpoInMinutes=5,disasterRecoveryApproach=HOT_STANDBY
aws resiliencehubv2 start-failure-mode-assessment --service-arn {service_arn}

# Phase 2: Test (from NGRH finding "AZ failure not validated")
aws fis create-experiment-template --description "Validate checkout-api survives us-east-1a AZ failure" \
  --actions '{"terminate-checkout":{"actionId":"aws:ec2:terminate-instances","targets":{"Instances":"checkout-instances"}}}' \
  --targets '{"checkout-instances":{"resourceType":"aws:ec2:instance","selectionMode":"ALL","resourceTags":{"resilience:service":["checkout-api"]},"filters":[{"path":"Placement.AvailabilityZone","values":["us-east-1a"]}]}}' \
  --stop-conditions '[{"source":"aws:cloudwatch:alarm","value":"arn:aws:cloudwatch:..."}]' \
  --log-configuration '{"cloudWatchLogsConfiguration":{"logGroupArn":"arn:aws:logs:us-east-1:123456789012:log-group:/fis/checkout-api:*"},"logSchemaVersion":2}' \
  --role-arn arn:aws:iam::123456789012:role/FISExperimentRole
# Enable experiment logging (above): the CloudWatch Logs group (or S3 bucket) receiving experiment
# logs SHOULD be KMS-encrypted (logs:AssociateKmsKey / SSE-KMS) — experiment logs can reveal
# infrastructure topology and failure modes.
# ⚠️ Present the fully-rendered template to the user and get explicit confirmation BEFORE the next line (see Step 3) — never auto-start a destructive experiment
aws fis start-experiment --experiment-template-id EXT123abc

# Phase 3: Operate
aws route53-recovery-control-config create-routing-control --cluster-arn {cluster_arn} --control-panel-arn {control_panel_arn} --routing-control-name checkout-us-east-1
aws arc-zonal-shift create-practice-run-configuration --resource-identifier {alb_arn} \
  --outcome-alarms '[{"alarmIdentifier":"arn:aws:cloudwatch:...","type":"CLOUDWATCH"}]'
aws arc-zonal-shift update-zonal-autoshift-configuration --resource-identifier {alb_arn} --zonal-autoshift-status ENABLED

# Phase 4: Update finding after validation
aws resiliencehubv2 update-failure-mode-finding --service-arn {service_arn} --finding-id {id} --status RESOLVED \
  --comment "Validated via FIS EXT123abc. RTO: 45s (target: 300s)"

Troubleshooting

User wants to skip Phase 1 and go straight to testing Advise against this — without NGRH assessment, they don't know which failure modes to test. Testing without targets means no way to measure success.

Findings show NOT_ACHIEVABLE but user wants to test anyway Explain that testing will confirm the failure but won't fix it. Architecture changes are needed first. Testing validates that a fix works, not that a broken architecture is broken.

User has existing FIS experiments but no NGRH Start with Phase 1 to get a complete picture. Existing experiments can be mapped to findings after assessment, and gaps in coverage will be identified.