All skills
hieutrtr avatar

/incident-response

@4afa41d

Production incident response procedures for Python/React applications. Use when responding to production outages, investigating error spikes, diagnosing performance degradation, or conducting post-mortems. Covers severity classification (SEV1-SEV4), incident commander role, communication templates, diagnostic commands for FastAPI/ PostgreSQL/Redis, rollback procedures, and blameless post-mortem process. Does NOT cover monitoring setup (use monitoring-setup) or deployment procedures (use deployment-pipeline).

Use this Skill: https://skilld.dev/gh/hieutrtr/ai1-skills/incident-response

This session only. Nothing lands on disk.

referencesescalation-contacts.md

≈985 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Escalation Contacts

Contact Matrix by Severity

Severity Primary Contact Secondary Contact Management Communication
SEV1 On-call engineer (PagerDuty) Backend lead + Frontend lead Engineering Manager + VP Eng Status page + #incidents
SEV2 On-call engineer (PagerDuty) Relevant team lead Engineering Manager #incidents
SEV3 On-call engineer (Slack) Relevant team lead Not required #incidents-low
SEV4 Team responsible for service Not required Not required Ticket only

On-Call Rotation

Week Primary Secondary
Odd weeks Engineer A Engineer B
Even weeks Engineer C Engineer D

On-call schedule: Managed in PagerDuty Rotation cadence: Weekly, handoff on Monday 09:00 UTC Override requests: Post in #on-call-swap channel

Service Ownership

Service / Component Team Primary Contact Slack Channel
Backend API (FastAPI) Backend Team Backend Tech Lead #team-backend
Frontend (React) Frontend Team Frontend Tech Lead #team-frontend
Database (PostgreSQL) Platform Team DBA Lead #team-platform
Redis / Caching Platform Team Platform Engineer #team-platform
CI/CD Pipeline Platform Team DevOps Lead #team-platform
Authentication Backend Team Auth Module Owner #team-backend
Infrastructure / Cloud Platform Team Infrastructure Lead #team-platform
Third-party integrations Backend Team Integration Lead #team-backend

Escalation Paths

Technical Escalation

On-call Engineer
    |
    v
Team Lead (for affected service)
    |
    v
Engineering Manager
    |
    v
VP of Engineering (SEV1 only, if not resolved within 30 minutes)

Data / Security Incidents

On-call Engineer
    |
    v
Security Lead
    |
    v
CTO + Legal (if data breach confirmed)

Third-Party / Vendor Issues

On-call Engineer
    |
    v
Integration Lead
    |
    v
Vendor support (using premium support channel)

Communication Channels

Channel Purpose Who Posts
#incidents Active SEV1/SEV2 incident coordination Incident commander, responders
#incidents-low SEV3/SEV4 tracking On-call engineer
#engineering Post-incident summaries Incident commander
#status-updates External-facing status updates Incident commander
PagerDuty Automated alerting and paging Monitoring system

Contact Information

Replace placeholders with actual team contacts.

Role Name Slack PagerDuty Phone (emergency)
On-call (primary) See rotation Via PagerDuty Auto-paged Via PagerDuty
On-call (secondary) See rotation Via PagerDuty Auto-paged Via PagerDuty
Backend Tech Lead TBD @backend-lead @backend-lead TBD
Frontend Tech Lead TBD @frontend-lead @frontend-lead TBD
DBA Lead TBD @dba-lead @dba-lead TBD
DevOps Lead TBD @devops-lead @devops-lead TBD
Engineering Manager TBD @eng-manager @eng-manager TBD
VP of Engineering TBD @vp-eng @vp-eng TBD

When to Page vs. When to Slack

Signal Action Channel
Service completely down Page immediately PagerDuty
Error rate > 5% Page immediately PagerDuty
Error rate 1-5% Slack notification #incidents
Performance degradation > 2x Page if sustained > 5 min PagerDuty
Single user report Investigate, no page #incidents-low
Multiple user reports Evaluate severity, likely page PagerDuty or #incidents
Scheduled maintenance issue Slack notification #incidents-low

Source: SKILL.md on GitHub

1 warning13d4 checks · Risk SAFE
  • Gen Agent Trust Hub13d

    This skill provides a structured framework for production incident response, including log analysis and report generation. The primary security considerations are a surface for indirect prompt injection via the processing of untrusted logs and a configuration mismatch between the allowed tools and the commands used in the instructions.

  • Socket13d

    No alerts

  • Snyk13d

    Risk: LOW · No issues

  • Runlayer7mo

    6/6 files flagged

Signed by skilld at 4afa41d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 2 months ago.

Dormantupdated 8 months ago
compatibility
Any backend/frontend stack
context
fork
All 1 allowed tools
Read Grep Glob Write Bash(curl:*) Bash(jq:*)
Other metadata
metadata
{
  "author": "platform-team",
  "version": "1.0.0",
  "sdlc-phase": "operations"
}

README badge

README badge for hieutrtr/ai1-skills/incident-response