Index of every page on this site: llms.txt
All skills
patricio0312rev avatar

/postmortem-writer

@79ea6af
Run it yourself

Creates comprehensive post-incident documents with timeline, root cause analysis, contributing factors, action items, and ownership. Follows SRE best practices for blameless postmortems. Use for "postmortem", "incident review", "RCA", or "post-incident".

Use this Skill: https://skilld.dev/gh/patricio0312rev/skills/postmortem-writer

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈68 tokens for metadata: the name and description. ≈1.7k when used: this file.

Description uses 3.4% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈274 characters, or 3.4%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Postmortem Writer

Document incidents for learning and improvement.

Postmortem Template

# Postmortem: API Outage - Database Connection Pool Exhausted

**Date:** 2024-01-15
**Authors:** Jane Doe (On-call), John Smith (DBA)
**Status:** Complete
**Severity:** P1 (Critical)

## Summary

On January 15, 2024, our API experienced a complete outage for 25 minutes (14:32 - 14:57 UTC) affecting 100% of users. The root cause was database connection pool exhaustion triggered by a connection leak introduced in deployment v2.3.4.

**Impact:**

- Duration: 25 minutes
- Users affected: ~50,000
- Requests failed: ~125,000
- Revenue impact: ~$15,000

## Timeline (All times UTC)

| Time  | Event                                            |
| ----- | ------------------------------------------------ |
| 14:15 | v2.3.4 deployed to production                    |
| 14:32 | First CloudWatch alarm: HighErrorRate            |
| 14:33 | PagerDuty alert sent to on-call (Jane)           |
| 14:35 | Jane acknowledges, begins investigation          |
| 14:38 | Identified: Database connection pool at 100%     |
| 14:40 | Attempted: Kill long-running queries (no effect) |
| 14:43 | Decision: Rollback to v2.3.3                     |
| 14:45 | Rollback initiated                               |
| 14:47 | Rollback complete, connections dropping          |
| 14:50 | Error rate returning to normal                   |
| 14:57 | All systems recovered, incident closed           |
| 15:30 | Postmortem meeting scheduled                     |

## Root Cause

A code change in v2.3.4 introduced a connection leak in the user profile endpoint. The new caching layer was not properly releasing database connections after queries completed.

**Code diff:**
\`\`\`diff

- await prisma.user.findUnique({ where: { id } });

* const client = await pool.connect();
* const user = await client.query('SELECT \* FROM users WHERE id = $1', [id]);
* // Missing: client.release() ❌
  \`\`\`

## Contributing Factors

1. **Insufficient testing:** Load tests didn't catch the leak

   - Tests only ran for 5 minutes
   - Not enough concurrent connections to exhaust pool

2. **Missing monitoring:** No alerts on connection pool metrics

   - Had alarms for query latency
   - No alarms for active connections count

3. **Inadequate code review:** Reviewer didn't spot missing release()

   - PR approved without running locally
   - No checklist for connection management

4. **Deployment process:** No gradual rollout
   - Deployed to 100% of production immediately
   - No canary deployment

## What Went Well

1. ✅ **Fast detection:** Alert fired within 3 minutes
2. ✅ **Clear runbook:** DBA runbook had exact steps to follow
3. ✅ **Quick decision:** Made rollback decision in 8 minutes
4. ✅ **Communication:** Status page updated every 5 minutes
5. ✅ **Rollback capability:** Automated rollback took <2 minutes

## What Went Wrong

1. ❌ **Code review missed bug:** Connection leak not caught
2. ❌ **Testing gaps:** Load tests insufficient duration
3. ❌ **No canary:** Deployed to all instances at once
4. ❌ **Late detection:** 17 minutes between deploy and alert

## Action Items

| Action                                        | Owner   | Due Date   | Priority | Status         |
| --------------------------------------------- | ------- | ---------- | -------- | -------------- |
| Add connection pool metrics to dashboards     | Jane    | 2024-01-20 | P0       | ✅ Done        |
| Create PR checklist for connection management | John    | 2024-01-22 | P0       | ✅ Done        |
| Extend load tests to 30 minutes minimum       | QA Team | 2024-01-25 | P1       | 🔄 In Progress |
| Implement canary deployment (10% → 100%)      | DevOps  | 2024-02-01 | P1       | 📋 Planned     |
| Add connection leak detection to tests        | Jane    | 2024-01-27 | P1       | 🔄 In Progress |
| Review all DB connection usage patterns       | John    | 2024-02-05 | P2       | 📋 Planned     |
| Improve alert routing (faster escalation)     | DevOps  | 2024-02-10 | P2       | 📋 Planned     |

## Lessons Learned

1. **Code review checklists work:** Need specific items for common issues
2. **Load tests need realistic duration:** 5min insufficient for leaks
3. **Gradual rollouts catch issues:** 10% canary would have limited impact
4. **Monitoring gaps are dangerous:** Add metrics before you need them
5. **Runbooks save time:** Clear procedures enabled fast response

## Related Incidents

- [2023-11-20] Database CPU spike (similar connection pool issue)
- [2023-08-15] Memory leak in cache layer

## Prevention

To prevent similar incidents:

1. ✅ Add connection management to code review checklist
2. ✅ Monitor connection pool utilization
3. ✅ Extend load test duration
4. ✅ Implement canary deployments
5. ✅ Add automated connection leak detection

## Appendix

### Monitoring Graphs

[Insert graphs of connection pool, error rates, latency during incident]

### Communication Log

[Insert status page updates and customer communication]

### Code Fix

PR #1235: Fix connection leak in user profile endpoint
\`\`\`typescript
const client = await pool.connect();
try {
const user = await client.query('SELECT \* FROM users WHERE id = $1', [id]);
return user;
} finally {
client.release(); // ✅ Always release
}
\`\`\`

Postmortem Best Practices

# Blameless Postmortem Guidelines

## Do ✅

- Focus on systems and processes, not people
- Use timeline with exact timestamps
- Identify contributing factors, not just root cause
- Create actionable items with owners and dates
- Document what went well (positive reinforcement)
- Share widely for organizational learning

## Don't ❌

- Blame individuals or teams
- Hide or minimize the incident
- Skip the postmortem (even for small incidents)
- Create action items without owners
- Forget to follow up on action items
- Make it a blame session

## Template Sections

1. **Summary** (2-3 sentences)
2. **Impact** (numbers: users, revenue, duration)
3. **Timeline** (chronological events)
4. **Root Cause** (technical explanation)
5. **Contributing Factors** (broader context)
6. **What Went Well** (positive reinforcement)
7. **What Went Wrong** (improvement areas)
8. **Action Items** (concrete, owned, dated)
9. **Lessons Learned** (key takeaways)

Output Checklist

  • Timeline created
  • Root cause identified
  • Contributing factors documented
  • Action items with owners
  • Lessons learned captured
  • Postmortem meeting held
  • Document shared widely
  • Follow-up scheduled ENDFILE

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo5 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill provides a structured framework for creating post-incident documents based on SRE best practices. It contains no executable code, network requests, or sensitive file access.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

  • Runlayer8mo

    1/1 file flagged

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 79ea6af. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last month.

Dormantupdated 10 months ago

README badge

README badge for patricio0312rev/skills/postmortem-writer