Runbook Generator

This is a runbook gap that compounds every incident. Here's the connection pool exhaustion runbook, purpose-built from your stack.

docs/runbooks/incident-db-connection-pool.md

# Incident Runbook: Database Connection Pool Exhaustion
**Stack:** Django + RDS PostgreSQL + EKS
**Target resolution time:** <10 min
**Last verified:** 2026-03-09

## Alert triggers this runbook
- `django.db.utils.OperationalError: FATAL: connection pool exhausted`
- `psycopg2.OperationalError: could not connect to server`
- P95 latency spike (often symptom of pool exhaustion)

## Phase 1 — Confirm (2 min)

```bash
# Check connection count vs max
psql $DATABASE_URL -c "
SELECT count(*) as active,
       (SELECT setting::int FROM pg_settings WHERE name='max_connections') as max
FROM pg_stat_activity;"

# If active/max > 0.8 → connection exhaustion confirmed

✅ If active < 80% of max: this is NOT pool exhaustion → escalate to L2

Phase 2 — Immediate mitigation (3 min)

# Kill idle connections held too long (>10 min idle)
psql $DATABASE_URL -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
  AND query_start < now() - interval '10 minutes'
  AND pid <> pg_backend_pid();"

✅ Expected: connection count drops. Django workers start accepting requests.

# Identify which app component is leaking connections
psql $DATABASE_URL -c "
SELECT application_name, count(*)
FROM pg_stat_activity
GROUP BY application_name ORDER BY count DESC;"
# If one Django worker has 80 connections: that pod has a leak

Phase 3 — Root cause

# Restart the leaking pod
kubectl get pods -n production | grep api
kubectl rollout restart deployment/api -n production

# Verify pool recovers
watch -n5 'psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"'

✅ Expected: connection count stabilizes below 80% of max_connections

Post-incident action items

• Add PgBouncer (connection pooler) — RDS direct connections are expensive
• Set Django CONN_MAX_AGE = 60 to reuse connections without leaking
• Add alert: pg_stat_activity_count / pg_settings_max_connections > 0.8

**Staleness check:** This runbook references `pg_stat_activity` (always current) and `CONN_MAX_AGE` from `settings.py` — run `git log -1 --format=%ci -- app/settings.py` to verify it hasn't changed.

Here's a CI job that detects when config files referenced in runbooks have changed more recently than the runbook's Last verified: date.

Step 1 — Add a staleness header to each runbook:

# docs/runbooks/deployment.md
**Last verified:** 2026-03-09
**References:**
- vercel.json
- prisma/schema.prisma  
- .github/workflows/deploy.yml

Step 2 — CI staleness check script:

#!/usr/bin/env python3
# scripts/check-runbook-staleness.py
import subprocess, re, sys
from pathlib import Path
from datetime import datetime

errors = []
for runbook in Path('docs/runbooks').glob('*.md'):
    content = runbook.read_text()
    
    # Parse Last verified date
    match = re.search(r'\*\*Last verified:\*\*\s*(\d{4}-\d{2}-\d{2})', content)
    if not match:
        errors.append(f"{runbook}: Missing 'Last verified:' date")
        continue
    last_verified = datetime.fromisoformat(match.group(1))
    
    # Parse referenced config files
    refs = re.findall(r'^- (.+)$', content[content.find('**References:**'):], re.MULTILINE)
    
    for ref_file in refs:
        ref_file = ref_file.strip()
        if not Path(ref_file).exists():
            continue
        result = subprocess.run(
            ['git', 'log', '-1', '--format=%ci', '--', ref_file],
            capture_output=True, text=True
        )
        if not result.stdout.strip():
            continue
        file_modified = datetime.fromisoformat(result.stdout.strip()[:10])
        if file_modified > last_verified:
            errors.append(
                f"{runbook.name}: '{ref_file}' modified {file_modified.date()} "
                f"(runbook last verified {last_verified.date()})"
            )

if errors:
    print("STALE RUNBOOKS DETECTED:")
    for e in errors: print(f"  - {e}")
    sys.exit(1)
print("All runbooks up to date")

Step 3 — Add to GitHub Actions (runs weekly):

# .github/workflows/runbook-staleness.yml
name: Runbook Staleness Check
on:
  schedule:
    - cron: '0 9 * * MON'  # Monday 9am
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python3 scripts/check-runbook-staleness.py

This will flag all 8 of your runbooks immediately — then you schedule 1h to update them, add the **References:** sections, and set today's date. The weekly job keeps them fresh going forward.