Release Manager

v3.0.0 with breaking changes and a DB migration is the highest-risk release category. Here's the complete readiness checklist:

Breaking change documentation (must be done now):

• CHANGELOG.md has a ### Breaking Changes section listing every removed v1 endpoint with migration paths
• Migration guide written at docs/migration-v2-to-v3.md — link from release notes
• API changelog updated at docs/api-changelog.md
• Customer-facing announcement drafted (email to API users + status page notice)

Database migration validation:

# Run on a production-size staging DB (not dev with 100 rows)
DATABASE_URL=$STAGING_DATABASE_URL npx prisma migrate deploy
# Time it — if >30s, you need to assess downtime impact
time DATABASE_URL=$STAGING_DATABASE_URL npx prisma migrate deploy

# Verify migration is reversible
npx prisma migrate diff --from-schema-datamodel prisma/schema.prisma --to-schema-datasource

New auth flow testing:

• Integration tests cover all auth edge cases (expired token, invalid token, refresh flow)
• Verified old v1 auth tokens are rejected (not silently accepted)
• SSO flows tested with at least one enterprise customer in staging

Go/no-go criteria (48h before cutover):

□ All CI checks green on main
□ Staging deployed with v3.0.0 for >24 hours with no P1/P2 issues
□ DB migration runs in <5 min on prod-size data (if >5 min, plan maintenance window)
□ Rollback procedure tested in staging (including DB rollback)
□ On-call engineer briefed and available +4h post-deployment
□ Feature flag for new auth flow (in case partial rollout needed)
□ External API consumers notified minimum 72h in advance (ideally 2 weeks)

Deployment sequence recommendation:

1. T-2h: Run DB migration on production (additive changes only)
2. T-0: Deploy v3.0.0 (new code reads both old and new schema)
3. T+1h: Monitor error rates — rollback trigger if 5xx >0.5%
4. T+4h: Go/no-go decision on keeping v3.0.0
5. T+1 week: Decommission v1 endpoint infrastructure

P1 — execute immediately.

T+0 (now) — Stop the bleeding:

# Option A: Rollback the deploy (fastest, <2 min)
vercel rollback [previous-deployment-url]
# OR for ECS/K8s:
aws ecs update-service --cluster prod --service api \
  --task-definition api:PREVIOUS_REVISION

# Option B: Feature flag (if email sending is behind a flag)
feature-flags set payment_confirmation_email false

T+5 min — Confirm rollback is working:

# Trigger a test payment in staging, verify email sends
# Check error logs: are new payment completions generating emails?
vercel logs --since=5m | grep -i "email\|confirmation\|sendgrid\|ses"

T+10 min — Address the 1,200 affected customers:

-- Find all payment completions in the window
SELECT id, user_id, amount, completed_at
FROM payments
WHERE completed_at BETWEEN '2026-03-09 10:20:00' AND NOW()
  AND confirmation_email_sent = false;

# Queue backfill job to send delayed confirmations
node scripts/backfill-confirmation-emails.js --from='2026-03-09T10:20:00Z'

Communicate to support team: customers may receive confirmation 1–2h late.

Hotfix branch process (for the actual fix):

git checkout -b hotfix/payment-email-fix v3.0.0  # branch from the release tag
# Make the minimal fix
git commit -m "fix(payments): restore email confirmation after queue worker crash"

# Expedited review: 1 reviewer, 15 min max
# Run only payment-related test suite:
npm test -- --testPathPattern=payment

# Version bump: v3.0.0 → v3.0.1
python3 scripts/version_bumper.py --bump patch

git tag v3.0.1
git push origin hotfix/payment-email-fix --tags

Post-incident (within 24h):

• Incident timeline written
• Root cause: why did the deploy break email sending?
• Action item: add email delivery monitoring alert that would have caught this in <5 min

Active exploitation changes everything — this is not a normal hotfix, it's an emergency response with a release component. Speed matters, but a botched deployment of a security patch at 11pm is worse than a brief delay to do it right.

Hotfix Branch Workflow

Branch from the production tag or your default branch's latest deployed commit — never from develop or a feature branch:

# Find the tag that's currently deployed
git tag --sort=-creatordate | head -5

# Branch from it
git checkout -b hotfix/sec-CVE-2026-XXXX v3.2.1
# Apply your patch
git add -p  # stage only the security fix, nothing else
git commit -m "fix(security): patch [brief description of vulnerability type]"
# Do NOT include CVE details in the commit message — push to private repo, detail in post-mortem

Do not include any unrelated changes. The delta between the current production build and your hotfix should be exactly the security patch and nothing else. This makes rollback clean and audit straightforward.

Who Must Approve

Security incident approval chain — all three must sign off before deployment:

Get verbal/Slack confirmation from all three before proceeding. One thumbs-up emoji in the incident channel from each is sufficient — you can capture the formal approval record in the post-incident review.

Minimum Viable Testing for a Security Hotfix

You cannot skip testing entirely. You can compress it:

□ Unit tests for the affected code path (must pass)
□ Smoke test: does the application start and respond to health check?
□ Regression test: does the specific attack vector no longer work?
□ Does the fix break any authentication or authorization flows? (manual spot-check)
□ Staging deploy + 5-minute soak before production

What you can skip: full regression suite, performance testing, UI acceptance testing, non-adjacent features.

Run only the tests that could catch a regression in the patched component:

# Run targeted tests, not full suite
npm test -- --testPathPattern="auth|security|middleware"
# Time box: if tests take >10 minutes, run only the directly relevant suite

Compressed Hotfix Checklist

Cannot skip:

• Patch reviewed by at least 1 engineer (not the author)
• CISO + CTO + on-call SRE approval
• Staging deployment succeeds
• Attack vector manually verified as blocked in staging
• Rollback plan documented and tested (see below)
• Version bumped (patch: v3.2.1 → v3.2.2)

Can defer to post-deployment:

• Full regression suite
• Performance benchmarks
• Documentation updates
• Full security audit of adjacent code

Rollback Plan (Define Before You Deploy)

Write this down before starting the production deployment:

# Rollback procedure — confirm this works in staging FIRST
# Option A: Re-deploy previous version
git checkout v3.2.1
./scripts/deploy.sh production

# Option B: Container rollback
kubectl rollout undo deployment/api
kubectl rollout status deployment/api

# Rollback trigger: if error rate exceeds 1% within 10 min of deployment, rollback immediately

State your rollback decision criteria explicitly in the incident channel before deploying: "If we see error rate >1% or health check failures after deployment, we roll back without discussion."

Post-Deployment Verification

# 1. Health check passes
curl -f https://yourapp.com/health

# 2. Attack vector is blocked (test the specific exploit path)
# Run a controlled test of the exploit against production — confirm 4xx, not 2xx

# 3. Error rates normal
# Check your APM for 5xx rate — should return to baseline within 2 min

# 4. Auth flows working
# Manually test login, token refresh, and any auth path adjacent to the fix

Hold 30 minutes post-deployment before declaring all-clear.

Communication

Internal (immediate, before deployment):

Security hotfix deployment in progress for active vulnerability.
Deployment window: ~30 min starting now.
Brief service interruption possible during restart.
IC: @you | SRE: @name | Security: @name

Customers: Do not publish details of the vulnerability until after deployment. Post a generic maintenance notice if downtime is expected, or nothing if it's a zero-downtime deploy.

Post-deployment (within 24h): Notify affected customers if their data was at risk — coordinate with Legal before sending. File a CVE if applicable.

Full post-mortem within 5 business days: Root cause, attack timeline, how it was missed in development, and what changes to the SDLC prevent recurrence. This is the artifact that matters long-term.