Observability Designer

400 alerts/week is a classic alert fatigue problem. The goal is high-precision alerting: every page is actionable.

Audit your current alerts into four buckets:

1. Symptom-based (keep these) — user-facing impact: 5xx rate high, latency SLO burning
2. Cause-based (convert or delete) — CPU >80%, disk >70% — these fire constantly and rarely need immediate action
3. Informational (route to Slack, not PagerDuty) — deployment notifications, scale events
4. Redundant (delete) — multiple alerts firing for the same root cause

The rationalization process:

Run this query on your PagerDuty history:

# Which alerts fired >10x last month?
gh api /orgs/{org}/... # or use PD API
# Sort by: frequency DESC, resolution_time DESC
# Alerts that fire often AND take long to resolve = noisy and hard to fix

Quick wins (implement this week):

• Remove CPU/memory alerts — replace with saturation SLO: container_cpu_throttling_ratio > 0.5 for 15m
• Add for: duration — for: 5m eliminates flapping on transient spikes
• Add hysteresis — alert fires at 95% saturation, resolves at 80%
• Group related alerts — Alertmanager group_by: [service, severity] collapses 12 alerts into 1 per service

Target state: <20 pages/week

• 12 services × 3 SLO alerts each (availability, latency P95, error budget burn) = 36 alert rules
• 36 well-tuned rules will generate far fewer pages than 400 poorly-tuned ones

Generate a Grafana dashboard showing alert frequency by rule name — that prioritization view alone will show you where to cut first.

With Datadog and Node.js, you can add traces with minimal code changes using auto-instrumentation.

Step 1 — Add the Datadog tracer (one change per service):

npm install dd-trace

// At the very top of your entry file, before any other imports
const tracer = require('dd-trace').init({
  service: 'orders-service',  // unique per service
  env: process.env.NODE_ENV,
  version: process.env.APP_VERSION,
  logInjection: true,  // auto-correlates logs with trace IDs
})

This auto-instruments Express, HTTP, and popular DB drivers — no other code changes needed for basic traces.

Step 2 — Propagate trace context through RabbitMQ (manual):

// Publisher — inject trace context into message headers
const { propagation, context } = require('@opentelemetry/api')
const headers = {}
tracer.inject(tracer.scope().active(), 'text_map', headers)
channel.publish(exchange, routingKey, Buffer.from(body), { headers })

// Consumer — extract trace context
const parentContext = tracer.extract('text_map', msg.properties.headers)
const span = tracer.startSpan('process_order', { childOf: parentContext })
// ... process message ...
span.finish()

Step 3 — Add spans for business operations that matter:

// Wrap slow business logic with named spans
const span = tracer.startSpan('compute_pricing')
try {
  const price = await computePricing(items)
  span.setTag('item_count', items.length)
  return price
} finally {
  span.finish()
}

What you'll see in Datadog APM:

• Full flame graph across all 8 services for a single request
• P50/P95/P99 per service and per operation
• Automatic anomaly detection on span duration
• Logs linked to traces via trace_id in structured log output

Sampling config for cost control:

tracer.init({ sampleRate: 0.1 })  // trace 10% of requests
// Or tail-based: always trace errors + slow requests