Observability and Designing for Failure

The difference between a system that works and a system that stays working is observability — knowing why it's behaving the way it is, before users tell you. Design for failure isn't optional; it's the difference between an incident and an outage.

The three pillars

• Logs — structured ({"event": "order.created", "order_id": "..."}), not printf. A request ID that flows through every service lets you trace a single user action end-to-end.
• Metrics — counters (requests served), gauges (memory in use), histograms (latency distribution). RED metrics for services (Rate, Errors, Duration); USE for resources (Utilization, Saturation, Errors).
• Traces — a distributed span across service boundaries: which hop took the time, which one errored. OpenTelemetry is the standard; without it, debugging a slow request is guesswork across ten services.

SLOs and error budgets

An SLO is a promise: "99.9% of requests succeed in <200ms." An error budget is what's left — 0.1% failure rate allowed before the SLO breaks. The budget is what lets you ship fast: when it's full, you take risks; when it's spent, you freeze features and fix reliability. SLOs turn "is it fast enough?" into a measurable contract.

Failure handling patterns

• Timeouts — every outbound call needs one; a missing timeout is a thread waiting forever.
• Retries with backoff + jitter — retry at 100ms, 200ms, 400ms + random; retries amplify load on a failing dependency, so cap them and jitter to prevent thundering herds.
• Circuit breaker — after N consecutive failures to a dependency, stop calling it for a cooldown period (fail fast instead of waiting for timeouts). Half-open → probe → close when healthy.
• Bulkheads — isolate thread pools/connection pools per dependency so one failing service doesn't exhaust shared resources.

Field notes

• A dashboard that doesn't answer "is it broken, and where" in 30 seconds is a screenshot, not a dashboard.
• Alerts on symptoms (user-facing latency, error rate), not causes (CPU, disk) — causes are debugging aids, symptoms are what pages you.
• Postmortems without blame are how the system improves — the failure is always in the design, never in the operator.