Logs, Metrics, Traces: What Each Signal Is For

Observability isn't a tool — it's three different answers to three different questions. Teams drown in data because they ask every signal every question.

Logs: "what happened, in this one instance?"

The forensic record — event detail, stack traces, the specific request that failed. High volume, low structure unless you discipline it (structured logging: fields, not sentences). Retention is expensive; logs are for investigation, not aggregation — counting errors in logs is metrics' job, badly.

Metrics: "what's the shape, over time?"

Counters, gauges, histograms — cheap, aggregatable, alertable. Requests/sec, error rate, p99 latency, queue depth. Metrics answer dashboards and pages: they're lossy by design (no per-request detail) and therefore cheap enough to keep forever.

Traces: "where did the time go?"

A trace follows one request across services — spans for each hop, with timing. When p99 is bad, the trace shows which span ate it. Tracing is sampled in high-traffic systems (head-based 1-10% or tail-based on errors) — the insight survives sampling because slow paths are overrepresented in what you keep.

Choosing

• "Is the service healthy?" → metrics.
• "Why did this request fail?" → logs, ideally correlated by trace ID.
• "Why is checkout slow only sometimes?" → traces of the slow ones.

Field notes

• A trace_id in every log line connects the two cheapest signals into most of the value of tracing.
• Start with metrics+logs; add tracing when you cross service boundaries you can't explain.
• Cardinality is the metric tax: user_id as a label is a million series and a dead Prometheus — labels are dimensions, not identifiers.