live
Checkout returns 500s only at peak
A Friday-evening traffic spike turns a healthy checkout service into a 500 factory.
The scene
Reported: 2024-11-29 19:02 UTC — Black Friday traffic ramp.
Symptom: POST /api/checkout returns 500 for ~30% of requests. All other endpoints healthy. CPU < 40% on every node. Pods up.
app logs:
19:02:11 ERROR checkout: pq: remaining connection slots are reserved for
non-replicated superuser connections
19:02:11 ERROR checkout: acquire conn: timeout after 30s
19:02:12 WARN checkout: retrying request (attempt 3)
19:02:14 ERROR checkout: pq: too many clients alreadydb metrics: pg_stat_activity.count = 412 / max_connections = 400 … idle in transaction: 287
app config: DB_POOL_MAX=200 across 3 replicas
deploys today: none.
Diagnoses — hidden until the incident closes
Sign in to post your root-cause diagnosis.