Incident Simulator

DNS works everywhere except Building C

The new service resolves on every subnet except one. Same resolv.conf, same zone, same everything. Allegedly.

OOMKilled at 18% memory

The pod dies with exit 137 every ~40 minutes. Memory graphs are flat at 180Mi of a 1Gi limit. The kernel disagrees with Grafana.

The NAT gateway hates new friends

Long-lived connections are fine. Every brand-new outbound connection hangs, then dies. CPU idle, bandwidth idle.

EMFILE o'clock

Every weekday at 15:00 UTC the API 502s for 8 minutes, then heals itself. Weekends? Perfect.

The 30-second handshake

A fresh VM pauses exactly ~30s on every first HTTPS call after boot. The same image on bigger hardware is instant.

Cinderella cache

Every night at 00:00 the DB pegs at 100% and the site 503s for ~90 seconds. Like clockwork. Like a pumpkin.

Schrödinger's price tag

After a price change, refreshing the page flips between $49 and $39. Every. Single. Refresh.

The deadlock diner

The new nightly settlement job dies on deadlock_detected 4% of the time. Checkout code hasn't changed in months.

Down for exactly the people with good ISPs

15% of users say the site is dead. Monitoring says 100% uptime. Both are telling the truth.

Every Sunday at midnight, auth dies for an hour

Service-to-service auth fails 00:00–01:00 every Sunday. Manually rotating the token fixes it. Until next Sunday.

SSH freezes after exactly one screenful

Over the VPN, any command that prints more than ~1400 bytes freezes your SSH session forever. `ls` fine. `cat` doom.

The leak that isn't on the heap

Process memory grows 50MB/day forever. Heap dumps are flat and boring. Only a restart helps.

Log out? Roll the dice

The logout button works about half the time. Sometimes it logs you into… someone else. Behind an ALB.

40× slower because of lower()

A login lookup went 25ms → 1s after Friday's deploy. The table has an index on email. Postgres ignores it.

Partition three is having a day

Kafka consumer lag: one partition at 1.4M, every other partition near zero. Rebalancing does nothing.

Tokens born already expired

Freshly minted JWTs are rejected as expired the instant they're issued. NTP is 'on'. Only mobile breaks.

/opt is growing and nobody is writing to it

/opt/app gains ~200MB a day. find -newer shows nothing. No new files. No cron. Pure vibes.

Replica reads are a time machine

Users update their profile, hit save, and the page shows the old value. Refresh a few seconds later: correct. Sometimes 40s later.

Checkout returns 500s only at peak

A Friday-evening traffic spike turns a healthy checkout service into a 500 factory.

The backup that ran 400 nights then ghosted

Nightly backup: 400+ clean runs, then nothing. No error, no alert, no log line. Just vibes.

First blood: sudosecurity

The disk is full of nothing

df says 100%. du says 2.7G of 50G. Somebody is lying, and it isn't the kernel.

First blood: sudosecurity

Attack from the middle of the ocean

The login map shows 4,000 sign-ins from 0°N 0°E. Is a botnet floating in the Gulf of Guinea?

First blood: sudosecurity