live40 ptscloses 11/10/2026, 10:07:33 AM

Cinderella cache

Every night at 00:00 the DB pegs at 100% and the site 503s for ~90 seconds. Like clockwork. Like a pumpkin.

The scene

Reported: 00:00:00–00:01:30 UTC every night: DB active queries go 20 → 900, p99 → 20s, health checks fail, the LB pulls nodes.

def get_product(pid):
    v = cache.get(f"product:{pid}")
    if v is None:
        v = db.query(PRODUCT_SQL, pid)   # ~2s
        cache.set(f"product:{pid}", v, ttl=86400)
    return v

~2,000 rps, concentrated on ~40 hot products.
A nightly job at 00:00 does cache.set(...) on every product to "warm" the cache.
Redis INFO keyspace at 00:00:05: hits drop to near zero for ~90s.

The scene

Diagnoses — hidden until the incident closes