EMFILE o'clock

Reported: api-03 throws 502s 15:00–15:08 UTC, weekdays only. It recovers without anyone touching it.

App log during the window:

accept4: too many open files
accept4: too many open files

• ulimit -n = 65535, confirmed via /proc/<pid>/limits.
• ls /proc/<pid>/fd | wc -l during the window: 65535. Normal: ~2,000.
• ~90% of those fds are TCP ESTABLISHED -> 10.0.9.17:5432 (pgbouncer).
• A weekday-only finance export starts at 15:00 and pushes a burst of failing reports (missing FX rates) through the API.

db helper:

def run(q):
    conn = pool.getconn()
    rows = conn.execute(q).fetchall()   # raises on bad report
    pool.putconn(conn)
    return rows