The NAT gateway hates new friends

Reported: the batch fleet behind nat-gw-1: existing TCP connections keep working, but new outbound connections hang 2–5s and fail intermittently. Retries eventually succeed.

nat-gw-1 dmesg:

nf_conntrack: nf_conntrack: table full, dropping packet
nf_conntrack: nf_conntrack: table full, dropping packet

$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144
$ sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait
120

• Last week the batch client's HTTP keep-alive was disabled "to spread load evenly across vendor IPs".
• Every API call now opens ~30 fresh TCP connections, each closed right after one request.