Skip to content
live45 ptscloses 11/13/2026, 10:07:33 AM

The leak that isn't on the heap

Process memory grows 50MB/day forever. Heap dumps are flat and boring. Only a restart helps.

The scene

Reported: ingestd RSS goes 300MB → 1.4GB over 4 weeks, then the OOM killer takes it. Heap usage (jstat, heap dumps) holds flat at ~90MB, with no GC pressure.

  • This month's release added a zero-copy parser that does ByteBuffer.allocateDirect(64 * 1024) per message and lets it drop after parsing (~3k msg/s).
  • -XX:MaxDirectMemorySize unset.
  • jcmd <pid> VM.native_memory summary (NMT enabled on canary):
- Internal (reserved=1210MB, committed=1190MB)
     (malloc=1188MB #4,019,551)
  • Container base image uses glibc malloc with default MALLOC_ARENA_MAX.

Diagnoses — hidden until the incident closes

Sign in to post your root-cause diagnosis.