Skip to content
SudoSecurity

Sections

solved30 pts

The disk is full of nothing

df says 100%. du says 2.7G of 50G. Somebody is lying, and it isn't the kernel.

The scene

Paged: 03:12 UTC — disk_used_percent > 95 on db-02. Writes start failing at 03:19.

$ df -h /var
Filesystem   Size  Used Avail Use% Mounted on
/dev/xvdf     50G   50G   20M 100% /var

$ du -shx /var
2.7G    /var
  • No bind mounts or overlays under /var; mount | grep var shows one filesystem.
  • ls -la /var/tmp is nearly empty.
  • Uptime 41 days. report-worker (PID 2231) has also been up 41 days.
  • Logrotate ran at 00:00 and reported success.
  • df -i shows inodes at 3% — not an inode problem.

Diagnoses

    The real writeup

    Root cause: a deleted-but-still-open file.

    Logrotate removed report-worker.log, but report-worker still held the file descriptor and kept appending. The directory entry is gone, so du and ls can't see it. The inode stays allocated until the last fd closes, and df counts those blocks.

    $ ls -l /proc/2231/fd | grep deleted
l-wx------ 1 app app 64 ... 5 -> /var/log/report-worker.log (deleted)

    Fix (no restart needed): : > /proc/2231/fd/5 truncates through the fd. The permanent fix is copytruncate in the logrotate stanza, or a postrotate that signals the worker to reopen its logs.

    Lesson: when df and du disagree, check lsof +L1 or /proc/*/fd before anything else. Restarting fixes it too, but then you'll never know why.