journalctl: finding the one line that matters

When a service dies at 3 a.m., the answer is almost always in the journal — but journalctl with no arguments pages through everything, which is why people grep /var/log instead and miss things. Learn the six flags that cover 95% of real queries and the journal becomes your fastest diagnostic tool.

The working set

journalctl -u nginx -b                              # this boot only, unit nginx
journalctl -u nginx --since "2026-09-30 22:00" --until "2026-10-01 02:00"
journalctl -p err -b                                # priority err and worse, this boot
journalctl -f -u payments-worker                    # follow, like tail -f
journalctl -g "timeout" -i -u app                   # case-insensitive pattern match
journalctl -o json-pretty -u app -n 5               # structured fields — scriptable

-b accepts offsets: -b -1 is the previous boot — the correct answer to "why did it crash right before the reboot?" journalctl --list-boots shows every boot still on disk.

Filters stack and AND together; repeated -u values OR. Beyond unit names you can filter journal fields directly: _PID=, _UID=, _COMM= (executable name — handy when you don't know which unit logged something), _SYSTEMD_UNIT=.

Persistence: the setting that decides whether history exists

journald's durable home is /var/log/journal/; if that directory is absent, everything lives in /run/log/journal/ — RAM-backed, wiped on reboot. Check first:

ls /var/log/journal 2>/dev/null || echo "journal is volatile"
journalctl --disk-usage

To persist: Storage=persistent in /etc/systemd/journald.conf and restart systemd-journald (or mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal). Set SystemMaxUse= — 2G is reasonable — or journald claims up to 10% of the filesystem. journalctl --vacuum-size=1G trims on demand.

For a security team this matters doubly: volatile journals mean the attacker's tracks vanish with the box. Persistent local logs plus remote shipping are table stakes for forensics.

Reading around a crash

journalctl -u app -b -p warning..alert   # severity band: warning through alert
journalctl -k -b -1                      # kernel ring buffer, previous boot
journalctl --since "-30m" | less         # everything recent, unfiltered

If a process died to the OOM killer, the unit's own log ends abruptly with no error — the truth is in the system journal (app.service: A process of this unit has been killed by the OOM killer) and the kernel's oom-kill lines, which -k surfaces. Same trick for segfaults: the kernel logs them even when the app can't.

Try this

On a box that's rebooted at least once: journalctl --list-boots, then journalctl -b -1 -p err. Read the previous boot's errors top to bottom — you will almost always find a unit that failed and nobody noticed. Then run journalctl -u ssh -g "Accepted" --since "-7d" (or _COMM=sshd) and scan who actually logged in this week.

Pitfalls

• journalctl as non-root sees little. You get your own user journal; use sudo or add operators to the systemd-journal group.
• Rate limiting eats logs. journald drops bursts beyond its configured rate (RateLimitIntervalSec/RateLimitBurst) — a crash-looping service can lose its own first, most informative lines. Raise the limits or drop duplicates at the source.
• --since "yesterday" means calendar midnight, not 24h ago. Use --since "-24h" when that's what you mean.
• Rotated-away history is gone if /var/log/journal never existed — check before assuming "there are no logs" means "nothing happened."