Reading a systemd unit file like an operator

The alert says payments-worker is "failed" and the runbook says "restart it." Before you reach for systemctl restart, spend thirty seconds reading the unit — half of production restarts go sideways because nobody knew what Restart= or an Environment= override was already doing. Unit files are the source of truth for how a process actually runs, and they are plain text you can read faster than you can guess.

Where units live

systemd reads units from three directories, in increasing order of precedence:

• /usr/lib/systemd/system/ (Debian/Ubuntu: /lib/systemd/system/) — shipped by packages. Never edit these. Your change is silently overwritten on the next package update.
• /run/systemd/system/ — runtime units, gone on reboot.
• /etc/systemd/system/ — yours. Local units and overrides live here.

Anatomy of a unit

[Unit]
Description=Payments worker
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/opt/payments/bin/worker --config /etc/payments/worker.toml
Restart=on-failure
RestartSec=5s
User=payments
Environment=PAYMENTS_ENV=production
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

[Unit] is metadata and ordering. After=/Wants= control sequencing, not hard dependencies — Requires= is the hard version, and reaching for it first is usually a mistake. [Service] defines the process: Type= tells systemd how to know the service started (simple, notify, forking, oneshot), and ExecStart= is the command — and it is not a shell, so no pipes, redirection, or $(...) unless you wrap it in /bin/sh -c. Restart=on-failure plus RestartSec= gives bounded retries; StartLimitIntervalSec/StartLimitBurst in [Unit] bounds the restart loop. [Install] only matters at enable time: WantedBy=multi-user.target means "start on a normal boot."

Overrides — the mechanism to actually use

To change a packaged unit, don't copy the whole file to /etc. Use drop-ins:

systemctl edit payments-worker   # writes /etc/systemd/system/payments-worker.d/override.conf
systemctl cat payments-worker    # effective unit: package file + every drop-in, with filenames
systemctl revert payments-worker # throw away all local overrides

ExecStart is a multi-value directive — to replace it you must clear it first, or the daemon tries to run both lines:

[Service]
ExecStart=
ExecStart=/opt/payments/bin/worker --config /etc/payments/worker.toml --debug

After editing: systemctl daemon-reload, then systemctl restart payments-worker.

Try this

Create /etc/systemd/system/heartbeat.service for a one-shot script:

[Unit]
Description=Heartbeat probe

[Service]
Type=oneshot
ExecStart=/usr/local/bin/heartbeat.sh

systemctl daemon-reload && systemctl start heartbeat.service, then read it back with systemctl cat and check the run with systemctl status. Now write a drop-in via systemctl edit adding Environment=PROBE_TARGET=10.0.0.1 and confirm with systemctl show heartbeat -p Environment.

Pitfalls

• Editing the file in /usr/lib and losing the fix on the next apt upgrade — the bug that ships.
• Forgetting daemon-reload after a manual file edit: systemd keeps running the old parsed unit.
• Type=forking without a correct PIDFile=: systemd thinks the service died instantly and restarts it forever.
• Putting shell syntax in ExecStart: ExecStart=/app/run.sh > /var/log/out silently writes nothing — redirect inside the script instead.