Skip to content
SudoSecurity

Sections

solved40 pts

The backup that ran 400 nights then ghosted

Nightly backup: 400+ clean runs, then nothing. No error, no alert, no log line. Just vibes.

The scene

Reported: an auditor asks for last night's backup. The newest one is 6 days old.

crontab (root):

SHELL=/bin/sh
0 3 * * * /opt/backup/run.sh

/opt/backup/run.sh:

#!/bin/sh
exec /opt/backup/backup.py >> /var/log/backup.log 2>&1
  • backup.log last line, 6 days ago: done (412 MiB). No errors.
  • The host was rebooted 6 days ago for kernel patching. The config-management agent ran on boot and "restored managed files".
  • syslog still shows CRON[...]: (root) CMD (/opt/backup/run.sh) every night at 03:00.
  • Running sh /opt/backup/run.sh by hand works perfectly.
  • Alerting is configured on "backup job exit code != 0".

Diagnoses

    The real writeup

    Root cause: the script lost its execute bit, and the alerting couldn't notice.

    On boot, the config-management agent re-materialized /opt/backup/run.sh from its template with the default mode 0644. cron still launches the command every night (hence the CMD line in syslog), but the exec fails with Permission denied. That error goes to cron's mail, which nobody reads. The script never runs, so it never writes to backup.log and never returns a non-zero exit code to anything that alerts.

    sh run.sh works because it doesn't need +x. That was the tell.

    Fix: set mode: "0755" in the managed-file resource. Change the alert to "newest backup older than 26h", which checks the outcome instead of the exit code.

    Lesson: alert on what the job should leave behind, not on whether it failed. A job that never starts never fails.