The day my cleanup script ate the deploy directory

The incident — a "harmless" cleanup cronjob wiped /srv/deploy on a Friday night.

What happened — the script globbed the wrong prefix. This is roughly what ran:

import glob, os, shutil

prefix = "/srv/deploy"  # was meant to be /srv/deploy/old
for path in glob.glob(prefix + "*"):
    print("removing:", path)
    # shutil.rmtree(path) if os.path.isdir(path) else os.remove(path)

Blast radius — 40 minutes of downtime before the last-good artifact was restored from CI.

The fix — restored from the CI artifact cache, then pinned the cron to an exact path.

Lessons — dry-run destructive jobs, scope globs tightly, and never trust a Friday cron.

(Try the Run button on the code block if you're signed in.)