Hashing files and walking trees with hashlib and pathlib

You're triaging a box and just pulled a suspicious binary into quarantine. First questions: what's its SHA-256, and is it already in the quarantine folder under a different name? Both are hashing-and-walking problems, and both have a bug that ships constantly: reading entire files into memory, and recursing into symlink loops.

Hash a file in chunks — a 4 GB memory dump or disk image will not fit in RAM, and hashlib doesn't need it to:

import hashlib

def sha256_file(path, chunk_size=1024 * 1024):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while chunk := f.read(chunk_size):
            h.update(chunk)
    return h.hexdigest()

The walrus loop reads a megabyte at a time; memory stays flat regardless of file size. One megabyte is a comfortable chunk — the exact size barely matters, but avoid tiny reads on huge files.

For finding files, pathlib.Path.rglob handles the recursion:

from pathlib import Path

def find_duplicates(root):
    seen = {}
    for p in Path(root).rglob("*"):
        if not p.is_file() or p.is_symlink():
            continue
        try:
            digest = sha256_file(p)
        except (PermissionError, OSError):
            continue
        seen.setdefault(digest, []).append(p)
    return {d: ps for d, ps in seen.items() if len(ps) > 1}

Two details carry the load here. p.is_symlink() skips links before hashing — otherwise a quarantine copy linked back to /etc/shadow gets hashed too, and symlink cycles (a directory linking to its own parent) send rglob in circles on some filesystem layouts. And the try/except is not optional: real trees always contain files you can't read — running daemons' working files, other users' directories, mid-write logs.

Matching against threat intel

The pattern you'll use weekly: hash a file, look it up. Most public intel (VirusTotal, MalwareBazaar, abuse.ch feeds) keys on MD5, SHA-1, or SHA-256 of the file — that's lookup, not integrity verification, so MD5 is fine there. But when you sign or verify artifacts — a release tarball, a disk image, an evidence package — use SHA-256 and never MD5 or SHA-1; collision attacks against both are practical, not theoretical.

Try this

Write dupscan.py: take a directory on the command line, hash every regular file, and print groups of duplicates with their sizes. Then point it at ~/Downloads and see how much space copies are eating. Bonus: have it emit rm commands for every copy after the first, commented out, so nothing destructive runs by accident.

Pitfalls

• Encoding edge cases: filenames on Linux are bytes, not necessarily UTF-8. pathlib mostly hides this, but if you str()-format paths into a report or JSON, a name with raw bytes (b'\\xff') will crash the encoder. Use os.fsdecode or keep paths as Path objects until output.
• Don't hash directories by "file list + mtime." Order and timestamps aren't stable. If you need a directory fingerprint, hash the sorted list of (relative_path, file_sha256) pairs — deterministic and reproducible across machines.
• os.walk still earns its keep when you need followlinks=True control or want to prune subdirectories mid-walk; rglob is the comfortable default, not the only option.
• The bug that ships: calling Path.read_bytes() or open().read() on arbitrary paths in a scanner — one multi-GB pagefile or core dump and your tool eats all the RAM on the host you're triaging.