Hardline docs

Everything you need to run Hardline. New here? The getting started guide walks you through your first server, and guides covers real-world setups. Short version: audit, then apply one phase at a time, then confirm from a new SSH session.

Requirements

• Ubuntu 26.04, Ubuntu 24.04, Ubuntu 22.04, Debian 13, Debian 12 and RHEL 9 / Rocky Linux 9 / AlmaLinux 9. Hardline refuses any other OS rather than guess.
• Root (sudo).
• An SSH key that already works for your user, before you apply phase 1.
• Ideally a provider web console, as a second way in while you learn the tool.

Install

Download the binary for your CPU, verify the release signature and checksum, and install it where only root can write. Signature Verified Successfully proves the checksum list came from SudoSecurity; OK proves your download matches it.

cd "$(mktemp -d)"
base=https://sudosecurity.org/hardline/releases/latest
bin=hardline-$(uname -m)-unknown-linux-gnu        # x86_64 or aarch64
curl -fsSLO "$base/$bin" -O "$base/SHA256SUMS" -O "$base/SHA256SUMS.sig"

# The SudoSecurity release key. Pin it; don't fetch it from the same server.
cat > release.pub <<'EOF'
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAPD8oWSZySVfdT2iWIXtGhZ81gxiRJ9SK3OWReFUf25A=
-----END PUBLIC KEY-----
EOF
openssl pkeyutl -verify -pubin -inkey release.pub -rawin -in SHA256SUMS -sigfile SHA256SUMS.sig
sha256sum -c --ignore-missing SHA256SUMS            # must print: OK

sudo install -o root -g root -m 755 "$bin" /usr/local/bin/hardline

Needs OpenSSL 3 for Ed25519 (every supported distro ships it). The revert timer runs the binary as root, so Hardline refuses to arm it from a path a normal user could change, like ~/.cargo/bin. On RHEL-family systems, sudo doesn't search /usr/local/bin; run sudo /usr/local/bin/hardline.

From source: cargo build --release --locked, then install target/release/hardline the same way.

Your first run

1. Audit. sudo hardline audit prints every check by phase, a link for each finding, and your grade. It changes nothing.
2. Preview. sudo hardline apply --phase 1 shows the exact lines it would write.
3. Apply. sudo hardline apply --phase 1 --commit validates with sshd -t, reloads SSH and starts a 120-second countdown.
4. Confirm from a new terminal. Open a new SSH connection and run sudo hardline confirm. If you can't log in, do nothing; the change reverts on its own.
5. Repeat for phases 2–8, then sudo hardline report --out /root/report.html.

Commands

hardline audit [--phase N] [--json | --sarif]

Read-only scan. Exit 0 when everything passes, 1 when there are findings, 3 when a check couldn't run. --json prints the versioned report; --sarif prints SARIF 2.1.0 for code-scanning dashboards.

hardline apply --phase N

Dry run: shows the exact files and lines that would change.

hardline apply --phase N --commit [--timeout SECS]

Applies, validates, and for SSH and the firewall waits for confirm (30–900 seconds, default 120).

hardline apply --all [--commit]

Every phase in one run. Phases 3–8 commit one by one; SSH and the firewall go last and share one revert timer, one boot guard and one confirm.

hardline apply --phase 2 --firewall nftables

Uses Hardline's own inet hardline nftables table instead of ufw. Other tables are never flushed.

hardline confirm [--from-console]

Keeps a pending change. Must run from a new SSH session; --from-console is for a provider's web console.

hardline rollback --phase N [--force]

Restores the files saved before that phase. Refuses if you edited them since, unless --force.

hardline diff [--json]

Shows drift since your last confirmed apply.

hardline status

Shows a pending change, the countdown, the last revert, and snapshots.

hardline report [--out FILE] [--audit]

Audits and writes a self-contained HTML report with an A–F grade. --audit writes a printable evidence table for auditors instead: every control with observed and expected values, STIG/CCI references, accepted risks and a sign-off line (print or save as PDF). --out creates a new root-only (0600) file and never overwrites.

hardline accept [<id> --reason "..." [--expires 90d]] / unaccept <id>

Records a finding you chose to keep, and why. It shows as accepted risk but no longer lowers the grade, fails audit, alerts watch or gets changed by apply. --expires sets a review date. No id lists them.

hardline explain <id> [--json]

Explains one control offline: why it matters, what it checks, what apply changes, how to undo it. No root needed.

hardline watch enable [--schedule hourly|daily|weekly]

Installs a systemd timer that re-audits (default daily). watch run, watch status, watch disable.

hardline watch notify set|test|remove [--format auto|discord|slack|ntfy|json]

Posts regressions to Discord, Slack, ntfy or a JSON webhook. The URL is read from stdin, tested before it is saved, and stored root-only in /etc/hardline/notify.toml.

hardline dashboard [--out FILE]

Audits and writes a local status page: the report with “How to fix” steps, a “who can get in” panel (public listening ports and their processes, accounts that can log in with admin rights, passwords and SSH key counts, and recent SSH logins), the score over time, recent watch runs, a check history (when each check started failing, for how long, and when it was fixed), a change timeline and the accepted-risk register. Default /var/lib/hardline/dashboard.html (0600). Nothing is served or sent anywhere.

hardline fleet <host>... [--hosts-file FILE]

Runs on your computer: audits each server over your SSH setup and writes one local page with every grade, worst first, and the failures they share. No root needed.

hardline controls [--json]

Lists every control for the current profile.

hardline self-update [--check]

Installs the latest signed release. Refuses a bad signature, a checksum mismatch, a downgrade, or a pending change. --check only reports.

hardline apply --phase 4 --allow-irreversible

Also makes audit rules immutable (stig only; undoing needs a reboot).

Global option: --profile baseline|stig|cis (default baseline). stig and cis only run on Ubuntu 24.04: cis is the CIS Ubuntu Linux 24.04 LTS Benchmark v1.0.0 Level 1 Server recommendations Hardline checks, and each of those controls lists its CIS numbers in refs.cis.

Phases

Profiles

baseline (default) is what SudoSecurity runs on its own servers: modern, practical settings that keep OpenSSH's strong default ciphers. stig (Ubuntu 24.04 only) uses the exact values from the DISA Ubuntu 24.04 STIG, including FIPS-only SSH algorithms, and every control lists its STIG and CCI IDs.

Watch for drift

sudo hardline watch enable installs a systemd timer that re-audits at low CPU and I/O priority. A run fails, so it shows in systemctl --failed, when the grade drops, a passing check starts failing, or someone edits a file Hardline manages. Attach an alert with OnFailure= on hardline-watch.service; each run also logs one line to the journal (journalctl -u hardline-watch.service).

Accepted exceptions

Real servers keep a few findings on purpose. Record them, with a reason, so the grade stays honest without hiding the risk:

sudo hardline accept ssh-password-authentication --reason "bastion: passwords + hardware MFA"
sudo hardline accept ssh-x11-forwarding --reason "GUI admin box" --expires 90d   # or 12w, or 2027-01-31
sudo hardline accept            # list: id, date, who, why, review date
sudo hardline unaccept ssh-password-authentication

A review date makes the acceptance temporary. When it passes, the finding counts against the grade and exit code again, the next watch run reports accepted risk … expired once, and the dashboard lists it as EXPIRED (or DUE SOON within 14 days). Dates are UTC midnight and at most five years out; entries without one keep working.

An accepted finding keeps its real status and is never counted as a pass. Audit shows it as ACPT, the HTML report as ACCEPTED, and JSON adds "accepted": {"reason", "by", "at"} plus a score.accepted count. It is left out of the grade and the audit exit code. watch doesn't alert on it, apply skips it, and SARIF reports it with a suppressions entry so GitHub code scanning shows it as dismissed. Entries live in /etc/hardline/accepted.toml, which must be owned by root and not writable by anyone else.

Alerts to Discord, Slack or ntfy

Give watch a webhook and every regression posts there. The URL is read from stdin so it never lands in shell history or ps, a test alert has to go through before it is saved, and it is stored in /etc/hardline/notify.toml owned by root with mode 600. Hardline refuses the file if anyone else can read it.

sudo hardline watch notify set < webhook.txt         # discord.com, hooks.slack.com and ntfy hosts are detected
sudo hardline watch notify set --format json < url   # anything else gets a JSON body
sudo hardline watch notify test
sudo hardline watch notify remove

The JSON format posts {"source":"hardline","version","host","at","grade","percent","regressions":[...],"test":false}. Only https:// URLs without a user name or password are accepted. curl gets the URL through its stdin config (--config -, with --proto =https, no redirects and a 20-second limit), so it never appears in arguments, the journal or Hardline's history; errors name the host only. Discord alerts set allowed_mentions to none.

Dashboard and metrics

Everything stays on the server. Each watch run appends to /var/lib/hardline/watch-history.jsonl (last 720 runs, 0600), rewrites /var/lib/hardline/dashboard.html and writes Prometheus metrics. hardline dashboard builds the same page on demand. The page is one file with no scripts and a default-src 'none' CSP, so copy it to your computer and open it:

sudo hardline dashboard
scp root@server:/var/lib/hardline/dashboard.html . && open dashboard.html

Every open finding has a How to fix box: why it matters, what Hardline changes, and the exact commands (preview, --commit, and confirm from a new SSH session for lockout-protected phases). Below the history chart, a change timeline lists the last 40 applies, confirms, rollbacks, reverts, accepts, updates and watch regressions from /var/lib/hardline/history.log, and the accepted risks register shows each acceptance with its review date.

Fleet view

Run hardline fleet on your own computer to see every server on one page. It runs hardline audit --json on each host over SSH, with your keys, agent and ~/.ssh/config, and nothing is uploaded or stored except the page:

hardline fleet web1 admin@db1 deploy@10.0.0.7 --out fleet.html
hardline fleet --hosts-file servers.txt --jobs 16    # one host per line, # comments

# on each server, if you don't log in as root:
admin ALL=(root) NOPASSWD: /usr/local/bin/hardline audit --json

Hosts must be a name, user@host or an ssh alias; anything starting with - or containing shell characters is refused before ssh runs. Connections use BatchMode=yes and a 180-second limit per host. The page lists servers worst first with their open findings, then the failures servers share, so one fix can be rolled out everywhere. Exit 0 when all are clean, 1 with open findings, 3 if a server couldn't be reached. Hardline's SSH firewall rule allows 6 new connections per 30 seconds per address, so rapid reruns can be refused.

Metrics go to node_exporter's textfile directory, /var/lib/prometheus/node-exporter/hardline.prom, when prometheus-node-exporter is installed (Ubuntu/Debian package default), otherwise to /var/lib/hardline/hardline.prom. Point --collector.textfile.directory there on other setups. No port is opened by Hardline.

hardline_grade_percent{profile="baseline"} 95
hardline_controls_passed{profile="baseline"} 48
hardline_controls_applicable{profile="baseline"} 52
hardline_controls_failing{profile="baseline"} 4
hardline_control_failing{id="ssh-x11-forwarding"} 0
hardline_files_modified 0
hardline_regressions 0
hardline_last_run_timestamp_seconds 1793155320

Ready-made files are in the repo under contrib/:

• contrib/prometheus/hardline.rules.yml: alerts HardlineRegression (a check got worse), HardlineFilesModified (a file Hardline wrote was edited), HardlineGradeLow (below 80% for an hour) and HardlineWatchStale (no watch run for 2 days). Unit tests for them: promtool test rules contrib/prometheus/hardline.rules.test.yml.
• contrib/grafana/hardline-dashboard.json: import it in Grafana (Dashboards → New → Import) and pick your Prometheus data source. Panels: grade per server, regressions, files changed, time since the last watch run, score and failing checks over time, and a table of every failing check per server.

# prometheus.yml
rule_files:
  - /etc/prometheus/hardline.rules.yml

Updating

self-update installs a new release only if it's signed with the SudoSecurity release key built into your current binary:

sudo hardline self-update --check     # "Hardline 0.6.0 is available (this is 0.5.1)"
sudo hardline self-update
Updated Hardline 0.5.1 -> 0.6.0: /usr/local/bin/hardline
Signature and checksum verified against the SudoSecurity release key.
The previous binary is kept at /var/lib/hardline/hardline.previous.

1. Fetches SHA256SUMS, SHA256SUMS.sig and VERSION from https://sudosecurity.org/hardline/releases/latest/ over HTTPS only (TLS 1.2+, no plain-HTTP redirects, 64 MiB cap).
2. Verifies the Ed25519 signature over SHA256SUMS with openssl pkeyutl -verify -rawin before reading anything else. A mirror or CDN that's been tampered with can't get past this step.
3. Parses the list strictly: 64-hex hashes, no duplicate names, no paths. VERSION must match its signed hash and be newer than the running version, so a replayed old release is ignored.
4. Downloads the bare binary for this CPU (x86_64 or aarch64) and checks it against the signed hash.
5. Refuses while a change is waiting for confirm, or if the current binary isn't owned by root or is writable by anyone else.
6. Copies the current binary to /var/lib/hardline/hardline.previous (0700), swaps the new one in with an atomic rename, and logs the update in history.

Exit codes: 0 updated or already current, 3 couldn't fetch or a tool is missing, 5 the signature or a checksum didn't match (nothing was changed). Undo an update with sudo install -m 755 /var/lib/hardline/hardline.previous /usr/local/bin/hardline.

Configuration

Hardline needs no config. To skip a control on purpose, create /etc/hardline/config.toml:

[skip]
ssh-x11-forwarding = "jump host needs X11 for legacy tooling"

The reason is required and is printed in every report, so a skip never hides silently. Config can only skip controls; it can't change what one checks.

JSON and exit codes

sudo hardline audit --json prints a stable, versioned report for CI (schema_version, grade.letter, grade.percent, one result per control).

The JSON report, field by field

sudo hardline audit --json (and --phase N to narrow it). Adding fields is a compatible change; renaming or removing one bumps schema_version, so pin on it in scripts.

{
  "schema_version": 1,
  "tool_version": "0.6.0",
  "os": "ubuntu 24.04",
  "profile": "baseline",
  "results": [
    {
      "id": "ssh-x11-forwarding",
      "phase": 1,
      "title": "Disable SSH X11 forwarding",
      "severity": "high",
      "status": "fail",
      "observed": "yes",
      "expected": "no",
      "refs": { "stig": ["UBTU-24-300022"], "cci": ["CCI-000366"] },
      "article": "https://sudosecurity.org/hardline/controls/#ssh-x11"
    }
  ],
  "score": { "passed": 30, "applicable": 36, "accepted": 0 },
  "grade": { "letter": "B", "percent": 82 }
}

Useful one-liners with jq:

# grade only
sudo hardline audit --json | jq -r '.grade.letter'
# failing high-severity control IDs
sudo hardline audit --json | jq -r '.results[] | select(.status=="fail" and .severity=="high") | .id'
# STIG IDs still open
sudo hardline --profile stig audit --json | jq -r '.results[] | select(.status=="fail") | .refs.stig[]'

SARIF and code scanning

sudo hardline audit --sarif prints a SARIF 2.1.0 log, the format GitHub code scanning, GitLab, Azure DevOps and most security dashboards import. Each failing or warning control becomes a result:

• Level: high → error, medium → warning, low → note; a warn status is always warning.
• Ranking: each rule carries security-severity (8.0 / 5.0 / 2.0) and tags for phase, profile, STIG ID and CCI.
• Location: a server has no source file, so each finding is pinned to a stable virtual path, hardline/<hostname>/<control-id>, with a matching partialFingerprints entry. Alerts stay de-duplicated across runs and one SARIF per host never collides.
• Errors: a check that couldn't run is a tool notification and sets executionSuccessful: false, so a broken probe can't pass silently.
• Grade and score are in runs[0].properties.

{
  "ruleId": "ssh-x11-forwarding",
  "level": "error",
  "message": { "text": "Disable SSH X11 forwarding: found yes; want no. Fix with `sudo hardline apply --phase 1`." },
  "locations": [{ "physicalLocation": {
    "artifactLocation": { "uri": "hardline/web-1/ssh-x11-forwarding" },
    "region": { "startLine": 1 } } }],
  "partialFingerprints": { "hardlineControl/v1": "web-1:ssh-x11-forwarding" }
}

Upload from a self-hosted runner on the server (GitHub code scanning needs a public repository or GitHub Advanced Security):

# .github/workflows/hardline.yml
on:
  schedule: [{ cron: "17 4 * * *" }]
  workflow_dispatch:
permissions:
  security-events: write
jobs:
  audit:
    runs-on: [self-hosted, web-1]
    steps:
      - run: sudo /usr/local/bin/hardline audit --sarif > hardline.sarif || [ $? -eq 1 ]
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: hardline.sarif
          category: hardline-web-1

Get alerted when watch finds a regression

hardline-watch.service exits 1 on a regression, so any systemd failure hook works. A webhook example; put the URL in a root-only file, not in the unit:

# /etc/systemd/system/hardline-watch.service.d/alert.conf
[Unit]
OnFailure=hardline-alert.service

# /etc/systemd/system/hardline-alert.service
[Unit]
Description=Tell someone Hardline found a regression
[Service]
Type=oneshot
# WEBHOOK_URL=https://... in a root-owned 0600 file
EnvironmentFile=/etc/hardline/alert.env
ExecStart=/usr/bin/curl -fsS --max-time 10 -H 'Content-Type: application/json' \
  -d '{"text":"Hardline: %H got worse. Run: journalctl -u hardline-watch.service"}' ${WEBHOOK_URL}

# then: sudo systemctl daemon-reload

Fleets: Ansible and cloud-init

Audit is read-only and safe to run everywhere. Committed changes to SSH and the firewall need a new SSH connection to confirm, which is exactly what a second Ansible task gives you, since Ansible opens a fresh connection when meta: reset_connection runs.

- name: Audit (read-only)
  ansible.builtin.command: /usr/local/bin/hardline audit --json
  register: audit
  changed_when: false
  failed_when: audit.rc not in [0, 1]

- name: Apply phase 1 (starts a 120 s revert countdown)
  ansible.builtin.command: /usr/local/bin/hardline apply --phase 1 --commit
  when: (audit.stdout | from_json).results | selectattr('phase', 'eq', 1) | selectattr('status', 'eq', 'fail') | list | length > 0

- name: Open a fresh SSH connection
  ansible.builtin.meta: reset_connection

- name: Confirm from the new connection (proves you can still log in)
  ansible.builtin.command: /usr/local/bin/hardline confirm

cloud-init can install Hardline, record a first audit in the boot log and turn on daily monitoring. Leave committed changes to a person or to the Ansible flow above, so every SSH and firewall change is proven by a real login:

#cloud-config
runcmd:
  - [sh, -c, "cd /tmp && curl -LO https://sudosecurity.org/hardline/releases/latest/hardline-x86_64-unknown-linux-gnu.tar.gz && curl -LO https://sudosecurity.org/hardline/releases/latest/hardline-x86_64-unknown-linux-gnu.tar.gz.sha256 && sha256sum -c hardline-x86_64-unknown-linux-gnu.tar.gz.sha256 && tar xzf hardline-x86_64-unknown-linux-gnu.tar.gz && install -o root -g root -m 755 hardline-*/hardline /usr/local/bin/hardline"]
  - [/usr/local/bin/hardline, audit]
  - [/usr/local/bin/hardline, watch, enable]

Gate a deploy on the grade

grade=$(ssh web-1 sudo /usr/local/bin/hardline audit --json | jq -r .grade.letter)
case "$grade" in A|B) echo "grade $grade, deploying" ;; *) echo "grade $grade, refusing" >&2; exit 1 ;; esac

Files Hardline touches

Every path Hardline can write, and nothing else. Each file is snapshotted before Hardline first changes it, so rollback puts it back byte for byte. Hardline never edits /etc/ssh/sshd_config itself.

Path	Phase	Notes
/etc/ssh/sshd_config.d/10-hardline.conf	1	Drop-in only; sorts before cloud images' 50-cloud-init.conf. sshd -t must pass.
ufw rules (/etc/ufw/user*.rules, ufw.conf), or /etc/hardline/firewall.nft + hardline-firewall.service, or the firewalld default zone	2	One backend per server; listening SSH and service ports kept open.
/etc/audit/rules.d/60-hardline.rules, 99-hardline-immutable.rules	4	Immutable rules only with --allow-irreversible (stig).
Packages: ufw, auditd/audit, apparmor, aide-common/aide, libpwquality, unattended-upgrades/dnf-automatic	2–8	Installed only when a control needs them, and logged in history.
/etc/apt/apt.conf.d/52hardline-auto-upgrades	3	Turns on daily unattended-upgrades; sorts after Debian's 20auto-upgrades. On RHEL, dnf-automatic-install.timer is enabled instead.
/etc/docker/daemon.json	6	Merged key by key, never replaced; dockerd --validate must pass.
/etc/sysctl.d/99-zz-hardline.conf, /etc/modprobe.d/hardline.conf	7	Sorts last so it wins; values applied live with sysctl.
/etc/security/pwquality.conf, /etc/login.defs, /etc/default/useradd	8	Only the managed lines; everything else is left as is.
/etc/hardline/config.toml, accepted.toml, notify.toml	—	Yours: skips, accepted exceptions, the webhook (0600). Root-owned or refused.
/etc/systemd/system/hardline-guard.service, hardline-revert.service and .timer	1–2	Only while a change waits for confirm; removed after.
hardline-watch.service and .timer	—	Only after watch enable.
/var/lib/hardline/	—	State, snapshots, history, watch baseline, hardline.previous.
The hardline binary	—	Only self-update, after signature and checksum checks.

Troubleshooting

My new SSH session can't log in

Do nothing. Keep the old session open if you still have it; the change reverts when the countdown ends. If the server reboots first, a boot guard reverts it before SSH starts. sudo hardline status shows the countdown and the last revert.

"No SSH session detected"

You're on a provider console or a local terminal. Confirm with sudo hardline confirm --from-console.

"must be owned by root and not group- or world-writable"

Reinstall the binary with sudo install -o root -g root -m 755 into /usr/local/bin. The revert timer runs it as root, so a user-writable copy would be a privilege-escalation path.

Phase 1 skips ssh-permit-root-login

You're logged in as root. Log in as a normal user with an SSH key and run sudo hardline, so turning off root login can't cut off the session you're using.

Rollback refuses because a file changed

Someone edited a file after Hardline wrote it. Check sudo hardline diff, then use rollback --phase N --force if you still want Hardline's saved copy back.

Uninstall

Run sudo hardline watch disable and sudo hardline rollback --phase N for each phase you want undone, then remove /usr/local/bin/hardline and /var/lib/hardline/. Packages Hardline installed (such as ufw or auditd) stay installed.