Failure Handling, Execution Strategies, and Speed

Two production scars taught me this lesson's material. One: a deploy playbook that kept going after a database migration failed on the primary — ignore_errors: true someone left behind, so every replica ran migrations against a broken schema. Two: a "why does this take 45 minutes" audit that found forks: 5, no pipelining, and SSH multiplexing off. Failure semantics and execution speed are the same skill: controlling when tasks run and what happens when they don't.

block/rescue/always: structured failure

- name: Apply migration safely
  block:
    - name: Stop writes
      ansible.builtin.service: { name: app, state: stopped }
    - name: Run migration
      ansible.builtin.command: /opt/app/bin/migrate
  rescue:
    - name: Roll back
      ansible.builtin.command: /opt/app/bin/migrate --rollback
    - name: Start app back up
      ansible.builtin.service: { name: app, state: started }
  always:
    - name: Tell the on-call
      ansible.builtin.command: /opt/bin/notify-oncall "migration finished"

rescue: runs when anything in block: fails (with ansible_failed_task and ansible_failed_result available); always: runs regardless — it's finally. This replaces "the playbook died mid-flight and left the app stopped."

failed_when, ignore_errors, and honesty

- name: Check cert expiry
  ansible.builtin.command: openssl x509 -checkend 604800 -noout -in /etc/ssl/acme.crt
  register: cert_check
  failed_when: false        # non-zero is data, not failure
  changed_when: false
# ...then a later task does:  when: cert_check.rc != 0

ignore_errors: true means "I refuse to know this failed" — a code smell unless you immediately inspect the registered result. failed_when/changed_when let you teach Ansible what failure means for commands it can't introspect. any_errors_fatal: true on a play kills the whole run on first failure — the opposite extreme, correct for migrations, wrong for "patch a thousand web servers."

serial, strategies, and async

For rolling deploys, serial: "30%" on the play batches hosts and max_fail_percentage: 10 aborts the rollout when a batch goes bad. strategy: free lets fast hosts run ahead instead of lockstep per task — great for patching, wrong for anything order-dependent. And for genuinely slow work, async: 1800 + poll: 0 fires the task and moves on; a later async_status task with until: job.finished collects it.

The speed knobs, in order of payoff

# ansible.cfg
[defaults]
forks = 50                 # parallel hosts — the default of 5 is why you're slow
gathering = smart          # only gather facts when the cache is stale
fact_caching = jsonfile
fact_caching_connection = ~/.ansible/facts

[ssh_connection]
pipelining = True          # pipe modules via stdin, skip the temp-file round trips
ssh_args = -o ControlMaster=auto -o ControlPersist=60s

ControlPersist amortizes the SSH handshake across a play; pipelining (needs requiretty off in sudoers) cuts per-task round trips. Together they're routinely a 3-5x win before anything clever. Profile with profile_tasks before reaching for Mitogen or pull mode.

Field notes

• serial batches are per-play — a playbook with three plays re-batches each one.
• delegate_to: localhost + run_once: true for "do this once total" (e.g., create the LB pool), not once per host.
• poll: 0 without a later async_status means you fired and forgot — always pair them, or the task is a rumor.