Flaky Tests: Finding and Fixing the Nondeterminism

A flaky test isn't a nuisance — it's a claim that the suite's output is unreliable, and an unreliable suite gets ignored. The fix isn't "retry until green"; it's finding the nondeterminism and removing it.

The three flake sources

Timing — the most common. A sleep(500) that's usually enough but sometimes not:

# flaky: races the render
sleep(500)
expect(page.get_by_role("button", name="Save")).to_be_visible()

# correct: wait on the condition
page.get_by_role("button", name="Save").wait_for(state="visible", timeout=5000)

Shared state — test A writes to a row test B reads; test C deletes it while B still expects it. Fix: isolated test fixtures (per-test transactions, unique IDs, seeded data per test), never a shared test database.

External dependencies — real API calls, real filesystem, real network. Fix: fake them with deterministic stubs or testcontainers — never assume an external service is up in CI.

Quarantine, don't tolerate

A flaky test that runs on every merge trains the team to click "rerun" — which is "sometimes ignore failures." Track flake rate per test (many CI systems do this natively), quarantine the offenders to a non-blocking job, and fix or delete them within a sprint. A suite that can't gate merges isn't a gate.

Detecting flake before it hits CI

# run the test 50 times — if it fails even once, it's flaky
pytest tests/test_that_flakes.py --count=50

New test? Run it 10 times locally before merging. The cost of catching a flake at write time is seconds; the cost of finding it in CI is a broken main.

::exercise(flake-source)

Field notes

• Flaky e2e tests are usually the symptom, not the disease — often the app is nondeterministic (unhandled races, uncontrolled timeouts). The test is just the reporter.
• pytest-repeat, pytest-randomly (shuffles test order), and CI's flake detection all expose different nondeterminism — use them deliberately.