Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Flakiness Wastes Developer Time and What to Do
Transcript
- Lucas: You push a commit. CI turns green. You merge. Two hours later, someone says, 'Hey, that test failed on main.' You rerun it. It passes. No code changed. That is a flaky test, and it is quietly destroying developer productivity across the industry. Luna: I have been on teams where we just accepted flaky tests as a fact of life. 'Oh, that one fails about 5% of the time, just rerun it.' But it adds up. Lucas: It really does. Google published a study last year based on internal data from 2025. They estimated that flaky tests cost their engineering organization roughly 16 developer-years per quarter. That is the equivalent of 64 full-time developers spending every hour just rerunning and debugging tests that should not have failed. Luna: Sixteen developer-years. Per quarter. So that's like 64 people doing nothing but fighting ghosts. Lucas: Exactly. And that is just Google. If you scale that across the industry, the wasted time is astronomical. But the fix isn't just 'write better tests.' There are structural patterns that cause flakiness, and once you know them, you can systematically eliminate them. Luna: What are the main categories? I've seen tests fail because of test order — one test leaves state that another test trips over. Lucas: That's the first big bucket: order-dependent tests. When tests share global state — a database, a filesystem, a singleton object — and they don't clean up after themselves, the order of execution matters. Run them in one order, they pass. Run them in another, they fail. The fix is to enforce deterministic ordering or, better, to isolate each test's state completely. Luna: Right. But then you have race conditions — tests that assert on asynchronous behavior without proper synchronization. Like waiting for a callback that hasn't fired yet. Lucas: That's the second bucket. Race conditions are especially common in UI tests or tests that involve network calls. A test clicks a button and immediately checks for a success message. But the message takes a few hundred milliseconds to appear. So the test fails 10% of the time. The fix is to use explicit waits — not arbitrary sleep timers, but condition-based waits that poll for a specific state. Luna: And the third bucket? That has to be brittle assertions — tests that assert on something that is technically correct but too precise. Like checking the exact text of an error message that includes a timestamp or a random ID. Lucas: Exactly. Brittle assertions are tests that fail because the assertion is too fragile for the actual behavior. A classic example is a test that asserts the exact HTML of a rendered page. One whitespace change from a frontend developer breaks the test. The fix is to assert on semantic properties, not exact string matches. Luna: Okay, so three buckets: order dependence, race conditions, brittle assertions. But even if you fix those, new flaky tests get introduced all the time. How do you manage that at scale? Lucas: That is where quarantine systems come in. The idea is simple: if a test fails, the CI system automatically reruns it. If it fails again, it's quarantined — removed from the main test suite until a human investigates. This prevents flaky tests from blocking merges while still tracking the issue. Luna: I've seen teams do that, but then the quarantined tests just pile up. Nobody ever gets around to fixing them. Lucas: That is a real risk. So you need a process: each quarantined test gets a ticket, and the team allocates a fixed percentage of each sprint — say 5% — to either fix or delete quarantined tests. Google's study found that teams that enforced that rule reduced their flaky test inventory by 80% over six months. Luna: Eighty percent is huge. But I want to push back on one thing: sometimes a flaky test is actually a symptom of a real problem in the code. I had a test that would fail intermittently because of a subtle race condition in the production code itself. The test wasn't flaky — it was catching a real bug that only happened under load. Lucas: That is an important nuance. Not every flaky test is a false alarm. Sometimes the test is correctly detecting non-deterministic behavior in the system under test. The key is to triage each flaky test: is it a fake failure caused by test infrastructure, or is it a real race condition in production that you need to fix? Luna: Exactly. And that takes time. But I think one concrete thing teams can do is introduce a 'flaky test budget' — a maximum number of flaky tests you tolerate before you stop feature work and fix them. Lucas: That is a great idea. We talked earlier about how listener support keeps shows like this going. If today's tech conversation gave you something usable, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Luna: Yeah, it really helps. Even small contributions make a difference. Lucas: So back to the flaky test budget — you set a threshold, say five flaky tests, and once you hit it, the team pauses new features and spends the next sprint cleaning them up. That creates accountability. Luna: I've seen that work. And I think there's a fourth technique that's underused: deterministic test ordering. By default, many test frameworks run tests in random order to catch order dependencies. But if you shuffle them, you get flaky failures that are hard to reproduce. Instead, run tests in a fixed, repeatable order, and only randomize in a separate CI job designed to detect order dependencies. Lucas: That is smart. You get the best of both worlds: a stable CI pipeline for developers, and a separate signal for the team to fix underlying issues. There is also the rerun strategy itself. Some teams rerun a failing test up to three times immediately. That can mask flakiness, but it also reduces false positives. Luna: What is the right number of reruns? I've heard everything from zero to five. Lucas: Google's recommendation is to rerun once, and if it fails again, quarantine it. Three reruns is too many because it hides chronic flakiness. One rerun gives you enough signal to distinguish a transient glitch from a genuine bug. Luna: And then there is the exponential backoff approach: if a test fails, wait a bit and rerun. If it fails again, wait longer. That works well for race conditions that depend on timing. Lucas: Right. But the ultimate goal is to have zero flaky tests. That might be unrealistic for large systems, but it should be the target. I want to ask you about something you mentioned earlier — the third-party API mock that was causing flakiness on your team. Luna: Oh yeah. We had a test that mocked a payment API. The mock would sometimes return a timeout, but the test expected a success response. The problem was that the test was flaky because the mock was flaky — it was a poorly written mock that introduced its own race conditions. Lucas: That is a great example of how flakiness can come from test infrastructure itself. The fix there is to use a more deterministic mock — one that simulates failure states explicitly rather than randomly timing out. Luna: We switched to a mock that let us inject specific responses per test case. No more randomness. The test went from failing 15% of the time to passing 100%. Lucas: So the takeaway is: flakiness is not inevitable. It's a technical debt that can be measured, managed, and paid down. Start by categorizing your flaky tests into the three buckets, apply appropriate fixes, use a quarantine system with a cleanup process, and consider a flaky test budget. Luna: And maybe one day AI will automatically detect and quarantine flaky tests before they even hit the CI pipeline. Lucas: That is the frontier. Some teams are already using machine learning to analyze test failure patterns and flag likely flaky tests. It is early, but promising. For now, the manual approach still works — if you have the discipline to follow through. Luna: Alright, that was episode 63. I'm Luna. Lucas: And I'm Lucas. See you next time.