Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Flakiness Wastes Developer Time and What to Do
Transcript
- Lucas: So last week I was talking to a friend who's a senior engineer at a mid-size fintech company — about 200 engineers — and he mentioned something that really stuck with me. He said that in their last quarter, flaky tests consumed roughly 20 percent of their developer time. Luna: Twenty percent? That's like one day a week for every dev just dealing with tests that can't make up their mind. Lucas: Exactly. And that's the thing about flaky tests — they're not just a nuisance. They are a measurable drag on productivity. A test that passes nine times out of ten but fails once for no clear reason erodes trust in the entire suite. Engineers start ignoring failures, or they rerun the pipeline until it's green, and suddenly your CI is a slot machine. Luna: Yeah, I've definitely been guilty of the 'rerun until green' approach. It's tempting when you're under deadline. But the cost adds up. Lucas: It does. And the fintech case my friend described had a specific pattern. Their biggest source of flakiness was race conditions in integration tests — especially around async messaging. They had services that communicated via Kafka, and the tests would sometimes complete before the message was fully processed. So the assertion would fail, but it wasn't a real bug. Luna: Right, so the test was faster than the system under test. Classic async problem. Lucas: Classic, but hard to fix without making tests slower. They tried adding fixed wait times, but that made the suite balloon to two hours. Then they tried polling with backoff, which was better, but still not deterministic. Eventually they moved to a virtual time approach — using test containers that could advance time in a controlled way. That cut flakiness by about 70 percent. Luna: Virtual time is such a clever pattern. You basically decouple the test clock from real wall time so you can simulate delays instantly. But it does require infrastructure changes. Not every team can just drop that in. Lucas: No question. And that's the tension. The teams that suffer most from flaky tests are often the ones that can least afford to overhaul their test framework. So what do you do in the meantime? One practice that I think is underused is the 'quarantine.' When a test is identified as flaky, you move it to a separate suite that runs less frequently — maybe nightly — and you log every failure with a stack trace. Then you prioritize fixing it based on how often it flakes. Luna: I like that because it preserves the test — you don't delete it — but it stops poisoning your CI pipeline. And it gives you data to triage. Lucas: Exactly. And another approach that's gaining traction is what Google calls 'retry budgets.' Instead of an unlimited rerun, you allow a test to retry a fixed number of times — say three — but if it fails all three, you fail the build. That way a one-off transient environment hiccup doesn't block you, but a genuinely broken test still gets flagged. Luna: That makes sense. But retries can mask real regressions. If your test infrastructure is flaky enough, you might ship a bug because the test never failed consistently. Lucas: Absolutely. Retries are a band-aid, not a cure. The real fix is making your tests deterministic. And that often means investing in things like database snapshots, mock servers, and fixture factories that give you a known state every time. It's upfront cost, but it pays back in trust. Luna: So what's the first step for a team that's drowning in flaky tests? Where do they even start? Lucas: First, measure. You can't fix what you don't track. Look at your test history — most CI platforms give you pass/fail trends. Identify the tests that flake more than, say, two percent of the time. Then categorize the failures: are they async timeouts, shared state pollution, or environment issues like disk space or network latency? Luna: I've seen some teams use a simple dashboard that tags tests by failure reason. That alone helps prioritize. Lucas: Yeah. And once you have that data, you can decide whether to quarantine, add retries, or invest in infrastructure. The fintech team I mentioned ended up spending about three sprints on test infrastructure — moving to deterministic test containers and eliminating shared mutable state. After that, their flaky test rate dropped from about 15 percent to under one percent. Luna: Fifteen percent flaky is brutal. But three sprints of investment to get it under one percent sounds like a no-brainer, especially when you're losing 20 percent of dev time. Lucas: Right. And it's not just about time. Flaky tests erode confidence. When developers stop trusting the test suite, they start skipping tests, or worse, they stop running them locally. That's how regressions sneak in. So the ROI on fixing flakiness is both productivity and quality. Luna: I also think there's a cultural angle. Teams that treat flaky tests as 'just a fact of life' never fix them. You need someone to own the problem — a 'test guardian' or rotating duty. Lucas: Totally. And honestly, if today's conversation gave you something usable — maybe a concrete step to take on Monday — that's the kind of thing that makes this show worth doing. If you found value in it, you can always throw a coffee our way at buy me a coffee dot com slash fexingo. No pressure, but it genuinely helps keep this ad-free and focused on the stuff that matters. Luna: Yeah, every little bit adds up. And we really appreciate the listeners who do. Now, back to the fintech case — one thing I was curious about: they moved to deterministic test containers, but what about tests that need to interact with external APIs? Those are notoriously hard to make deterministic. Lucas: That's a great point. For external APIs, the best practice is to use contract testing. You record the expected response from the real API once, then use a mock server that replays that response. That way your test is deterministic — it always sees the same data. And you run a separate integration test against the real API periodically to catch when the contract changes. Luna: Right, so you split the problem: fast deterministic tests for your logic, and a slower but real check for external dependencies. I think a lot of teams try to do everything in one test and that's where the flakiness comes from. Lucas: Exactly. Another common source is shared mutable state — like tests that write to the same database table or S3 bucket in parallel. If test A inserts a row and test B expects a specific count, you get flakiness. The fix is to isolate each test's data: use unique keys, temporary tables, or test fixtures that clean up after themselves. Luna: I've seen teams use a pattern called 'test fixtures per test' — each test creates its own set of data in a transaction that gets rolled back. That way there's zero interference. Lucas: Exactly. And that's actually what the fintech team did for their integration tests. They used a wrapper that started a transaction before each test and rolled it back after, so the database was always in a known state. That eliminated a whole class of flaky failures. Luna: So if I'm on a team that's struggling, the takeaways today are: measure your flakiness, categorize the causes, quarantine the worst offenders, and then invest in deterministic infrastructure — starting with async handling and shared state isolation. Lucas: That's a perfect summary. And one last thing: don't underestimate the value of logging. Every flaky test failure should produce enough context to replay the scenario. If you can't reproduce it, you can't fix it. So invest in good diagnostic output — timestamps, thread IDs, request traces. That alone can cut debugging time in half. Luna: Good point. Flaky tests are a symptom of systemic issues in your test design or infrastructure. They're not just random bad luck. Lucas: Right. They're a signal. And the teams that treat them as such — rather than just rerunning the pipeline — end up with faster, more reliable deployments. And honestly, that's the goal.