Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Flakiness Is Your Team's Real Bottleneck
Transcript
- Lucas: You ship a clean commit. CI turns green. Then you push the same code again an hour later — no changes — and a test fails. Your first instinct? Blame the test, not the code. And you're probably right. Luna: That's the classic flaky test. It passes and fails without any real change. And honestly, once a team starts seeing those, trust in the entire test suite starts to erode. Lucas: Exactly. And erosion of trust is maybe the single most expensive thing that can happen to a QA process. If developers learn to ignore red builds because 'oh, that test is flaky', then the whole safety net has a hole in it. Microsoft published a paper back in 2014 analyzing Windows test failures — they found 14 percent of all failures were flaky. Not from code bugs. Just environmental noise. Luna: Fourteen percent. That's huge. And that was over a decade ago. I'd imagine that number has only grown as CI pipelines have gotten more complex, more services, more microservices. Lucas: Right. More moving parts means more sources of flakiness. Google did a more recent internal survey — I believe it was 2023 — and they estimated that teams spend anywhere from 10 to 30 percent of their sprint time just re-running tests and debugging false failures. Luna: Thirty percent of a sprint. That's effectively one out of every three weeks wasted on noise. I think most engineering leaders don't even realize how much this adds up. Lucas: They don't, because flaky tests hide in plain sight. Nobody logs 'spent three hours today rerunning a test that failed because the database container didn't start in time.' People just grind through it. But the productivity drain is real, and it compounds. Luna: So what actually causes a test to be flaky? I mean, there's the classic async timing issue — waiting for an element to appear, or a response to come back — but that's just one flavor. Lucas: That's the most common one. An assertion fires before the system is in the right state. The system is eventually consistent, but the test checks immediately. Beyond timing, you get test ordering dependencies — one test assumes a certain database state left behind by a previous test, and when the order changes, it breaks. Then there's environment-specific flakiness: a network call times out in the staging environment but not locally, or a file path that works on Linux but not Windows. Luna: And those are hard to catch because they don't show up in every run. So a developer pushes a fix, the test passes locally, but in CI it fails intermittently. They assume it's environmental, they rerun, and eventually it passes. No one ever investigates the root cause. Lucas: That's exactly the pattern. And what's dangerous is that every time you rerun without investigating, you're deepening a kind of test debt. Just like code debt, test debt accumulates interest. The longer you let flaky tests live in your suite, the more normalized false failures become, and the harder it is to tell a real regression from a spurious one. Luna: So the first step is admitting you have a problem. Most teams don't even track flakiness as a metric. They track pass/fail rate, maybe coverage percentage, but not 'how many of our failures are false alarms.' Lucas: Right. And if you don't measure it, you can't manage it. So step one: surface flaky tests. Most modern CI tools — Jenkins, GitHub Actions, CircleCI — have plugins or logs that can flag tests that fail and then pass on retry. You want to tag those, and then quarantine them. Move them to a separate suite that doesn't block the build. Luna: Quarantine is key. Otherwise, one flaky test can bring your entire deployment pipeline to a halt. But then you have to actually fix the quarantined tests, not just leave them in purgatory forever. Lucas: Absolutely. Set a service-level objective. For example, no test stays quarantined for more than one sprint. If it can't be fixed in that time, you delete it. A flaky test is worse than no test, because it costs you trust and time. And here's a concrete number I like: target a flaky-failure rate of under 2 percent of all test runs. If more than 2 percent of your test executions are flaky failures, you have a systemic problem. Luna: Two percent. That's a good benchmark. I've read that Google internally targets something like that. But achieving that requires some real engineering discipline — like making tests hermetic, using deterministic data, and avoiding shared mutable state. Lucas: Yes. Hermetic tests are the ideal. Each test sets up its own data, runs in its own container, doesn't depend on external services or global state. But that's a heavy lift for an existing codebase. So teams often start with the low-hanging fruit: fix obvious async waits, add proper retry logic for network calls, and make sure test cleanup runs regardless of pass or fail. Luna: One thing I've seen work well is a 'flaky test tracker' in the team's standup. Every day, someone checks the CI dashboard for any new flaky tests. It keeps it top of mind, and it prevents that slow creep. Lucas: That's a great practice. And it ties into something we've talked about on this show before — the idea of treating your test suite as a product. You need to invest in its reliability just like you would any other software product. And I think that's a good moment to mention something about this show itself. Luna: Go ahead. Lucas: We keep Software Testing with Fexingo completely ad-free. No sponsors, no pre-rolls, no mid-roll interruptions. That's a deliberate choice — we think the content should stand on its own. If you find value in these conversations and want to support that choice, you can find us at buy me a coffee dot com slash fexingo. It's a simple way to help keep the show independent. Luna: Yeah, no ads means we can go deep on topics like flaky tests without having to cut away for a commercial. If that matters to you, the link is there. Lucas: Exactly. So back to test flakiness — one other dimension I want to touch on is the cultural side. Because even with the best tools, if your team doesn't value test stability, it'll never improve. I've seen teams where developers are expected to just rerun builds until they're green, and management doesn't question it. Luna: That's a terrible incentive. It rewards ignoring the problem. Instead, you should reward people who fix flaky tests. Make it part of the sprint goals, maybe even a rotating 'test czar' role each sprint. Lucas: Exactly. And when you do fix a flaky test, document what the root cause was. Was it a race condition? A test ordering dependency? A brittle mock? Over time, you'll build a pattern catalog, and your team will get faster at identifying and fixing flaky tests because they've seen the same causes before. Luna: That's smart. A flaky test retrospective, almost. I think a lot of teams don't realize that flakiness is often a symptom of deeper design issues — like too much coupling between components, or not enough isolation in the test environment. Lucas: Yes. Flaky tests are a signal. If you have many flaky integration tests, maybe your services are too tightly coupled. If your UI tests are flaky, maybe your application has too many async race conditions. So fixing flaky tests often leads to better architecture. Luna: That's a really good point. It's not just about making the tests pass; it's about improving the system itself. Lucas: Right. So to wrap up, here's a concrete takeaway: next week, start tracking your flaky-failure rate. Use the CI logs to identify tests that fail and then pass on retry. Quarantine them, set a two percent threshold, and don't let any test stay quarantined beyond one sprint. The goal is to make your test suite trustworthy again. Luna: And if you need help convincing your team, tell them that fixing flaky tests directly improves developer velocity. Less time rerunning, more time building features. Lucas: Exactly. Thanks, Luna. And thank you for listening. We'll be back next week with another angle on making software more reliable.