Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Flakiness Is a Team Problem Not a Code Problem
Transcript
- Lucas: So here's a scenario I've seen at three different companies now. You push a tiny CSS change — like a button color. Your CI suite runs four hundred tests. Three fail. You rerun them. They pass. No one touches the code. That's flaky. Luna: And everyone's first instinct is to blame the test itself — 'this test is broken, someone fix it.' But you're saying that's the wrong lens? Lucas: Exactly. The test is symptomatic of a deeper issue. Look, a flaky test means your test suite has lost its authority. Developers stop trusting the green checkmark. They start merging without confidence. And that's a team culture problem, not a code problem. Luna: Okay, so let's pin that down. What does a 'team problem' look like in practice? Lucas: Say you've got a microservice that calls an external API. The test mocks that API, but the mock has a race condition — sometimes the mock response arrives after the timeout, sometimes before. The test logic is fine. The mock logic has a subtle timing bug. But no single person owns that mock. It was written by a contractor two years ago, and everyone assumes someone else will fix it. Luna: Right — it becomes this zombie that nobody wants to touch. And since it only fails, what, ten percent of the time? It's easy to ignore. Lucas: Exactly. And that's where the culture piece comes in. If the team commits to a flakiness budget — say, no more than zero point five percent flaky failures per suite per day — then that mock becomes a priority. You either fix it or quarantine it. Luna: Quarantine meaning you move the test out of the critical path, right? So it still runs, but it doesn't block the build. Lucas: Exactly. It goes into a separate bucket. The team reviews it weekly. If it's still flaky after two weeks, you delete it. And that sounds aggressive, but honestly, a flaky test is worse than no test at all. Luna: I've heard engineers push back hard on deletion. 'But we spent time writing it, it covers an edge case.' Lucas: Yeah, I get that. But let's be honest — if the test is flaky, it's not covering that edge case reliably. You're getting false negatives, which waste debugging time, and false positives, which erode trust. A deleted test is honest. A flaky test is deceptive. Luna: So what's the right process for deciding whether to fix or delete? Lucas: First, root cause analysis. Not just 'this test failed' — why did it fail? Was it a race condition? Network timeout? Unstable test data? If the root cause is in the test infrastructure — like a shared database that another test polluted — that's a systemic fix. If it's a logic error in the test itself, that's a quick fix. But if it's a timing issue in a complex integration, sometimes the smartest move is to delete the test and replace it with a more targeted unit test. Luna: So the idea is: don't just patch the test. Ask whether the test is even worth keeping at that level of the pyramid. Lucas: Precisely. And this brings us back to ownership. If a test has no clear owner — no person or team who's responsible for its health — it will inevitably rot. So one thing I've seen work is assigning a 'test guardian' per module. That person reviews flaky tests during sprint planning, not as an afterthought. Luna: That makes sense. It's basically treating test debt like code debt — you have to allocate time for it. Lucas: Right. And there's a great case study from a mid-size SaaS company I worked with. They had around twelve hundred end to end tests. About eight percent were flaky. The team spent roughly twenty hours per sprint just rerunning and triaging failures. They implemented a flakiness budget — one percent — and quarantined any test above that threshold. Within two months, flaky tests dropped to under two percent, and developer confidence in CI went way up. Luna: What did they do with the quarantined tests? Did they fix them or delete them? Lucas: Mix of both. About half got fixed — usually by adding proper waits or isolating test data. The other half were deleted and replaced with integration tests at a lower level. The key was that they made the decision systematically, not case by case in a panic. Luna: And that systematic approach — that's the team culture shift you're talking about. It's not about being tough on tests. It's about being honest about what your test suite is telling you. Lucas: Exactly. And honestly, this is the kind of conversation that makes me glad we keep this podcast ad-free. We can talk about practical engineering culture without pitching a tool. If today's conversation gave you something usable, and you want to support that choice, the link is buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to keep the show independent and focused on what actually works. No ads, no sponsors — just engineers talking through real problems. Lucas: So back to flakiness. Another angle is test impact analysis. Instead of running your entire suite on every commit, you run only the tests that could be affected by the code change. That reduces the noise floor dramatically. Luna: But doesn't that require a really good mapping of code to tests? Most teams don't have that. Lucas: True. But even a coarse mapping helps. If you know that module A maps to test suite A, you can at least skip running suite B when nothing in module B changed. And the benefit is twofold: fewer tests means fewer opportunities for flaky failures, and faster feedback. Luna: So test impact analysis reduces flakiness by reducing the total surface area of tests that run. That's clever. Lucas: Yeah. And it forces teams to think about test granularity. If your tests are too coarse — one giant test that covers an entire feature — impact analysis doesn't help. You need smaller, more focused tests. That's a design principle that pays off in multiple ways. Luna: I want to push back a little on something. You said deleting a flaky test is often the right call. But what if that test covers a critical business flow? Like, an end to end test for checkout. Even if it's flaky, isn't it better to have it than nothing? Lucas: I'd argue no — not if it's flaky. Because here's what happens: that test fails during a release. The team investigates for two hours. They can't reproduce. They decide it's flaky and bypass it. The release goes out. Four days later, a real checkout bug surfaces in production. The team then wonders why the test didn't catch it. But the test did catch it — that flaky failure was real. They just didn't trust it. Luna: Ouch. So the flaky test actually gives a false sense of security and then a false sense of alarm. Lucas: Exactly. That's the hidden cost. A flaky test trains your team to ignore failures. And once that habit sets in, real bugs slip through. So I'd rather have no test and rely on code review and monitoring than have a test that cries wolf. Luna: That's a strong stance. But I've seen teams that are terrified to delete tests because the test coverage metric drops. Management sees the number go down and asks questions. Lucas: And that's another culture problem — measuring the wrong thing. Coverage percentage tells you almost nothing about test quality. A suite with eighty percent coverage and ten percent flaky is less reliable than a suite with sixty percent coverage and zero flaky. Measuring flakiness rate and mean time to detect real bugs — those are better metrics. Luna: So what's the first step for a team that wants to tackle flakiness today? Lucas: Audit your test suite. Run each test ten times. Any test that fails at least once in those ten runs is flaky. Quarantine all of them. Then decide per test: fix or delete. Set a two-week deadline. After that, any test still flaky gets deleted. That cleans the slate and lets you rebuild with ownership. Luna: And after the cleanup, how do you prevent new flaky tests from creeping in? Lucas: Two things. First, make every test review include a flakiness check — does the test have hardcoded waits? Does it depend on shared state? Second, enforce a flakiness budget in CI. If a new test introduces flakiness above the threshold, it gets rejected. That shifts the responsibility to the developer writing the test. Luna: So it's really about accountability and process, not just better mocking libraries. Lucas: Exactly. The tools help, but the root cause is almost always human. And I think that's the interesting part — flaky tests are a mirror of team dynamics. Fix the team, and the tests follow. Luna: Alright, final question. Do you think AI will eventually solve flaky tests? Like, an AI that automatically detects and fixes race conditions in test code? Lucas: I think AI can help detect patterns — like which tests tend to fail together, or which environment variables correlate with failures. But the ownership problem? That's not a technical fix. As long as teams treat tests as second-class code, flakiness will persist. AI might mask it, but the culture still needs to change. Luna: That's a good note to end on. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.