Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How To Fix Flaky Tests Without Killing Velocity
Transcript
- Lucas: The most expensive bug in any software system isn't the one that crashes production at three in the morning. It's the one that never happens reliably enough to fix, but fails often enough to make everyone doubt their own sanity. We're talking about flaky tests, those non-deterministic automated checks that pass when you run them once and fail when you run them twice, leaving engineering teams stuck in a loop of restarting pipelines. Luna: Right, because a failing test tells you something is broken, but a flaky test just tells you the system is lying to you. And nobody trusts a liar for long. Lucas: Exactly. And the cost isn't just emotional. When I look at the data from mid-sized tech companies in late October two thousand twenty-six, the average team spends roughly fifteen percent of their CI compute budget rerunning failed jobs that turn out to be noise. That is literally burning cash to generate uncertainty. So the question becomes, how do you stop the bleeding without slowing down release velocity? Luna: Most teams try to fix the tests themselves, which is like trying to patch a leaky roof while it's still raining. They spend weeks refactoring code that might not even be the problem. Lucas: That is the classic trap. You start digging into a flaky test, assuming it's a timing issue or a race condition, only to realize later that the test was relying on shared global state from a previous run. The real solution isn't always better code; it's better architecture around the test itself. Let's look at a concrete example from a large e-commerce platform that cut their flakiness rate from four percent to under one percent in six months. Their secret wasn't more developers; it was stricter test isolation. Luna: So they stopped letting tests talk to each other? Because historically, integration tests have always been expected to share some context. Lucas: They stopped sharing state, period. Before the overhaul, if Test A created a user account and Test B tried to log in with that same account, they were coupled. If Test A failed, Test B would also fail, but for the wrong reason. The platform implemented a rule where every test run starts with a clean, immutable database snapshot. No shared fixtures, no lingering sessions. Each test becomes a closed universe. Luna: That sounds expensive. Creating a fresh database snapshot for every single test run must add significant latency to the pipeline. Lucas: It does add latency, but it removes the unpredictability. The key insight here is that speed and reliability are trade-offs, but flakiness is a waste of both. By accepting a slight increase in setup time, they eliminated the need for retries entirely. And here is the thing: they didn't just fix the tests. They changed how the team treats failures. Luna: You mean they stopped treating every failure as a critical incident? Because usually, a red light in the pipeline stops everything. Lucas: Correct. They introduced what they call a quarantine pipeline. Any test that fails more than twice in a rolling window of ten runs gets automatically flagged and moved out of the main blocking gate. It doesn't stop deployments, but it triggers a high-priority ticket for the QA lead. This prevents one bad actor from holding up the entire release train while engineers hunt for ghosts. Luna: That feels risky though. If you let a test go green while it's actually broken, aren't you just shipping bugs faster? Lucas: Only if the test is fundamentally useless. The quarantine process requires a root cause analysis within forty-eight hours. If they can't prove it's flaky, it goes back to the main gate. If they can prove it's flaky, they either fix the isolation issue or delete the test. It forces accountability. You can't hide behind a noisy metric anymore. Luna: I like that. It shifts the focus from preventing failure to managing noise. But how do you decide which tests are worth saving? Some of those flaky tests are covering critical security paths. Lucas: That is the hardest part of the equation. Not all flakiness is equal. Security tests, for instance, should rarely be flaky because they are deterministic by nature. If a security check is failing randomly, it suggests a deeper infrastructure problem, not just a coding error. The platform categorized their tests by risk. High-risk, high-value tests get priority investment. Low-risk, low-value tests get quarantined or deprecated immediately. Luna: So it's an exercise in prioritization rather than perfection. You accept that some things will be imperfect so the rest can be reliable. Lucas: Precisely. Perfection is the enemy of deployment. If you wait until every test is perfect, you never ship. But if you allow uncontrolled chaos, you lose trust. The middle ground is controlled, visible imperfection. You know exactly which tests are unreliable, and you know why they are there. Luna: It reminds me of how financial audits work. You don't check every single receipt; you sample based on materiality and risk. Same logic applies here. Lucas: That is a perfect analogy. Materiality matters. A flaky test that checks the color of a button is annoying, but a flaky test that checks payment processing is catastrophic. The quarantine pipeline helps you separate the trivial noise from the critical signals. Luna: Okay, so we have isolation, we have quarantine, we have prioritization. What about the human element? Do developers resist moving tests to quarantine? Lucas: They absolutely do. It feels like admitting defeat. Engineers want their code to be right, and moving a test to quarantine feels like giving up on fixing it. The trick is framing it as a temporary measure, not a permanent exile. The forty-eight-hour deadline creates urgency. It turns the quarantine into a sprint goal rather than a graveyard. Luna: That makes sense. Urgency drives action. Silence breeds procrastination. If a test is sitting in quarantine for weeks, nobody cares about it anymore. Lucas: Exactly. Visibility is power. The dashboard shows exactly how many tests are in quarantine, who owns them, and how long they've been there. It becomes a public scorecard. Nobody wants to be the person holding up the team with stale, broken tests. Luna: I think that social pressure is underrated. Engineering teams respond well to clear metrics and shared responsibility. Lucas: It works because it aligns incentives. When the pipeline is green, everyone wins. When it's red due to flakiness, everyone loses. Quarantine protects the green status for the majority, allowing the team to move forward while addressing the outliers. Luna: This feels like a mature approach to quality. It acknowledges that software is messy, but the process doesn't have to be. Lucas: It does. And it scales. As your codebase grows, manual testing becomes impossible. Automated testing becomes essential, but automation introduces its own complexity. Flakiness is the inevitable side effect of scaling automation without scaling discipline. Isolation and quarantine are the disciplines that keep the system honest. Luna: So, for listeners running smaller teams, is this overkill? Or can they adopt just the quarantine idea? Lucas: Start with quarantine. It is the lowest hanging fruit. You don't need a massive infrastructure overhaul to tag tests as flaky. You just need a simple script that counts consecutive failures. Once you see the numbers, the rest follows naturally. You will be shocked at how much noise you can filter out with a simple threshold. Luna: Simple thresholds can reveal huge problems. Sometimes seeing the scale of the mess is the first step to fixing it. Lucas: Absolutely. Data precedes solutions. You cannot manage what you do not measure. And measuring flakiness is easier than most people think. It is just a count of retries versus successes. Luna: Before we dive deeper into the technical implementation of those scripts, I want to pause for a second. If today's conversation gave you something usable for your own pipeline, the way these stays ad-free is listener support. You can buy me a coffee dot com slash fexingo — that's F. E. X. I. N. G. O. Lucas: Yeah, and really, it is that simple. Your support keeps the research sharp and the episodes consistent. Thank you for making that possible. Luna: Thanks for listening, and thanks for keeping the show going. Lucas: Let's get back to the mechanics. Once you have identified your flaky tests, the next step is the root cause analysis. The most common culprit is external dependency reliance. A test that calls a third-party API is inherently flaky because that API might throttle, timeout, or change response formats. Luna: So you mock everything? Even the parts that seem stable? Lucas: Especially the parts that seem stable. Stability is an illusion in distributed systems. The best practice is to mock all external calls. If you need to verify real behavior, use a staging environment that you control, not a live service you depend on. Control is the antidote to flakiness. Luna: That requires a lot of upfront work. Building mocks for every dependency sounds like a maintenance nightmare. Lucas: It is work, but it is sustainable work. Mocks are static. They do not change unless you update them. External APIs change constantly. Maintaining mocks is a one-time cost per version. Relying on live services is a recurring cost per run. The math favors mocks every time. Luna: I guess I had it backwards. I thought mocks were fragile, but they are actually the anchor. Lucas: They are. And another big source of flakiness is concurrency. Running too many tests in parallel can cause resource contention. CPU spikes, memory leaks, disk I/O bottlenecks. These are hardware issues masquerading as code issues. Luna: So scaling up your CI runners might actually make things worse if you hit a resource wall? Lucas: Exactly. More runners do not always mean faster results. They can mean more collisions. The sweet spot is finding the balance between parallelism and isolation. Too little parallelism slows you down. Too much creates chaos. Measure your resource usage, then optimize. Do not guess. Luna: This is getting very tactical. How do you convince a manager that investing in test infrastructure is worth the delay? Lucas: You speak their language: time. Every hour spent debugging a flaky test is an hour not spent building features. Show them the delta. Calculate the hours lost to retries last month. Multiply by the hourly wage. That is your savings number. It is hard to argue against free time. Luna: Free time is a powerful motivator. Managers love efficiency gains that show up on the bottom line. Lucas: They do. And once the pipeline is green, velocity increases organically. Developers trust the system again. Deployments become routine instead of risky. The cultural shift is as important as the technical one. Luna: Trust is the ultimate currency in engineering. When you lose it, everything slows down. When you rebuild it, momentum returns. Lucas: Momentum. That is the goal. Flaky tests kill momentum. Reliable tests sustain it. It is a choice between friction and flow. Luna: So, if you take nothing else away from this, just start counting your failures. See how many are real and how many are noise. The answer will surprise you. Lucas: It will. And once you see the noise, you will never want to listen to it again. The silence of a reliable pipeline is worth every bit of effort it takes to build.