Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Your CI Pipeline for Flaky Test Fatigue
Transcript
- Lucas: So there's this moment that every engineer knows. You push your code, the CI pipeline kicks off, and you watch the little yellow spinner. Then, five minutes later, a red X. You click 'rerun failed tests'. Green. You merge, never think about it again. That's the exact behavior that should terrify you. Luna: Rerunning until green is basically an industry ritual at this point. But I'm guessing today's episode is about why that ritual is so dangerous. Lucas: Exactly. And it's not just about the occasional flaky test. It's about what happens when flakiness becomes a systemic part of your pipeline. I've been looking at this software consultancy that works with a mid-sized payments company — I can't name them, but think of the kind of fintech that processes a few million transactions a day. Their QA lead shared some numbers with me. Eighteen months ago, their CI suite had about 40 tests. Now it's over 800. They were proud of that growth. But in the last quarter, they hit a point where roughly 12 percent of their test runs failed for non-deterministic reasons. Not because the code was broken, but because the tests themselves were flaky. Luna: Twelve percent. That's huge. I mean, that's one out of every eight runs failing for no good reason. Developers must have been losing their minds. Lucas: The QA lead told me the real cost wasn't just the wasted compute. It was the trust. Developers started automatically rerunning failed tests before even looking at the logs. They called it 'playing the lottery'. Eventually, some of them would just merge if the code review looked fine, regardless of the test result. And that's the classic flaky test fatigue spiral. Luna: So how did they break out of it? Or are they still stuck in the spiral? Lucas: They're actually pulling out of it now, and the way they did it is a great case study. But before I get into their fix, I want to talk about the core problem. Flaky tests aren't just a nuisance. They're a leading indicator of deeper issues in your codebase and your testing strategy. When you have a test that passes on the rerun but failed the first time, it's evidence that either your test has a race condition, your environment is unstable, or your code has a subtle bug that only surfaces under certain timing or load. And when you just rerun it and move on, you're burying that evidence. Luna: Right. So the rerun isn't just a productivity hack, it's actively hiding information from you. Lucas: Exactly. And that's what they had to confront. They had a team of about thirty engineers, and they were spending an estimated 15 to 20 minutes per developer per day just dealing with flaky test reruns and the occasional investigation. That adds up to something like 250 hours a month across the team. That's more than a full-time engineer's worth of time, every month, just on reruns. That was the number that got the engineering director's attention. Luna: Two hundred fifty hours a month. That's not just a quality problem, that's a business cost. And that's the kind of number that makes a director stand up. Lucas: So what did they do? They made a few changes, and the first one was to stop treating every test failure as a binary thing. They created a classification system. When a test fails, the CI system automatically tags it as either 'suspected flaky' or 'real failure' based on a few heuristics. For example, if the failure happens in a test that has a history of flakiness, it gets tagged. If the failure is in a test that's never failed before, it's treated as a real failure and blocks the merge. Luna: So they're using the test's own history to inform the response. That makes sense, but what about the first time a test flakes? You don't have history yet. Lucas: That's a good point. They had to be conservative there. For the first few runs, every failure is treated as real. That's the safe default. But once a test has failed, say, three times out of the last ten runs, it gets flagged as a high-risk flaky candidate. And here's where the interesting part comes in: they built a 'flaky bot' that automatically reruns the test on the same commit, but with a few variations. It runs it three times in a row, and if it passes all three times, it marks the test as 'flaky confirmed' and quarantines it. That means the test is no longer part of the required CI gate, but it still runs in a separate job that's not blocking. Luna: Quarantine. I love that term. It's like the test is in time-out until it can prove it can behave. Lucas: Exactly. And the quarantine isn't a punishment, it's a triage. The team then has to fix the test before it can rejoin the suite. They have a quarterly 'flaky test day' where each engineer is assigned a few quarantined tests to fix. They found that most flaky tests can be fixed by making the test more deterministic, like adding explicit waits instead of sleep calls, or mocking external services that have unpredictable response times. Luna: So they're not just covering up the problem, they're actually taking time to fix the root causes. That's a culture shift. What did that do to their numbers? Lucas: After about six months, their flaky rate dropped from 12 percent to under 3 percent. And the rerun rate fell dramatically. Engineers started trusting the pipeline again, and the 'just merge' behavior disappeared. But they also discovered something interesting: a lot of the flakiness was concentrated in a few tests. The Pareto principle held — about 80 percent of the flaky failures came from 20 percent of the flaky tests. So they focused their fix efforts on those high-impact tests first. Luna: That's a smart way to prioritize. And it reminds me of something you said earlier about the cost. You mentioned 250 hours a month. Did they actually see that time come back? Lucas: They did. After the changes, they estimated the time spent on reruns and flaky test investigations dropped to about 40 hours a month. That's a savings of over 200 hours a month. That's the equivalent of hiring an extra engineer and a half, just by making tests reliable. And that's the kind of return on investment that gets engineering leaders excited. She also told me that the codebase had a few hidden bugs that were actually being masked by the flakiness. Once they fixed the tests, those bugs surfaced and were fixed quickly. Luna: So by chasing flaky tests, they actually improved their product's reliability. That's a double win. But I'm wondering — what about the emotional side? You mentioned trust earlier. How do you rebuild trust in a pipeline that's been crying wolf for months? Lucas: That's a great question, and it's probably the hardest part. The engineering manager told me that they had to treat it like a public relations campaign inside the company. They started publishing a weekly 'flakiness report' that showed the top flaky tests, who owned them, and whether they were quarantined or fixed. They also made it a rule that any engineer could challenge a test failure and ask for a 'flakiness review' if they thought it was flaky. That gave engineers a sense of control. But the real turning point was when the CI pipeline had a streak of two weeks with zero flaky failures. That was the first time in months. And that's when people started to believe it was real. Luna: Two weeks without a flaky failure must have felt like a miracle. I can imagine the team celebrating that. But I'd love to dig into the technical side a bit more. What are the most common causes they found for flakiness in their tests? Lucas: The number one cause was time. Tests that depended on timing, like waiting for a response from a service, used hard-coded sleep calls. Those are inherently flaky because they assume a certain response time that can vary. They replaced many of those with explicit waits that poll for a condition with a timeout. That alone fixed about a third of the flaky tests. Another big one was test data. Tests that used shared databases would sometimes collide with other tests that were running in parallel. They solved that by isolating each test's data using unique namespaces or in-memory databases where possible. Luna: So test data isolation is a big one. Makes sense. And I bet there was also some environmental flakiness, like network calls to external services. Lucas: Absolutely. They had a few tests that hit a real credit card processor's sandbox environment, and that sandbox would occasionally have latency spikes or even return errors. They mocked those calls entirely. The lesson is that external dependencies in tests should be mocked unless you're explicitly testing the integration. They also found that some tests were creating random data using the current time or a random seed, which would occasionally produce data that violated a uniqueness constraint. They fixed that by using deterministic random generators where you can seed the value. Luna: So it's a mix of test design and environment control. But I'm thinking about the bigger picture: how do you measure flakiness over time? They must have had some metric to track their progress. Lucas: They used something called a 'flakiness score' for each test. It's a weighted average of recent failures, where a failure ten runs ago counts less than a failure in the last run. They thought about using a simple percentage, but that's overly sensitive to a spike in the last few runs. The weighted score smooths that out. They set a threshold: if a test's score exceeds 0.1, meaning it fails more than 10 percent of its runs, it automatically gets flagged for review. And they track the median score across the entire suite. That's their 'suite flakiness index'. It went from 0.08 down to 0.02 over six months. Luna: A suite flakiness index. That's a nice, concrete metric that the whole team can rally around. It's like the health score of your test suite. Lucas: Exactly. And here's the thing: they didn't just track it for tracking's sake. They made it a part of their definition of done for new features. Any new test has to have a flakiness score below that threshold for two consecutive weeks before it's considered stable. That prevents new flaky tests from sneaking in and undoing all their progress. Luna: That's a great gate. So they've basically built a system that doesn't just react to flakiness, it prevents it from coming back. And I think that's a lesson for any team that's drowning in flaky tests. It's not just about fixing the ones you have; it's about changing the culture so that new ones don't appear. Lucas: Right. And I think that's the deeper point. Flaky tests are a symptom of a team that's moving too fast and not paying attention to test reliability. The fix isn't just technical, it's cultural. It's about making reliability a first-class citizen in your CI pipeline. And that's what this team did. They turned a frustrating problem into a measurable, manageable part of their engineering process. Luna: So for teams listening who are in the thick of flaky test fatigue, what's the first step they should take? Besides maybe banning the 'rerun' button for a week. Lucas: Banning the rerun button is actually not a bad idea. But the first concrete step is to start measuring. You can't fix what you can't see. Set up a simple script that parses your CI logs and counts how many failures are followed by a subsequent pass on the same commit. That's your raw flaky rate. Just knowing that number can be eye-opening. Then start tagging tests with a 'flaky' label when you confirm a rerun. That's the beginning of a system. Luna: And I'd add that you should also look at which tests are most often rerun. That'll give you a quick list of the usual suspects. Lucas: Exactly. The data is right there in your CI system. You just have to mine it. And remember, the cost of flaky tests isn't just the time you spend rerunning. It's the bugs that slip through when everyone stops trusting the pipeline. That's the real price you pay. Luna: It's fascinating how something as mundane as a flaky test can have such a ripple effect across a whole engineering org. And I love that they turned it into a measurable, managed process. I think that's a story worth sharing with any team that's struggling. Lucas: Absolutely. And honestly, if today's conversation gave you something useful, something you can apply to your own team, that's what we're here for. If you think it was worth a coffee, the link is buy me a coffee dot com slash fexingo. Every little bit helps us keep the show independent and ad-free. Luna: Yeah, we're a small team, and listener support genuinely makes a difference. But no pressure. Now, back to the technical side — one thing I want to dig into is how they handled the quarantine process. Did they have a specific SLA for how long a test could stay quarantined before it had to be fixed or removed? Lucas: Great question. They set a 30-day SLA. If a quarantined test isn't fixed within 30 days, it's automatically removed from the suite. That sounds harsh, but it prevents a graveyard of 'temporarily disabled' tests that never come back. The test owner gets a reminder at 20 days and again at 25. They found that having a deadline actually motivates people to fix it rather than letting it linger. Luna: A 30-day SLA makes sense. It keeps the quarantine honest. It's not a permanent limbo, it's a holding cell with an execution date. Lucas: Exactly. And they also made sure that the quarantine process is transparent. Any engineer can see which tests are quarantined and who owns them. That creates a bit of healthy peer pressure, but also a sense of shared responsibility. The team knows that the health of the suite is everyone's job, not just the QA team's. Luna: That's a great way to end it. It really comes down to treating test reliability as a team sport. Thanks for unpacking this with me, Lucas. Lucas: Anytime, Luna. And to our listeners, if you've got a flaky test story of your own, we'd love to hear it. Keep your pipelines green, and we'll catch you next time.