Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Result Flakiness Adds Hidden CI Costs
Transcript
- Lucas: Luna, I want to run a number by you. A mid-sized engineering team of about forty developers runs, say, ten thousand automated tests in their CI pipeline. If just five percent of those tests are flaky — meaning they fail and pass without any code change — how much time do you think that team burns every month dealing with false failures? Luna: I'd guess it's more than most people think. Maybe a couple hundred hours across the team, between investigating failures, rerunning builds, and context-switching? Lucas: You're in the ballpark. A survey from a CI vendor last year actually tracked this — teams with a five percent flakiness rate reported an average of one hundred and twenty developer-hours per month just on false-positive triage. That's three full weeks of one person's time, every month, spent on tests that are lying to you. Luna: And that's just the direct time. What about the pipeline throughput hit? Every rerun delays feedback to the whole team. Lucas: Exactly. If each CI run takes fifteen minutes to fail a flaky test, and that test fails three times before passing, you've added forty-five minutes of wall-clock time to that merge. Multiply that across dozens of commits per day and your median CI time balloons by thirty to fifty percent. Some teams end up waiting three hours for a result that should take ninety minutes. Luna: And the trust erosion is real too. I've seen engineers start ignoring test failures entirely because so many are false alarms. That's how real regressions sneak into production. Lucas: Right. So let's talk about what teams actually do about it. The most common first step is quarantining. You flag a test as flaky after it fails on main with no code change, and you move it to a separate bucket that doesn't block the build. You still run it, but results are logged separately. Luna: But doesn't quarantining let bad tests just pile up? You need a way to fix or delete them. Lucas: That's the trap. Some teams quarantine aggressively and never circle back. The better approach is setting a time limit — say, two weeks to investigate and fix, or the test gets removed. A friend at a payments company does exactly that: every Monday, the team triages the quarantine list. They've cut their flakiness rate from eight percent to under two percent in six months. Luna: Two percent feels like a reasonable floor, but is there a cost to getting lower? I imagine certain integration tests are inherently nondeterministic. Lucas: That's the tension. You can spend a lot of engineering effort chasing the last few percentage points. The rule of thumb I hear most often is: keep flakiness below two percent of total test runs, but don't obsess over getting to zero. The ROI on going from five percent to two percent is huge — the survey I mentioned estimates it saves about two hundred thousand dollars a year in developer time for a forty-person team. Going from two percent to zero probably saves less than that but costs more in effort. Luna: So the real leverage is in the first few percentage points. That makes me think about how you even measure flakiness. Are teams tracking it as a percentage of test case executions, or of unique test cases? Lucas: Both, but the more useful metric is flaky test runs as a percentage of total test runs — because that's what impacts pipeline time. If one test is flaky and runs a hundred times a day, it matters more than a flaky test that runs once a week. So you weight by frequency. Some CI platforms now have built-in flakiness dashboards that compute this automatically. Luna: And that leads to the rerun strategy question. I've seen teams set a maximum rerun count — like try three times, then fail permanently. But doesn't that just mask the problem? Lucas: It can, if you don't log the rerun history. But a controlled rerun limit is actually smart. The key is that you never rerun more than, say, twice. And you always surface the rerun count in the failure report. That way, if a test consistently needs three attempts to pass, it gets flagged for investigation. Without a limit, you can have a test that reruns ten times, passes on the tenth, and everyone assumes it's fine. Luna: I've also seen teams use machine learning to predict flaky tests based on historical patterns — like tests that fail more often on Mondays or after a certain time of day. Is that practical for most teams? Lucas: It's becoming more accessible. Some test analytics platforms now flag tests with high flakiness probability. But the simpler version is just a weekly report: which tests failed and passed without a code change in the last seven days. That alone can reduce flakiness by forty percent, according to one case study I read. You don't need ML — you need discipline to look at the data and act. Luna: And part of that discipline is writing better tests in the first place. I think a lot of flakiness comes from shared mutable state — tests that depend on a database row that another test might delete. Lucas: That's the biggest category. Next is timing-dependent tests — waiting for an async operation with a fixed sleep instead of polling for a condition. And then there's environment coupling: tests that pass on a developer's laptop but fail in CI because of a different timezone or locale setting. All of these are fixable, but they require a mindset shift toward deterministic testing. Luna: You know, this conversation reminds me why I appreciate shows like this that dig into the practical details. A couple of dollars a month from listeners is genuinely what keeps these episodes ad-free and focused on real engineering trade-offs, not sponsored fluff. If you've gotten something useful out of today's conversation, buy me a coffee dot com slash fexingo — it makes a real difference to our ability to keep doing this. Lucas: Absolutely. And we mean it — small contributions add up and let us spend time researching topics like this rather than chasing advertisers. So thank you to everyone who has supported the show. Lucas: Alright, back to flaky tests. One more strategy I want to mention is test impact analysis combined with flakiness detection. Some teams only rerun flaky tests if their associated code paths have changed. That can cut rerun volume by sixty percent. Luna: That's smart — you're not rerunning a flaky test that's unrelated to the commit. But doesn't that require a pretty sophisticated mapping of tests to code? Lucas: It does, but the tooling is getting better. Some CI systems now offer dependency analysis that traces which source files a test touches. If the test is flaky but no dependency changed, you skip the rerun entirely. That saves time without sacrificing safety. Luna: So the net takeaway for me is: measure flakiness as a percentage of total test runs, set a target under two percent, quarantine aggressively with a two-week fix deadline, and use rerun limits with full logging. And invest in deterministic test design from the start. Lucas: That's a solid playbook. I'd add one more: treat flakiness as a bug, not a nuisance. When a test fails without a code change, that's a defect in your test suite. File a ticket, assign it, and track it like any other bug. The teams that do that see the biggest improvements. Luna: I like that framing. It elevates the priority. Flaky tests are technical debt with a direct cost per hour. Lucas: Exactly. And the data shows that the cost is real — not just a feeling. So next time your CI pipeline takes forever, check your flakiness rate first. It might be the cheapest fix you can make.