Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Flakiness Metrics Predict Release Risk
Transcript
- Lucas: Luna, I want to talk about something that quietly eats engineering teams from the inside: flaky tests. Not the big catastrophic failures that make the pipeline go red, but the tests that pass nine times out of ten and then randomly fail on the tenth for no obvious reason. Luna: The ones you retry and they pass, so everyone just shrugs and moves on. I've definitely been on teams where that becomes the default response. Lucas: Exactly. And the standard advice is always 'just fix flaky tests' — as if it's that simple. But what gets less attention is how you can use flakiness as a leading indicator for release risk. If you measure it systematically, flakiness tells you something about the health of your entire test suite and, by extension, your deployment pipeline. Luna: So instead of just treating flaky tests as noise, you treat them as a signal. I like that. What does the signal look like in practice? Lucas: Let me give you a concrete example. I worked with a mid-sized e-commerce company last year — they process about two million orders a month — and their CI pipeline had around three thousand end to end tests. At any given time, roughly twelve percent of those were flaky. Not failing consistently, but flaky enough that engineers would retry two or three times to get a green build. Luna: Twelve percent sounds low, but with three thousand tests, that's over three hundred potentially unreliable signals. That's a lot of noise. Lucas: Right. And here's where it gets interesting. They started tracking three specific metrics. First, the flakiness rate — the percentage of test runs that produce a non-deterministic result. Second, the retry rate — how often a test is re-run after failing. And third, what they called the stability trend — a moving average of flakiness over the last two weeks. Luna: So the stability trend would capture whether things are getting better or worse over time, not just the snapshot. That makes sense. Lucas: Exactly. And once they had these numbers, they started correlating them with deployment frequency and post-release incidents. What they found was that when the flakiness rate exceeded eight percent, the probability of a hotfix within 48 hours of a release more than doubled. From about seven percent to over seventeen percent. Luna: That's a big jump. So flakiness wasn't just annoying — it was actually predictive of real problems in production. Lucas: That's the key insight. Flaky tests erode confidence. When engineers can't trust the test suite, they start overriding failures, they merge with yellow flags, they skip pre-release verification. The metrics become noise, and the safety net gets holes. Luna: And once that cultural shift happens — 'oh, it's probably just flaky' — it's hard to walk back. I've seen teams where every red build gets an automatic retry before anyone even looks at the failure. Lucas: That's exactly what was happening at this company. They had a Jenkins pipeline that would retry up to three times automatically. So the first failure basically got ignored. By the time they realized a test was genuinely broken, it had been failing for hours across multiple commits. Luna: So part of the fix was probably to reduce or remove auto-retry and make flakiness visible. Lucas: Yeah. They did two things. First, they reduced auto-retry to exactly one — so a test still gets a second chance, but if it fails twice, it's surfaced immediately. Second, they built a dashboard that showed the flakiness rate per team per week. Suddenly, the data was visible and it became part of the engineering review. Luna: And I imagine the teams with high flakiness had to justify why they weren't fixing those tests. That's a strong incentive. Lucas: Absolutely. Within three months, their overall flakiness rate dropped from twelve percent to about four percent. And the deployment frequency actually increased by about thirty percent, because teams trusted the pipeline more. Luna: That's a really nice outcome. So the metric itself became a driver of better behavior. Before we go deeper, Lucas — and I'm saying this because it's genuinely on my mind — this kind of practical, data-driven conversation is exactly why I value the work we do here. And I know a lot of listeners feel the same. Lucas: I appreciate that. And honestly, it's also why a small group of listeners choose to support the show directly through buy me a coffee dot com slash fexingo. That support is what keeps this ad-free and lets us spend time on stories like this one, where we can really dig into the numbers. Luna: It's not a big ask — just that if you find yourself using one of these frameworks or sharing a stat with a teammate, and you want to help keep the show going, that's the way. No pressure ever. Lucas: Exactly. Okay — back to flakiness. So the company I mentioned also started tracking what they called the 'flakiness debt.' Similar to technical debt, but specifically the cost of not fixing flaky tests — the time wasted on retries, the false alarms, the delayed releases. Luna: How did they quantify that? I imagine it's tricky to put a dollar figure on something like a false alarm. Lucas: They approximated by logging every instance where an engineer interrupted their workflow to investigate a test failure that turned out to be flaky. They measured the average time spent — about 12 minutes per incident — and multiplied by the frequency. Over a quarter, it came to roughly 110 engineering hours lost to flaky tests. That's almost three work weeks. Luna: Three weeks of engineering time just clicking 'retry' and staring at logs. That's real money. Lucas: Right. And when they presented that number to leadership, suddenly fixing flaky tests became a priority. They dedicated one sprint per quarter to what they called 'stability week' — no new features, just flakiness reduction. Luna: I bet that had a huge impact on morale too. Nothing drains a developer more than fighting a test that doesn't test anything. Lucas: For sure. But here's the next layer: they also started using flakiness metrics to decide whether to block a release. Their rule was: if the flakiness rate in the last 48 hours exceeded six percent, the release required manual sign-off from a senior engineer. That created accountability and forced teams to stabilize before pushing. Luna: So now flakiness isn't just a metric — it's a gating condition. That's a big cultural shift from 'oh, it's probably fine.' Lucas: Exactly. And it worked. Over six months, their post-release defect rate dropped by about forty percent. Not entirely because of flakiness, but because the overall quality culture improved. Luna: I think the key takeaway for our listeners is: don't just track flakiness to fix it — track it to understand the risk in your pipeline. Use it as a leading indicator. Lucas: That's exactly it. And start small. Just log the retry rate for your top ten flakiest tests. Watch the trend. I guarantee you'll start seeing patterns that tell you something about your release health. Luna: And if you do start tracking it, let us know what you find. We love hearing real-world examples. Lucas: Definitely. Alright, that's all for this episode. Until next time.