Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Flakiness Metrics Reveal CI Pipeline Health
Transcript
- Lucas: You know that test in your CI suite that fails maybe one in twenty runs, and you just hit 'Re-run' and move on? That test is lying to you — and it might be costing your team hours each week without anyone noticing. Luna: Oh, I know that test. It's always the same one — something about an order-total calculation that only fails between 2 and 3 AM. We've got a few of those. Lucas: Right. And the standard response is to track the percentage of flaky tests — if it's under, say, 5 percent, people feel fine. But that metric is dangerously misleading. I want to talk about why, and what actually measures the damage. Luna: Alright, I'm in. What's a better way to think about it? Lucas: Let me start with a case. A mid-size e-commerce company — let's call them ShopFast — had about twelve thousand tests. Their flaky test rate hovered around 3 percent. That sounds low, right? But those roughly 360 flaky tests were failing, on average, once every six runs. Each failure triggered a manual rerun, and each rerun cost about four minutes of developer time to triage, re-kick, and wait. Luna: So 360 tests failing one out of six runs means about 60 failures per full CI cycle. At four minutes each, that's four hours of developer time just for reruns per cycle. Lucas: Exactly. And they ran CI about ten times a day. So they were burning 40 minutes per developer per day on reruns. That's not sustainable, and the 3 percent metric hid it completely. The fix was to introduce what I call 'flaky test weight' — instead of counting unique flaky tests, you measure total failures per execution and the average rerun cost. Luna: So you're weighting by frequency and cost. That makes sense — one test that fails every other run is way more expensive than ten tests that fail once a month. Lucas: Precisely. ShopFast's flaky test weight was about 0.8 failures per execution. After they prioritized fixing the top 20 tests by weight, that dropped to 0.2, and developer time wasted on reruns fell by 75 percent. The key insight is that flaky tests aren't just a quality issue — they're a productivity tax on every developer in the pipeline. Luna: Our team at the fintech startup had a similar wake-up call. We tracked flaky test count, but our CI was still slow. Turned out we had a few tests that were flaky due to shared state — they'd pass in isolation but fail when run in a specific order. We reduced flakiness by 70 percent by introducing test splitting and deterministic seed values for random number generators. Lucas: That's a great point. Deterministic seeds are one of the most underused tools. If you set a seed for any test that uses randomness, you can reproduce failures reliably. The test may still fail, but at least it fails consistently — which means it's no longer flaky, it's just a regular bug. Luna: Right. And once it's reproducible, you can fix it. So beyond measuring weight, what's the next step? Do you set a formal service-level objective for flakiness? Lucas: Absolutely. I'd argue every team should have an SLO for flaky test weight, not just flaky test count. For example, target a weight of less than 0.1 failures per execution. If you exceed that, the CI pipeline should automatically quarantine flaky tests — move them to a separate run that doesn't block the main build. That way, developers aren't constantly derailed. Luna: But doesn't quarantining let teams ignore the problem? I've seen teams just let tests rot in quarantine forever. Lucas: That's a risk, yes. So you need a secondary SLO: a maximum number of quarantined tests, say 50, and a time limit — if a test stays quarantined for more than two weeks, it gets escalated to the team that owns it. The goal isn't to hide flaky tests, it's to stop them from blocking everyone while giving the owner room to fix them properly. Luna: That makes sense. Another thing we learned is that flakiness often points to a deeper infrastructure issue. In our case, the shared state was actually caused by a test database that wasn't being reset properly. We fixed the database reset, and a bunch of tests stopped being flaky overnight. Lucas: That's a classic root cause. Flaky tests are frequently symptoms of non-determinism in the test environment — things like time-dependent behavior, network latency, or data races. A lot of teams treat each flaky test individually, but the real fix is often environmental. Luna: So if you see a sudden spike in flaky test weight, you should look at recent infrastructure changes before blaming the test itself? Lucas: Exactly. A common pattern is a deployment that changes a timeout or adds a new shared resource. Suddenly, tests that were passing 99 percent of the time start failing 10 percent of the time. The test didn't change — the environment did. So your flaky test metrics should be paired with change-log correlation. If a flaky spike correlates with a deployment, you've got your culprit. Luna: That's a good practice. At my current company, we actually graph flaky test weight against deployment times. It's been eye-opening — we found one service that introduced flakiness every time it was deployed because of a race condition in its API response time. Lucas: And that's the kind of insight you'd never get from just counting flaky tests. So to summarize the framework: measure flaky test weight, set an SLO, automate quarantine with an expiration policy, and correlate flakiness with deployments. That'll surface the real costs and root causes. Luna: I want to zoom in on the automation part. How do you actually quarantine a test automatically? Do you need a custom tool? Lucas: Not necessarily. Most CI systems have a concept of 'test retry' — you can configure a test to be retried on failure. But that's the opposite of quarantine. What I recommend is a script that runs after the test suite completes, parses the results, and if a test fails more than a threshold in the last N runs — say, twice in the last ten — it moves that test to a separate suite file. Then the pipeline excludes that file from the main build. Luna: So it's a simple file-based approach. You'd need a history store, like a database or even a file with per-test failure counts. Lucas: Right. The history can be stored in a simple JSON file in your build artifact. It's not sophisticated, but it works. Over time, you can build a dashboard. The important thing is to start. Even with a manual process of reviewing flaky tests weekly, you'll see improvement. But automation scales. Luna: One thing I've wondered: should you ever delete a flaky test? Or always fix it? Lucas: That's a great question. If a test has been flaky for months and no one can figure out why, and it exercises a low-risk code path, sometimes it's better to delete it. Flaky tests erode trust in the entire suite. Developers start ignoring test failures because 'oh, that one's always flaky.' That's dangerous. So I'd rather have fewer, reliable tests than many flaky ones. Luna: I've seen that happen — teams get desensitized. They see a red build and don't even check. That's when bugs slip into production. Lucas: Exactly. So treating flakiness seriously is about preserving trust in your CI signal. If the build is green, you should be able to deploy with confidence. If it's red, you should know it's a real problem. Flaky tests blur that line. Luna: Totally agree. And if you're listening and thinking 'yeah, we have that one test,' the first step is just to measure the flaky test weight. You don't need a fancy tool. Lucas: Right. And if today's conversation gave you something useful — a new metric, a quarantine idea, a reason to look at your infrastructure — the way we keep these episodes free and ad-free is through listener support at buy me a coffee dot com slash fexingo. It's a small gesture that lets us keep digging into these topics without any sponsors. Luna: Yeah, and it genuinely helps. We've had listeners say they implemented flaky test weight after a previous episode, and it saved them hours. That's what keeps us going. Lucas: So back to the practical side — let's talk about one more technique that complements this: using test impact analysis to skip tests that aren't affected by a change. If you combine flaky test weight with test impact analysis, you can dramatically reduce CI time and noise. Luna: That's a good next step. But for today, the takeaway is: stop counting flaky tests, start measuring their weight. Set an SLO, quarantine automatically, and look at your infrastructure when flakiness spikes. Lucas: Perfect. Thanks, Luna. And thanks to everyone listening. We'll be back next week with another angle on making your test suite actually reliable.