Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Flakiness Metrics Reveal CI Pipeline Health
Transcript
- Lucas: We hit episode 100 today, and I wanted to talk about something that quietly eats developer time across the industry — nondeterministic tests. Tests that pass when you run them locally but fail on the CI server, or pass on one node but fail on another, for no obvious reason. Luna: Flaky tests. Every team I've worked with has at least a few. And the typical response is just to rerun them and hope they pass. Lucas: Right. And that costs real money. A 2023 study from Google found that across their monorepo, roughly 16 percent of test failures were flaky. Not real bugs — just noise. Engineers spend an estimated 10 to 20 percent of their time investigating false failures or rerunning jobs. Luna: That's huge. For a team of fifty engineers, you're losing something like five to ten full-time salaries to nothing productive. Lucas: Exactly. And the problem gets worse as your test suite grows. Facebook — now Meta — published data showing that their test suite had about 17 million tests running daily. Even a flakiness rate of one percent means 170 thousand unreliable results every day. Luna: So how do you even start measuring flakiness systematically? I feel like most teams don't have a dashboard for it. Lucas: Most don't. But the first step is simple: track rerun rates. Every time CI reruns a test automatically after a failure, that's a signal. If a test fails on the first run but passes on the second, flag it. After a few occurrences, you know it's flaky. Luna: Rerun budgets. Some teams set a threshold — if a test fails more than, say, two percent of the time, it gets quarantined. Lucas: Yes. And quarantine is the key. Instead of blocking the build, you move the flaky test to a separate bucket. It still runs, but its failure doesn't fail the pipeline. You track it separately and fix it when you have time. Luna: But doesn't that risk ignoring real bugs? If a test is flaky but also sometimes catches regressions, you might miss something. Lucas: That's a valid concern. The way to mitigate it is to set a clear time limit — if a quarantined test isn't fixed within two weeks, it gets removed from the suite entirely. The cost of maintaining a flaky test often outweighs its benefit. Luna: I've seen teams use machine learning to classify flaky tests automatically. They train a model on historical runs — timing, dependencies, environment variables — to predict which tests are likely to be flaky. Lucas: That's getting more common. At Spotify, they built a tool called Flake Finder that uses random forest classifiers. It identifies patterns like a test that only fails when run after a certain other test, or when the database is under load. Luna: Right. Because a lot of flakiness comes from test order dependencies or shared mutable state. Tests that mutate global data and don't clean up after themselves. Lucas: Exactly. And that's where hermetic testing helps. If every test starts with a clean environment — fresh database, no shared files, isolated services — you eliminate a huge source of nondeterminism. Luna: But hermetic tests are expensive to build. You need to mock or containerize every dependency. Lucas: True. So a pragmatic approach is to target the flakiest tests first. Use your rerun data to identify the top ten offenders, fix those, then move on. The return on investment is immediate because those ten tests were probably wasting the most time. Luna: I think a common mistake is treating flakiness as a testing problem rather than an infrastructure problem. Sometimes the CI environment itself is unstable — network timeouts, disk space issues, CPU contention. Lucas: Absolutely. I've seen cases where a test always failed on a particular build agent because it had a different version of a library installed. That's configuration drift, not a test bug. Luna: So you need to monitor your CI infrastructure alongside your test suite. If a test only fails on one machine, don't rewrite the test — fix the machine. Lucas: Right. And that's why having a unified view of test history is so important. Tools like TestRail or even a custom dashboard that shows pass rates over time per test, per node, per commit. You can spot trends. Luna: There's also the human side. If developers don't trust the test suite, they start ignoring failures. That's dangerous. Lucas: That erosion of trust is maybe the worst outcome. Once people start merging code with red tests, quality drops fast. So investing in flakiness reduction isn't just about efficiency — it's about maintaining a culture of quality. Luna: If today's conversation gave you something useful — maybe a concrete idea for your own CI pipeline — that's exactly why we make this show. And we keep it ad-free because of listener support. If you'd like to help us keep going, you can find us at buy me a coffee dot com slash fexingo. Lucas: We really appreciate everyone who contributes. It lets us focus on the topics that matter to you. Now, one more thing about flaky tests — another approach I've seen work well is using a 'flakiness score' for each test, based on weighted recency. A test that flaked yesterday is more concerning than one that flaked six months ago. Luna: That makes sense. You could even surface that score in your pull request workflow, so developers see a warning before merging code that introduces a flaky test. Lucas: Exactly. And some teams tie it to performance reviews or engineering metrics — not in a punitive way, but to highlight that reducing flakiness is valued work. Luna: I've also seen companies run 'flakiness sprints' where the whole team focuses on fixing flaky tests for a week. It can dramatically improve CI reliability. Lucas: That's a great way to make it a shared responsibility rather than something the QA team owns alone. Because flakiness is everyone's problem — developers, testers, infrastructure engineers. Luna: And it's one of those problems that, once you start measuring it, you can't unsee it. The data will tell you exactly where your pipeline is bleeding time.