Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / The Hidden Cost of Flaky Tests in CI Pipelines
Transcript
- Lucas: There is a specific kind of dread that every engineering manager knows well. It happens at two-thirty in the afternoon on a Tuesday, when the build turns red, but rerunning it makes it green. That is not a bug in the code. That is a flaky test, and it is quietly draining millions of dollars in engineering salaries every year. Luna: Right, because you cannot trust the signal anymore. If the red light means nothing, developers stop looking at it immediately. Lucas: Exactly. And the cost isn't just the wasted compute cycles or the re-runs. It is the erosion of confidence. When I talk to CTOs about this, they often cite a figure around five to ten percent of their total test suite as being flaky. In a large organization with thousands of tests, that is hundreds of false alarms per week. Luna: I saw a report from a major cloud provider last month suggesting that up to fifteen percent of tests in complex microservices architectures can be nondeterministic under load. Lucas: That number feels conservative if we are talking about legacy monoliths moving to containers. The issue is rarely that the test logic is wrong. It is usually that the environment is unstable or the test shares state with another test that ran before it. Let us look at a recent case study from a fintech firm that tackled this head-on. Luna: I remember reading about that one. They had a payment processing service where the transaction logs would occasionally fail validation due to timestamp drift. Lucas: Precisely. They were mocking the system clock, which sounds like a standard practice, but they were doing it inconsistently across different test suites. Some tests used a fixed time, while others relied on the live system clock. When the network latency spiked during peak trading hours, those mocked timestamps desynchronized from the actual database records. Luna: So the test wasn't failing because the payment logic was broken, but because the fake time didn't match the real database record. Lucas: Correct. And the team initially tried to fix it by adding longer sleep statements in the test code to wait for synchronization. That is the classic mistake. You are essentially gambling that the network will be fast enough today. It works ninety-five percent of the time, which is enough to let the problem slide into production until it breaks completely. Luna: Sleeping in tests is basically asking for trouble. It makes the suite slower and still doesn't guarantee determinism. Lucas: It does both. The fintech company eventually solved this by implementing strict test isolation. They stopped sharing any mutable state between tests. Every test got its own isolated database schema, its own mock clock instance, and its own network namespace. It increased the setup time per test by two seconds, but it eliminated the flakiness entirely. Luna: Two seconds might sound small, but multiplied by thousands of tests running in parallel, that adds up to significant build time. Lucas: It does. But here is the trade-off: they reduced their overall pipeline duration by twenty percent because they stopped having to re-run failed builds. The upfront cost of isolation paid off in downstream efficiency. The key takeaway is that flakiness is usually a symptom of poor architecture, not poor testing. Luna: So the solution is not better testing tools, but cleaner code structure and stricter boundaries between components. Lucas: Exactly. If your unit tests depend on the order in which they run, you have an architectural problem. If your integration tests depend on external APIs that might be slow, you have an infrastructure problem. You have to identify the root cause before you try to patch it with retries. Luna: Retries are just masking the symptom. They make the build pass when it should have failed. Lucas: And that is dangerous. A retry hides a bug that could be catastrophic in production. The best practice now is to quarantine flaky tests. Do not let them block the main branch, but do not delete them either. Move them to a separate pipeline that runs nightly, so you can track their stability over time. Luna: That gives you visibility without letting bad code slip into the main release stream. Lucas: Right. It creates a feedback loop. You can see which tests are trending toward instability and prioritize fixing them. It is proactive rather than reactive. Now, before we dig deeper into the technical mechanics of isolation, I want to take a brief moment to talk about how this show operates. Luna: Sure, go ahead. Lucas: We produce these episodes daily without any advertising revenue. The only thing keeping the servers running and the research going is listener support through buy me a coffee dot com slash fexingo. It is a small monthly contribution that directly funds our ability to bring you these deep dives every single day. Luna: Yeah, and it really helps us maintain the quality and depth we aim for, so thank you to everyone who already contributes. Lucas: Appreciate it. Now, back to the isolation strategy. The fintech case study also highlighted the importance of deterministic mocks. When you mock an external dependency, you must ensure that the mock behaves exactly like the real service would under all conditions. Luna: Most teams just return static values. They forget that the real service might throw errors or delay responses. Lucas: Exactly. A good mock framework allows you to define failure modes and latency scenarios. You should be able to inject a fifty-millisecond delay or a five-hundred-error response into your test harness to verify that your application handles it gracefully. This moves you from happy-path testing to resilience testing. Luna: That changes the mindset from verifying that the code works to verifying that it survives when things go wrong. Lucas: Precisely. And it requires discipline. Developers hate writing tests for error cases because it is tedious. But those are the tests that prevent three-a.m. pager alerts. The question is how to balance speed with thoroughness when building out these robust test suites. Luna: You can't test everything, obviously. So you have to prioritize based on risk and complexity. Lucas: Risk is the right metric. Focus your high-fidelity, isolated tests on the core business logic. Your authentication flows, your payment calculations, your data transformation pipelines. These are non-negotiable. Peripheral features, like UI styling or logging formats, can get away with lighter, less isolated tests. Luna: It is about allocating engineering resources where they matter most. Don't spend days isolating a test for a button color change. Lucas: Spot on. And this ties back to the broader concept of test pyramid theory. We often hear people complain that the pyramid is collapsing, meaning too many tests are running at the integration level. Flaky tests are a major contributor to that collapse. Luna: Because integration tests are inherently more brittle. They depend on more moving parts. Lucas: They do. When an integration test fails due to flakiness, you have to spend time debugging the environment rather than the code. This slows down the entire development cycle. By pushing more logic into unit tests with strict isolation, you keep the integration layer clean and stable. Luna: So the goal is to minimize the surface area where flakiness can enter the system. Lucas: Yes. Limit the use of external services in your integration tests. Use containerized databases that spin up and tear down quickly. Avoid shared file systems. Every shared resource is a potential point of contention and nondeterminism. Luna: It sounds like a lot of overhead for the engineering team to manage. Lucas: It is overhead, but it is necessary overhead. Think of it as technical debt repayment. If you ignore flakiness, the debt compounds. Eventually, your CI pipeline becomes so slow and unreliable that developers stop trusting it entirely, and then you have no safety net at all. Luna: A broken safety net is worse than no safety net, because you think you are protected when you are actually exposed. Lucas: That is a crucial distinction. Trust is the currency of DevOps. Once you lose it, it is incredibly hard to regain. The organizations that succeed are the ones that treat test reliability as a first-class engineering concern, not an afterthought. Luna: They invest in the tooling and the culture to support that rigor. Lucas: Absolutely. It starts with leadership setting the expectation that a green build must mean something. If management accepts flaky tests as normal, the engineers will never fix them. It has to be a cultural mandate. Luna: So the technical solution is easy, but the cultural shift is the hard part. Lucas: The hard part. But once you make that shift, the velocity gains are immediate. Your builds become faster, your deployments become safer, and your engineers stop wasting time chasing ghosts. Luna: It turns the CI pipeline from a source of anxiety into a reliable tool. Lucas: Exactly. And that reliability allows teams to move faster with confidence. It is not about slowing down; it is about removing the friction that comes from distrust. Luna: I wonder if there are metrics companies use to track this progress beyond just the flake rate. Lucas: Lead time for changes is a good proxy. If your lead time drops while your test count increases, you know your tests are becoming more efficient and reliable. Another metric is the percentage of builds that require manual intervention. That should trend toward zero. Luna: Zero manual intervention. That is a high bar, but a worthy one. Lucas: It is. And it requires constant vigilance. As your codebase grows, new sources of flakiness will emerge. You have to stay ahead of them, not chase them. Luna: Proactive maintenance versus reactive firefighting. The difference between a mature engineering org and a chaotic one. Lucas: Precisely. It is one of those quiet disciplines that separates the scalable companies from the ones that burn out trying to scale. Luna: Something worth paying attention to as you grow. Lucas: It is. And for anyone listening who is currently drowning in red builds, start with isolation. Fix the shared state. Eliminate the sleeps. You might be surprised at how much faster things go. Luna: Simple steps that yield big results. Thanks for breaking that down, Lucas. Lucas: Anytime. The next step is always just one isolated test away.