Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Non-Deterministic Tests Are Sabotaging Your CI Pipeline
Transcript
- Lucas: You push a commit, the CI pipeline runs, and one test fails. You look at the failure—completely unrelated to your change. So you hit 're-run failed tests,' and this time it passes. Sound familiar? Luna: Yeah, that's the classic non-deterministic test—aka the flaky test. It's the bane of every engineer's day. Lucas: Exactly. And we're not just talking about minor annoyances. At a mid-size SaaS company I spoke with recently, a single flaky test was causing a 45-minute CI delay on average. Multiply that by the number of engineers hitting that test per week, and they calculated it was costing about twelve thousand dollars a month in wasted engineering hours. Luna: That's real money. And the fix isn't just to re-run the test—that's treating the symptom, not the cause. Lucas: If today's conversation gives you one idea to cut that kind of waste, that's the whole point. These episodes stay ad-free because listeners find them useful, and the way to keep that going is a small gesture at buy me a coffee dot com slash fexingo. Luna: It's a tiny thing that makes a big difference. So where do these non-deterministic tests come from? Lucas: Race conditions are the number one culprit. Say you have a test that checks a database state after an async operation. If the operation hasn't completed before the assertion runs, the test fails. But sometimes it finishes in time, and it passes. So you get a non-deterministic result. Luna: Right. And the worst part is, these tests erode trust. When engineers see a CI failure, they start ignoring it, assuming it's flaky. Then a real bug slips through. Lucas: That's the real cost. So what do you do about it? The first step is to measure the flakiness rate. Track each test's pass rate over, say, the last 50 runs. Anything below 99.9% is a candidate for investigation. Luna: I've seen teams set a threshold and then just re-run any test that fails below it. But that's the same as ignoring it. Lucas: Right. The more effective approach is to quarantine flaky tests. Put them in a separate CI job that doesn't block the pipeline. Then assign someone to fix them before they rejoin the main suite. Luna: But quarantine only works if someone actually fixes them. I've seen flaky tests sit in quarantine for months. Lucas: True. That's why you need a dedicated 'flaky test rotation'—one engineer per sprint whose job is to fix the top three flakiest tests. It's a small investment that pays off fast. Luna: Let's talk about deterministic test ordering. If tests run in the same order every time, you can catch dependencies between them. Like one test leaves a record in the database, and another test relies on it being there. Lucas: Exactly. Random test order exposes hidden dependencies, but it also introduces non-determinism. A better approach is to run tests in a fixed order and then periodically randomize to detect new dependencies. Luna: So you get the benefit of both worlds. Lucas: Another big one: containerizing test dependencies. If your test talks to a third-party API, mock it or use a local container. Network latency and timeouts are a major source of flakiness. Luna: I've seen teams use Docker Compose to spin up a full test environment—database, cache, message queue—all locally. That eliminates a lot of variability. Lucas: But not all flakiness comes from infrastructure. Sometimes it's in the test logic itself. For example, a test that asserts on a timestamp or a random ID. Those are inherently non-deterministic. Luna: So you stamp the timestamp or accept a range. Don't assert on exact values. Lucas: Right. Another pattern: tests that rely on the current time. If you assert that a record's 'created_at' is 'now,' that's fragile. Use a fixed time or allow a delta. Luna: What about probabilistic approaches? I've heard of teams using machine learning to predict which tests are likely to fail. Lucas: That's an emerging practice. You train a model on test run data—pass/fail history, the commit that triggered it, the environment variables. It can flag tests with a high probability of flakiness before you even run them. Luna: That's smart, but it requires a lot of data. For most teams, the simpler stuff—quarantine, fixed ordering, containerization—will get you 90% of the way there. Lucas: Absolutely. And the key is to treat flaky tests as bugs, not annoyances. If a test is non-deterministic, it's not testing anything useful. Luna: It's actually worse than useless—it's misleading. It gives false confidence or false alarms. Lucas: So here's a concrete takeaway: Pick your three most-failed tests this sprint. Analyze why they're flaky. Fix them. If you can't fix them in a day, quarantine them and schedule a fix for the next sprint. Then measure the impact on your CI cycle time. Luna: I'd add one more: after you fix a flaky test, leave it in quarantine for a week. Monitor its pass rate. Only move it back to the main suite once it's proven stable. Lucas: Good call. That prevents the fix-revert cycle. And if you're using a CI tool that supports it, set up a flaky test dashboard. Track the trend over time. You'll see the problem shrink. Luna: I've seen a dashboard turn a team's attitude around. Suddenly flakiness isn't just noise—it's a metric they can improve. Lucas: And that's the goal. Non-deterministic tests don't have to be a fact of life. They're a solvable engineering problem. Luna: Next time you see that flaky test failure, think about the twelve thousand dollars. Then decide if hitting re-run is really the best use of your time. Lucas: Thanks for listening. We'll be back next week with another deep dive.