Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Retry Strategies Hide Real Defects
Transcript
- Lucas: So you're scrolling through your CI dashboard, you see a test failed, so you hit 're-run' — and it passes. Feels like a win, right? No bug to fix, just a random glitch. Luna: Yeah, I've definitely done that. And honestly, sometimes it's the right call — network timeouts, container startup races, those are real. Lucas: Right. But here's the problem: when your pipeline automatically retries every failure, you stop noticing which failures are real. And that can hide defects for weeks or months. Luna: Before we get too deep — quick thing. Shows like this one only exist because listeners chip in a few bucks a month on Buy Me a Coffee. It's buy me a coffee dot com slash fexingo. Keeps it ad-free and independent. Lucas: Yeah, genuinely. A couple of dollars from people who get something out of these episodes is what keeps us going. So if that's you, we appreciate it. Luna: Alright — back to retries. You mentioned hiding defects. Got a specific example? Lucas: Yeah. There's a fintech company — I'll keep them anonymous — but their CI pipeline retried failed tests up to three times before reporting a final result. They had something like a 12 percent retry rate overall, which they just accepted as 'flaky infrastructure.' Luna: Twelve percent? That means one in eight test runs was failing on the first attempt. That's huge. Lucas: Exactly. And it was masking a race condition in their payment-ordering service. About one in every two thousand transactions would process in the wrong sequence. Their integration test caught it maybe thirty percent of the time on first run, but on retry the condition didn't trigger because the timing was slightly different. Luna: So the test would fail, then pass on retry, and the team would mark it as flaky and move on. Meanwhile the bug is live in production. Lucas: Exactly. They eventually traced a two-million-dollar incident — duplicate payments during a high-volume day — back to that exact race condition. And their CI had been showing the symptom for six weeks. Luna: So how do you distinguish between a real transient failure and a defect that just doesn't always reproduce? Lucas: First thing: stop auto-retrying everything. Or at least track retry rates per test as a first-class metric. If a test passes on retry more than, say, five percent of the time, it goes into quarantine. Luna: Quarantine meaning you remove it from the critical path and investigate separately? Lucas: Right. You don't block the build with it, but you also don't ignore it. You run it in a separate job with full logging, maybe even record the test's thread interleaving. The key is to treat a pass-on-retry as a yellow flag, not a green light. Luna: What about tests that are genuinely flaky due to environment issues — like a shared database that gets into a bad state? Lucas: That's a separate category, and you should fix the environment, not patch the test. But the dangerous pattern is when a test fails, retries, and passes, and nobody asks why it failed in the first place. The failure itself is a signal. Luna: I've seen teams set up retry budgets — like 'we allow two retries per test suite per day.' Above that, the pipeline fails and someone has to manually investigate. Lucas: That's a good practice. Another one is deterministic test ordering. If tests run in a fixed order, you can reproduce failures more reliably. Random order plus retries is a recipe for heisenbugs. Luna: Heisenbugs — bugs that disappear when you try to observe them. Fitting name. Lucas: Right. So let's talk about what a healthy retry rate looks like. If you're below one percent, you're probably fine. Between one and three percent, you should start auditing. Above five percent, you have a systemic problem. Luna: And that audit — what are you looking for? The distribution of failures across tests, or across time? Lucas: Both. You want to see if a small number of tests account for most retries — those are candidates for quarantine. And you want to see if retries spike during certain times — like when a new deployment goes out — which hints at a regression, not flakiness. Luna: So the retry rate becomes a leading indicator for deployment health. Lucas: Exactly. One team I worked with plotted retry rate per deployment. Every time it jumped above two percent, they'd roll back or investigate. After a few months, their retry rate dropped below half a percent, and their production incident rate dropped too. Luna: That's a nice feedback loop. What about tools — any you'd recommend for tracking this? Lucas: Most CI platforms expose the data. Jenkins, GitLab CI, GitHub Actions — you can extract test results and compute retry rates with a simple script. There are also open-source tools like Flaky Test Tracker or Test Analytics that visualise it. Luna: But the hard part isn't the tooling, it's the culture. Getting the team to treat a pass-on-retry as a defect, not a relief. Lucas: Yeah, that's the real challenge. It requires admitting that your tests aren't perfect, and that 'green build' doesn't mean 'no bugs.' But once you start tracking it, the data is pretty convincing. Luna: Alright, so practical takeaway: audit your retry rate. Quarantine tests that pass on retry more than five percent of the time. Fix the environment, not the test. And don't let flaky tests lull you into false confidence. Lucas: Exactly. Because the test that passes on retry might be the one that catches the next two-million-dollar incident — if you let it fail. Luna: Good note to end on. Thanks, Lucas.