Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Regression Testing Eats Your Sprint Velocity
Transcript
- Lucas: You know that moment when your team ships a feature, it passes all the tests, and then two weeks later something completely unrelated breaks in production? Luna: Oh, the classic 'we didn't touch that module' surprise. Happens at least once a quarter on most teams I've worked with. Lucas: Right. And usually, the knee-jerk reaction is 'we need more tests.' But here's the thing — more tests might actually be making the problem worse. I want to talk about regression test bloat, and one specific case I came across recently. Luna: Before you get into it — I think this is a good moment to mention that episodes like this one stay ad-free because of listeners who support the show at buy me a coffee dot com slash fexingo. If today's conversation gives you something usable, that's the place to keep it going. Luna: How unchecked are we talking? Lucas: From 400 tests to 7,200 tests in 18 months. Each new feature introduced, on average, 12 to 18 new regression tests. And nobody ever went back to ask: 'Is this test still pulling its weight?' Luna: So the suite became this giant safety net that was actually slowing everyone down. I've seen that — the full CI pipeline went from 20 minutes to over two hours. Lucas: In this case, 11 hours. The full regression run took 11 hours. So developers would kick off a build in the morning, and if it failed, they wouldn't know until the end of the day. That feedback loop destroys velocity. Luna: And the worst part is, most of those tests probably pass every time. They're low-signal. They take up time but never catch anything. Lucas: Exactly. I asked the QA lead to run an analysis. Out of those 7,200 tests, only 3 percent had ever failed in the last six months. Ninety-seven percent of the suite was green every single run. That's not a safety net — that's a tax. Luna: A tax on every commit. So what did they do? Lucas: They started pruning. But not blindly. They categorised each test by risk — what's the impact if this test misses a bug? A login-page test that checks button color is low risk. A payment-processing test that verifies transaction amounts is high risk. Luna: So they applied a risk-based test selection strategy. Keep the high-risk tests in the critical path, demote or delete the low-risk ones that never fail. Lucas: Right. They cut the suite from 7,200 to about 2,800. The CI pipeline dropped from 11 hours to 45 minutes. And in the next quarter, their sprint velocity went up by 25 percent. Luna: But here's the pushback I always get: 'What if we lose coverage and a bug slips through?' Lucas: That's the fear. But coverage is not the same as value. A test that never fails and checks something trivial — that's not giving you coverage, it's giving you noise. And noise has a real cost. Luna: I think the other issue is flaky tests. If a test fails randomly 10 percent of the time, developers start ignoring failures. They assume it's flaky, not a real bug. Lucas: That's a huge point. In that same team, they had about 200 flaky tests. Those were undermining trust in the entire suite. So they quarantined them — moved them to a separate pipeline that runs overnight, not blocking the main CI. Luna: Quarantine is a good tactic. But you have to fix or delete them eventually, not just hide them. Lucas: Absolutely. They set a 30-day SLA: either the flaky test gets fixed, or it gets deleted. That forced the team to either invest in making the test reliable or admit it wasn't worth keeping. Luna: So what's the one metric you'd track to prevent this kind of bloat from happening again? Lucas: I'd track 'regression test execution time per developer commit.' If that number starts climbing, you know your suite is growing faster than your ability to maintain it. Set a threshold — say, 30 minutes max. If it goes over, you have to prune before you can add new tests. Luna: That's a hard rule. I like it. It forces the team to treat test maintenance as a first-class engineering task, not an afterthought. Lucas: Exactly. And that's the mindset shift. Tests are code. They have maintenance cost. They have to be refactored, pruned, and sometimes deleted. A test suite that never shrinks is a test suite that's slowly killing your velocity. Luna: So the takeaway: don't just add tests. Manage them. Prune aggressively. And keep that CI gate fast. Lucas: Yeah. A 30-minute regression gate that catches real bugs is infinitely better than an 11-hour gate that catches nothing new. That's the trade-off worth making.