Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Risk-Based Testing Cuts Regression Suites by 60 Percent
Transcript
- Lucas: So a couple years ago, a financial trading platform I worked with had a regression suite of nearly fifteen thousand test cases. It ran for over twelve hours, slowed releases to a crawl, and somehow still let a critical pricing bug slip into production. Luna: That's the worst — a big suite that gives you false confidence. What did they do? Lucas: They switched to risk-based testing. Instead of running everything, they ranked each test by two factors: the business impact if the feature failed, and the technical probability that it would fail. Low-impact, low-probability tests got deprioritized or dropped. High-risk areas got maximum coverage. Luna: So you're basically saying, not all tests are created equal. But how do you measure 'business impact' in a way that's objective? Lucas: Great question. They used a simple three-by-three matrix — categories like 'critical,' 'high,' and 'low' for both impact and probability. A compliance feature that could trigger a regulatory fine? That's critical impact. A cosmetic font change on an internal dashboard? Low impact. Pair that with the team's historical defect data — modules that always break — and you get a score. Luna: And the result was cutting the suite by sixty percent? That seems drastic. Did they miss anything? Lucas: They did. First time, they accidentally dropped a test for a rarely used trade reconciliation feature. But because they'd built in a quick review cycle, they caught it before release. After refinement, they actually found more critical defects because the high-risk tests got deeper attention. Luna: That's the key — it's not just about removing tests, it's about rethinking your focus. How do you get product managers and business stakeholders to buy into ranking their own features' risk? Lucas: Honestly, it's not hard once you frame it in their terms. You ask: 'If this feature fails in production, what's the cost — revenue, reputation, legal?' Suddenly they're very motivated to assign an impact level. And developers already know which modules are flaky, so the probability side comes from them. Luna: It's really an exercise in curation. And that's something we appreciate around here — having someone curate what's worth your time. Lucas: Exactly. And if you find value in that kind of focused conversation, listener support is what keeps this show ad-free. People can contribute at buy me a coffee dot com slash fexingo — it genuinely helps us keep doing episodes like this. Luna: Yeah, it's a small way to say this kind of practical content matters. Now, back to risk-based testing — what tools actually support this approach? Lucas: Some teams start with a simple spreadsheet — test ID, feature, impact score, probability score, and a composite risk value. But dedicated tools like TestRail or qTest have built-in risk modules. For that trading platform, they built a custom plugin for Jenkins that recalculated the risk score after each deployment. Luna: Do you think AI could automate the whole risk assignment? Like, learn from past defects and code changes to predict which tests to run? Lucas: It can help, but I'd be cautious. A model trained on historical data might miss new risk patterns — like a new regulation or a change in customer behavior. You still need human judgment to set the business impact. The AI can suggest the probability, but the final curation is human. Luna: Curating tests feels like a skill that's undervalued. Everyone wants to write more automation, but fewer people want to decide which automation matters. Lucas: Right. And that curation is exactly what risk-based testing formalizes. It forces you to ask hard questions about what you're protecting. The biggest challenge I see is that risk changes over time — a feature that's low-risk today might be high-risk next quarter after a market shift. Luna: So how often should teams revisit their risk matrix? Quarterly? After each release? Lucas: At least quarterly, but ideally whenever there's a major product change, a regulatory update, or a spike in production incidents. That trading platform synced their risk review with the quarterly planning cycle. It became part of the conversation, not an extra meeting. Luna: What about teams that want to start but feel overwhelmed? What's the smallest first step? Lucas: Pick your top ten business processes. Not test cases — processes like 'user login' or 'payment checkout.' Rate each for impact and probability. Then look at your existing test coverage for those processes. You'll almost certainly find gaps in high-risk areas and over-coverage in low-risk ones. Fix that imbalance, and you're already doing risk-based testing. Luna: That's a tangible starting point. One final thought: do you think risk-based testing becomes less necessary as CI/CD speeds up and tests get cheaper to run? Lucas: I think the opposite. Faster pipelines mean more deploys, which means more chances for regressions. You can't test everything in a five-minute build. Prioritization becomes even more critical. Risk-based testing isn't a silver bullet — it's a lens. It keeps you asking: 'What will hurt most if we ship it broken?' That question never goes away.