Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Needs a Risk-Based Prioritization Strategy
Transcript
- Lucas: Let's say you've got a regression suite that takes four hours to run, and your team's about to push a release at end of day. What do you do when the build lands at three PM and you only have ninety minutes before the deploy window closes? Luna: I've been there. Panic-select a handful of tests and hope for the best? Lucas: Exactly. Most teams do that—or they run the full suite and just let the deploy slip. But there's a structured way to make that call: risk-based test prioritization. Instead of running tests in the order they were written or alphabetically, you rank them by the risk of a failure going undetected. Luna: So you're not just running the fastest tests first, you're running the ones that would hurt most if they broke. Lucas: Right. You assign each test case a risk score based on two factors: the likelihood of failure and the impact if it does fail. Likelihood might come from historical failure rates, code churn in the area being tested, or complexity of the feature. Impact could be things like revenue exposure, data integrity, or security implications. Luna: I like that because it forces you to be explicit about what matters. It's not just 'this test is important because we wrote it first.' Lucas: Yeah, and the scoring doesn't have to be fancy. You can use a simple 1-to-5 scale for each dimension, multiply them, and sort descending. The high-risk tests run first, and if time runs out, the low-risk ones get deferred to the next regression cycle. You're making an informed trade-off instead of a blind one. Luna: I'd want to make sure the risk model itself is accurate. What if a test that never fails suddenly starts failing because of a subtle change? Lucas: That's a good point, and it's why the risk model needs to be dynamic. You can't just set it once and forget it. Ideally, after each test run, you update the likelihood scores based on actual pass-fail results. Some teams also incorporate code coverage data—if a file that's heavily tested gets a commit, tests covering that file get a temporary boost. Luna: So the model evolves as the codebase evolves. That makes sense. Do you have a concrete example of where this worked well? Lucas: Yeah, there's a case from a payments startup I talked to last year. They had about eight thousand automated tests, and a full regression took around six hours. They implemented risk-based prioritization using a simple Python script that read test metadata and recent Git blame data. Luna: Git blame for likelihood—clever. Lucas: Right. They scored each test by how many commits touched its related source files in the last two weeks, plus the criticality of the feature—payment processing, user accounts, etc. The high-risk tests became their 'gate' suite, which took about ninety minutes. The medium-risk ran next, and low-risk ran overnight. Luna: And the result? Lucas: They cut their pre-deploy regression time by about seventy percent. More importantly, they caught three critical bugs in the first month that would have hit production because those bugs were in high-churn, high-impact areas. Before, those tests might have run later in the suite and been cut off by the deploy deadline. Luna: So they actually found bugs they would have missed. That's the real win. Lucas: Exactly. And the beauty is that this approach scales. You can start simple with a spreadsheet and a manual scoring session, then automate as you go. The key is the mindset shift: tests are not equal, and pretending they are leads to wasted time or missed bugs. Luna: I also like that it gives the QA team a way to communicate risk to product managers. 'We only had time to run the top thirty percent of risk-weighted tests, so there's a chance X or Y could be broken.' That's better than 'we didn't finish testing.' Lucas: Yeah, it turns testing from a binary pass-fail into a risk conversation. And that ties directly into the broader shift-left and continuous testing strategies we've talked about on the show. Luna: Absolutely. If today's deep dive into risk-based prioritization gave you a practical takeaway to try with your own test suite, that's exactly the kind of thing we love hearing about. And if you find the show useful, listener support is what keeps us ad-free and focused on the substance. You can help at buymeacoffee.com/fexingo—it's a simple way to keep the episodes coming. Lucas: Yeah, every contribution helps us spend more time on research and less on worrying about funding. But back to the risk model—one thing that trips teams up is deciding what 'impact' actually means. You can't just say 'high' and 'low.' Luna: So how do you define impact concretely? Lucas: A good framework is to tie it to business outcomes. For example, impact level 5 might be 'direct revenue loss or security breach.' Level 4 is 'major feature broken for all users.' Level 3 is 'feature broken for a subset.' Level 2 is 'cosmetic issue.' Level 1 is 'no user-facing effect.' You map each test to one of those. Luna: And likelihood? You mentioned Git churn. What else? Lucas: Historical failure rate is the most straightforward. You can pull that from your CI database. Also, complexity metrics like cyclomatic complexity of the code under test, or whether the test itself has been flaky. Some teams even include the developer's experience—if a junior dev made the commit, the risk is slightly higher. Luna: That might ruffle some feathers. Lucas: Yeah, you have to be careful with that. But the data doesn't lie—commits from newer team members do correlate with higher bug rates, at least initially. The point is, you can tailor the model to your context. The important thing is that it's transparent and data-driven, not political. Luna: Once you have the scores, how do you decide the cutoff? Do you always run all high-risk tests, or do you set a time budget? Lucas: The most common approach is time-boxed: you allocate, say, two hours for the pre-deploy regression, and you run tests in priority order until time runs out. That way, you're always maximizing risk coverage within the available window. Another approach is to run all tests with a risk score above a threshold—like 15 out of 25—and defer the rest. Luna: But what about the deferred tests? They still need to run eventually. Lucas: Right. They become part of the next regression cycle, or you run them as a nightly full-regression. The key is that they don't block the deploy. The team accepts the residual risk, documents it, and moves on. Over time, the data from those deferred runs feeds back into the risk model, so the prioritization gets smarter. Luna: I can see this being especially powerful for teams doing continuous deployment. If you're deploying multiple times a day, you can't run an eight-hour suite every time. Lucas: Exactly. In fact, for CD, you might run only the top 10 percent of risk-weighted tests per deploy, and then a full regression nightly. That's what some of the big SaaS companies do. They rely on risk-based prioritization to keep deployment velocity high while maintaining safety. Luna: There's also the maintenance overhead. The risk model itself needs tests. Lucas: Good point. You should validate periodically that the prioritization is actually catching bugs. You can do that by running the full suite in a staging environment and comparing which tests would have been deferred versus which ones actually found failures. If you see a pattern where deferred tests catch critical bugs, you need to adjust your impact or likelihood scoring. Luna: So it's a feedback loop, not a one-time setup. Lucas: Exactly. And the investment pays for itself quickly. That payments startup I mentioned—they spent about two weeks building the initial risk model and integrating it with their CI pipeline. Within a month, they estimated it saved them at least two production incidents. That's a huge ROI. Luna: Two weeks of work to save multiple incidents is a no-brainer. I'm surprised more teams don't do this. Lucas: I think it's because testing culture has historically treated all tests as equally important. But once you shift to a risk-based mindset, you start asking better questions. Not 'how many tests passed?' but 'what is the remaining risk after this test run?' That's a much more useful metric for the business. Luna: And it aligns QA with product goals. You're not just a gatekeeper, you're a risk advisor. Lucas: Exactly. That's the future of testing, in my opinion. It's not about running more tests, it's about running the right tests at the right time. Risk-based prioritization is one of the most practical ways to get there.