Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Needs a Risk Matrix
Transcript
- Luna: So I was looking at our test metrics last week, and I realized something uncomfortable. Lucas: What's that? Luna: We run over ten thousand automated tests every night. And most of them pass. But the ones that fail are almost never the ones I care about. Lucas: Yeah, that's a classic symptom. You've built a test suite that's thorough in coverage but blind to priority. It's testing everything equally, which means it's testing the wrong things harder. Luna: Exactly. And when I dig into the critical bugs that hit production last quarter, almost none of them were caught by our automated regression suite. Lucas: Right. That's the disconnect. A lot of teams optimise for test count or code coverage, but those numbers don't correlate with catching the bugs that actually hurt your business. There's a better approach — it's called risk-based testing, and it starts with a simple tool: a risk matrix. Luna: Okay, walk me through it. How does a risk matrix change what you test? Lucas: You plot every feature or user journey on two axes: the likelihood of failure and the business impact if it does fail. A feature that handles payments gets high impact. A feature that changes the colour of a button gets low impact. A legacy module nobody touches gets low likelihood. A new integration your team rushed out gets high likelihood. Luna: So you end up with four quadrants. High impact, high likelihood gets the most test attention. Low impact, low likelihood gets the least. Lucas: Exactly. And the matrix forces you to have a conversation about which of those quadrants you're under-testing or over-testing. I saw a real case study from a fintech company last year — they mapped their entire regression suite against a risk matrix and found that 60 percent of their test cases were covering low-risk paths. Luna: Ouch. So they were spending most of their compute and maintenance time on tests that barely mattered. Lucas: Exactly. They cut those tests and shifted the freed-up capacity to high-risk areas. Their critical defect detection rate went up by about 40 percent in the next quarter. And their regression suite ran in half the time. Luna: But what about the low-risk paths? Do you just stop testing them entirely? Lucas: You don't abandon them — you test them at a lower cadence. Maybe once a sprint instead of every build. Or you rely on manual exploratory testing for those paths. The key is intentionality. Luna: I can see how this would be especially relevant right now. With the regulatory pressure in fintech and healthcare, regulators want proof that you're testing critical functions thoroughly. Lucas: That's a great point. A risk matrix gives you a defensible rationale. You can show a regulator: we test this feature every single build because a failure here would cost millions and have a 20 percent probability of occurring. And we test this other feature once a month because failure would be an inconvenience, not a crisis. Luna: It also helps with the flaky test problem we talked about in a previous episode. If you know a test is flaky but it's covering a low-risk path, you can just reduce its run frequency instead of spending weeks debugging it. Lucas: Exactly. You're making explicit trade-offs instead of pretending every test is equally important. And that honesty is actually liberating for a QA team. Luna: There is a pushback I've heard, though. Some engineers say risk-based testing is just an excuse to skip writing good tests. Lucas: I've heard that too. But I think it misunderstands the point. Risk-based testing isn't about writing fewer tests — it's about writing the right tests. If your test suite is a bloated safety net with holes in the wrong places, you're not doing quality work. Luna: So how do you actually start building a risk matrix for your own product? Lucas: Gather a small group — a product manager, a senior developer, a QA lead. List every major feature or user story. Rate each one for impact and likelihood on a simple scale: low, medium, high. Then plot them on a whiteboard. The disagreements are where the insight lives. Luna: That sounds like a two-hour meeting that could save months of wasted test effort. Lucas: It really can. And you revisit the matrix every quarter because the risk profile changes. A feature that was low-risk last year might become high-risk after a new integration. Luna: So it's a living document, not a one-time exercise. Lucas: Exactly. And once you have it, you can start mapping your existing test cases to the quadrants. You'll almost certainly find that you're over-testing in one area and under-testing in another. Luna: I think this is one of those ideas that sounds obvious in hindsight, but almost nobody actually does it. Lucas: That's the thing about good engineering practices. They're simple in concept but require discipline to execute. And if today's conversation gave you something usable, I'll mention that listener support is what keeps this show ad-free and independent. It's a small thing, but if you find value in episodes like this, you can support us at buy me a coffee dot com slash fexingo. Luna: Yeah, it really helps us keep the focus on practical topics rather than chasing sponsors. And we love hearing from listeners too. Lucas: Alright — let's get back to the matrix. Once you've mapped your tests, the next step is to define your test strategy per quadrant. Luna: So for high-impact, high-likelihood features — you'd want full automation, run on every commit, with multiple environments? Lucas: Exactly. For high-impact, low-likelihood features, you might still want full automation but run it nightly rather than per commit. For low-impact, high-likelihood, you might rely on monitoring and quick manual checks. And for low-impact, low-likelihood — maybe just smoke tests once a sprint. Luna: That actually makes the decision process transparent. Everyone knows what to expect. Lucas: And it helps with resource allocation. If your team is stretched thin, you know exactly where to cut without increasing business risk. Luna: One thing I'm still wondering about — how do you measure impact objectively? Isn't it subjective? Lucas: It can be subjective, but you can anchor it to concrete metrics. Revenue at risk, number of affected users, regulatory fines, support ticket volume after a failure. The team can agree on thresholds: anything affecting more than five percent of users is high impact. Anything below one percent is low. Luna: So you calibrate the scale to your own context. Lucas: Exactly. And likelihood can be based on historical data — how often has this module changed? How many bugs have been reported in it? What's the cyclomatic complexity? Luna: That's a lot more data-driven than I expected. I thought it would just be gut feel. Lucas: Gut feel gets you started. But over time, you replace it with real metrics. The matrix becomes a feedback loop — you track whether your predictions match reality and adjust. Luna: Have you seen any teams actually do this well? Beyond the fintech case. Lucas: A healthcare software company I consulted with used a risk matrix to redesign their test suite for a patient scheduling system. They found that the appointment booking flow was high-risk and under-tested. They added automated tests for edge cases — like overlapping appointments and insurance verification — and caught three critical bugs before their next release. Luna: So it's not just about cutting tests, it's about adding them where they matter. Lucas: That's the whole point. A risk matrix doesn't tell you to test less. It tells you to test smarter. Luna: I think I'm going to try this with our team next sprint. Lucas: Start small. Pick one feature area. Build the matrix for that area. Map your existing tests. See what you find. Luna: And if nothing else, it'll make the next post-mortem a lot more interesting. Lucas: That's one way to put it. I think the real benefit is that you stop having arguments about test coverage and start having arguments about risk. And those are much more productive. Luna: Alright, I'm sold. Let's see how it goes. Lucas: Good luck. Let us know how it turns out.