Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Coverage Percentages Mislead You
Transcript
- Lucas: So there's this question I get a lot when I talk to engineering teams, and it's usually phrased as something like, 'how do we get our coverage up to eighty percent?' And look, I get it, coverage is easy to measure, it's a single number, it feels like progress. But the more I dig into it, the more I think that number is a mirage. Luna: A mirage? You mean teams chasing coverage and still shipping bugs? Because I've definitely seen that happen. Lucas: Exactly. I remember reading about a financial services company that had ninety percent line coverage on one of their core services. They were proud of it. And then a bug slipped through that affected a very specific error-handling branch. The kind of branch that only gets hit when an external API times out in a particular way. Luna: That sounds like a classic line coverage trap. The branch was there, but no test actually forced the code down that path. Lucas: Right. And line coverage would tell you that the line was executed, because maybe part of it ran before the exception. But the actual behavior in that failure mode was untested. So the metric looked great, but the risk was hiding in plain sight. Luna: So what's the alternative? Do we just throw coverage out the window? Lucas: No, it's not about throwing it out. It's about using it the right way. Coverage is useful as a diagnostic, not as a target. The moment you turn it into a KPI, teams game it. They write tests that hit lines but don't assert anything meaningful. Luna: I've seen that too. Tests that just call a function and don't even check the result. You get the coverage, but the test is worthless. Lucas: Right. So today I want to talk about why coverage percentages can mislead you, and what you can do instead to actually measure how good your tests are. And there's a technique that can help you see what your coverage number is hiding. Luna: Are you talking about mutation testing? Because that's been on my radar as something that goes deeper than coverage. Lucas: Exactly. Mutation testing is the idea that you introduce small changes into your code, like flipping a comparison from greater than to less than, and then you run your test suite. If your tests don't fail, that means they didn't catch that particular change. It's like a self-check for your tests. Luna: So it's almost like testing the tests themselves. That's clever. But it sounds like it could be slow and computationally heavy. Lucas: It can be. Running thousands of mutants on a large codebase isn't exactly fast. But you don't need to do it everywhere. You can start with the most critical modules, the ones where a bug would cost you the most. And modern tools have gotten smarter about only running a subset of tests against each mutant. Luna: That makes sense. So you target the high-risk areas first. But then how do you know which areas are high risk? Is that just a judgment call? Lucas: Part of it is judgment, but you can also look at factors like how often a piece of code changes, how many times it's been the source of a bug, and how complex it is. There's even a term for it, 'risk-based testing.' You prioritize your testing efforts based on where failures would have the most impact. Luna: And that's a much more nuanced approach than just saying, 'we need ninety percent coverage across the board.' Lucas: Exactly. And I want to give you a concrete example. There was a healthcare startup I read about, they had a microservice that handled patient billing. They had around seventy percent coverage, which is decent. But when they ran mutation testing on it, they found that the mutation score was only fifty percent. Half of the mutations survived, meaning half of the targeted behaviors were untested. Luna: Fifty percent mutation score on a billing service? That's terrifying. I'd want to know exactly which mutations survived. Lucas: And that's the beauty of it. The tool tells you, 'hey, this mutant survived, which means your tests didn't catch this change.' So you go look at that specific code and think, 'what would happen if this logic were slightly different?' And that's how you find the gap. Luna: So it's not just about the final percentage, it's about the specific surviving mutants that point you to the weak spots. Lucas: Right. And once you fix those tests, your coverage number might go up too, but more importantly, your confidence goes up. You know that the critical paths are actually protected. Luna: I've also heard that coverage can be a useful conversation starter between devs and QA. Like, instead of saying 'your code isn't tested enough,' you can say 'these branches are uncovered, what's the risk here?' Lucas: That's a great point. Coverage becomes a way to drive a conversation about risk, not a stick to beat people with. And I think that's the shift we need. We're moving from 'we need X percent' to 'we need to test the things that matter most.' Luna: And that's a much more mature approach. But I'm curious, are there any tools that make this easier? Like, can you integrate mutation testing into your CI pipeline without it taking forever? Lucas: There are tools like Stryker, for JavaScript, and PIT, for Java, that are designed to integrate with CI. They have options to run mutation testing in a focused way, like on changed code only, or on a schedule. It's not something you run on every commit, but maybe on a nightly basis or before a release. Luna: So you get the full analysis without slowing down the development loop too much. Lucas: Exactly. And the results can be tracked over time, just like coverage. But instead of watching a single number go up, you're watching the mutation score go up, which is a much better indicator of test quality. Luna: I'm sold on mutation testing being a better metric. But I feel like there's still a place for simple coverage numbers. Like, a really low coverage might indicate a serious problem, even if high coverage doesn't guarantee quality. Lucas: Absolutely. I'd never say ignore coverage entirely. It's a useful sanity check. If you have twenty percent coverage, you probably have big gaps. But once you're above, say, seventy percent, the marginal insight of an extra five percentage points is pretty low. That's when you need to shift your focus to something else. Luna: Right. So the conversation changes from 'how much' to 'where' and 'how well.' Lucas: Precisely. And I want to bring it back to the user's perspective. Because tests exist to catch bugs that would affect real people. So instead of just looking at code coverage, think about covering user journeys. Have you tested the main paths a customer would take? Have you tested the failure modes they'd encounter? Luna: That's a good way to frame it. Like, for an e-commerce site, you'd want to test the checkout flow not just the individual components. Lucas: Right. And that's what I call 'scenario-based testing.' You map your tests to real user interactions. And when you do that, you naturally cover more code, but you also cover it in the context of how it's actually used. That's much more valuable than just hitting a line for the sake of it. Luna: So in the end, we're back to the fundamentals: know your user, know your risks, and test accordingly. Lucas: And that's the takeaway. Now, before we wrap up, I want to touch on something that's on my mind because we're able to do this show without ads. We have a small group of listeners who chip in a few dollars a month through buy me a coffee dot com slash fexingo. It's not a big ask, but it genuinely helps us cover the costs of production and keeps the show free for everyone. Luna: And it's really those listeners who make this deep-dive possible. We appreciate every single one of them. Lucas: Anyway, back to testing. So if you're looking at your own coverage numbers, I encourage you to ask yourself: are you measuring the right thing? Are you testing what matters, or just ticking a box? And maybe give mutation testing a try on one critical module. See what it reveals. Luna: That's a great experiment. And I bet you'll find something you didn't expect. Lucas: I guarantee it. And if you want to share what you find, we'd love to hear from you. But for now, that's what we've got. Thanks for listening.