Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Coverage Metrics Hide Real Risk
Transcript
- Lucas: Let me ask you something. Have you ever seen a team celebrate hitting 90 percent code coverage, only to ship a bug that completely slipped through? Luna: Oh, absolutely. I think every engineer has that story. The coverage number looks great, but the test suite is basically checking that the code compiles. Lucas: Right. And that's exactly the trap. Coverage metrics — line coverage, branch coverage — they measure how much of your code has been executed during tests. They don't measure whether your tests actually catch anything meaningful. Lucas: I was looking at a postmortem from a payments startup last year. They had 95 percent line coverage across their checkout service. And then a bug caused double-charging on a subset of international transactions. It ran in production for three weeks before anyone noticed. Luna: Yikes. So 95 percent coverage, zero protection against that bug. What was the actual gap? Lucas: The tests covered every line of the currency conversion module. But they only tested it with a single currency pair — US dollars to euros. The branch that handled fallback rates for less common currencies was never tested with a realistic input. The code executed, but the behavior was wrong. Luna: So coverage was high, but the test data was shallow. One happy path, basically. Lucas: Exactly. And that's the core problem. Coverage is a necessary condition for quality, not a sufficient one. It tells you what code ran. It doesn't tell you what behaviors you validated. Luna: I've heard people call that the difference between 'code coverage' and 'logic coverage.' Is that a useful distinction? Lucas: I think it is. Code coverage asks: did this line execute? Logic coverage asks: did we test this decision's outcome for all meaningful inputs? And that's where mutation testing comes in as a reality check. Luna: Remind me — mutation testing is where you deliberately introduce small bugs into the code and see if your tests catch them, right? Lucas: Yes. You make a change like flipping a comparison from greater-than to less-than, or changing a constant value. Then you run the test suite. If the tests still pass, you've got a blind spot. The mutation 'survived.' Lucas: In that payments startup case, they eventually ran mutation tests on the checkout module. About 40 percent of the mutations survived. Their 95 percent line coverage was essentially a mirage. Luna: Forty percent survival rate is terrifying. That means almost half of the bugs you could insert wouldn't be caught. Lucas: Right. And the thing is, they're not an outlier. I've seen studies where teams with 80-plus percent line coverage still had 20 to 30 percent mutation survival rates. The metrics are decoupled. Luna: So if you're a team relying on a coverage gate in CI — like 'we won't merge below 80 percent' — you might be enforcing a false sense of security. Lucas: Exactly. And that's the argument for supplementing coverage with mutation testing, or at least with behavior-driven test design. Instead of asking 'what percentage of lines did we hit?', ask 'what percentage of possible failure modes did we check?' Luna: Honestly, if today's tech conversation gave you something usable, that's the link — buy me a coffee dot com slash fexingo. Keeps this show ad-free and independent. Lucas: Yeah, it's a small way to support the kind of deep-dive episodes we do. And it really does make a difference. So — back to the metrics. Luna: One practical question: if a team wants to start using mutation testing, what's the overhead? I've heard it's slow. Lucas: It can be. Running hundreds or thousands of mutations on a large codebase takes time. But you don't have to run it on every commit. Run it nightly, or on a subset of critical modules. The payments startup started with just their payment processing pipeline — maybe 15 percent of the codebase — and that gave them the highest-impact findings. Lucas: There are also tools that use incremental mutation analysis — you only test the parts of the code that changed. That brings runtime down to minutes. Luna: So the idea is to use mutation testing as an audit, not a gate. A way to surface blind spots, not a pass-fail check. Lucas: Right. And once you find a surviving mutation, you write a test for that specific behavior. Over time, your test suite gets genuinely stronger, not just numerically higher. Luna: What about branch coverage? Isn't that better than line coverage because it checks both sides of a condition? Lucas: It's better, but still limited. Branch coverage ensures you've tested both the true and false paths of an if-statement. But it doesn't check that you tested the right combination of conditions. For example, a method with two boolean inputs has four possible input pairs. Branch coverage might only test two of them. Lucas: There's an even stronger metric called modified condition/decision coverage, or MC/DC, that the aerospace industry uses. It ensures each condition independently affects the outcome. But most teams don't need that level. Mutation testing gives you a pragmatic proxy. Luna: So if I had to pick one thing to improve test quality this quarter, would you recommend mutation testing over, say, adding more integration tests? Lucas: I'd say: run a mutation test pilot on one critical module first. See how many mutations survive. That will tell you more about your test suite's actual strength than any coverage number. Then decide where to invest. Lucas: Because the real risk isn't low coverage. It's high coverage with low effectiveness. That's the silent failure mode. Luna: And it's surprisingly common. I remember a talk where a speaker said 'your 90 percent coverage might as well be 0 percent if it tests the wrong things.' Lucas: That's harsh, but not wrong. I think the lesson is: measure what matters. And what matters is whether your tests would catch the bugs that have historically bitten you. Look at your last three production incidents. Would your current test suite have caught them? If the answer is no, that's your starting point. Luna: That's a really concrete way to frame it. Not abstract coverage targets, but a gap analysis against real failures. Lucas: Exactly. And that's the approach more teams should take. Coverage is a tool, not a goal. The goal is confidence that your software behaves correctly under the conditions that matter. Luna: All right, so we've covered the problem, the example, and one practical fix. What's the one thing you want listeners to remember from this episode? Lucas: That a high coverage number can hide real risk. Don't celebrate 90 percent unless you've checked that your tests actually catch bugs. And mutation testing is one of the best ways to do that check. Luna: Good note to end on. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.