Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Coverage Metrics Lie and What to Use Instead
Transcript
- Lucas: So you've got 95 percent line coverage on your backend service. Your CI badge is green. Your team feels good. But here's the uncomfortable truth — that number is nearly meaningless when it comes to catching real bugs. Luna: That's a bold claim. I mean, coverage is one of the few metrics we actually track. Are you saying it's worthless? Lucas: Not worthless, but dangerously misleading. Line coverage tells you which lines of code executed during your tests. It does not tell you whether those tests actually verified the behavior. You can have a test that calls a function, checks nothing, and still get 100 percent line coverage. Luna: Okay, so what's the alternative? We can't just run tests and hope for the best. Lucas: There is a technique called mutation testing that directly measures test quality. Instead of counting lines executed, it introduces small faults — mutations — into your code and checks whether your tests catch them. If a mutant survives, you have a gap in your test suite. Luna: So it's like a vaccine for your codebase — inject a weak version of the bug and see if the immune system responds. Lucas: Exactly that. I saw this in practice at a FinTech startup I worked with briefly. They had a payment reconciliation module. Line coverage was 98 percent. Everyone was proud. Then they ran mutation testing on it. The mutation score was 62 percent. Nearly 40 percent of injected faults went undetected. Luna: That's a huge gap. What kind of mutations are we talking about? Like changing a greater-than to a greater than or equal? Lucas: Exactly. The tool flips operators, deletes method calls, changes return values, swaps boolean conditions. Simple changes that would cause real bugs. If no test fails, your test suite allowed that bug to pass through. In that FinTech case, one surviving mutant changed a fee calculation from rounding up to rounding down. That would have cost the company thousands per month in lost revenue. Luna: And the line coverage badge still said green. So what's a good mutation score to aim for? Lucas: In practice, 80 to 90 percent mutation score is excellent. Getting above 90 is very hard because of something called equivalent mutants — mutations that produce functionally identical code. For example, changing 'if ' to 'if ' when x is an integer — same behavior, but the tool flags it as a surviving mutant. You have to manually suppress those. Luna: So mutation testing has its own friction. How do teams actually integrate it without slowing down every commit? Lucas: Most teams don't run full mutation testing on every push. They run a targeted subset on changed files in the pull request, or they schedule a full run overnight. Tools like PIT for Java, Stryker for JavaScript and TypeScript, and Cosmic Ray for Python have gotten fast enough that a focused run takes minutes, not hours. Luna: I've heard of PIT. But Stryker — is that the one that also supports C sharp and Scala? Lucas: Stryker started as a JavaScript tool but now has versions for C sharp and Scala as well. It also gives you a nice HTML report showing exactly which lines survived mutation and what the mutation was. That report is gold for a code review — it tells your teammate 'hey, this new method has no tests that actually verify its logic'. Luna: So it shifts the conversation from 'coverage number' to 'did we actually test the behavior?' That seems like a healthier metric. Lucas: Healthier and harder to game. You can't cheat mutation score by writing trivial assertions. You have to actually assert something meaningful. And that's the whole point — we want tests that catch real bugs, not tests that make dashboards look good. Luna: What about the argument that mutation testing is too slow for large codebases? I've heard teams say it takes hours on a repo of a few hundred thousand lines. Lucas: That's a fair concern. But there are strategies. First, you can set a time limit per class — if mutation testing on a single class takes more than, say, 30 seconds, skip it for that run. Second, you can use incremental analysis so only mutated code is re-tested. Third, you can focus on high-risk modules — payment logic, authentication, data transformations — and skip trivial getters and setters. Luna: So you don't have to mutate everything. That makes it practical. And the payoff is higher confidence, especially before a release. Lucas: Right. Think about it this way: if you're about to deploy a critical update, and your mutation score on that module is 85 percent, you have real evidence that your tests are doing their job. If it's 50 percent, you know you need to invest more in testing before shipping. Luna: I like that it gives a concrete go/no-go signal. Not just 'coverage is high, ship it'. Lucas: Exactly. It also surfaces what I call 'zombie code' — lines that are covered by tests but never actually validated. You discover that a lot of your test suite is just exercising code, not checking it. And once you see that, you can never unsee it. Luna: And that connects back to something we've talked about before — test debt. Tests that exist but don't actually protect you are a liability, not an asset. Lucas: Exactly. And mutation testing is the best tool I know for measuring that liability. It exposes which tests are earning their keep and which ones are just taking up CI time. Luna: Speaking of taking up time — if listeners found this deep dive useful and want to support the show keeping it ad-free and independent, there's a simple way to do that. We've set up a page where you can buy us a coffee, literally. It's at buy me a coffee dot com slash fexingo. No pressure, but every bit helps us keep episodes like this coming. Lucas: Yeah, and genuinely, it means a lot. We don't run ads, so listener support is what keeps the lights on here. Anyway — back to mutation testing. One more thing I think is worth mentioning: the best teams I've seen treat mutation score as a trending metric, not a hard gate. They track it over time, look for regressions, and celebrate improvements. Luna: So it's more of a health indicator than a pass-fail. Lucas: Exactly. And when combined with other signals — like flakiness rate, time to feedback, and defect escape rate — it gives you a much fuller picture of your test suite's effectiveness. Line coverage alone just doesn't cut it anymore.