Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Coverage Metrics Lie and What to Use Instead
Transcript
- Lucas: Luna, I want to talk about a number that gives a lot of engineering teams a false sense of security: code coverage percentage. Luna: That green checkmark next to a high number in the CI pipeline. Feels good, right? Lucas: Feels great. Until you ship a bug through code that's supposedly 'covered.' I've got a concrete case. A mid-sized fintech company I know — let's call them PayFlow — had a test suite with 95 percent line coverage. That's enviable by most standards. Luna: And then they had a production outage? Lucas: Exactly. A payment reconciliation job failed silently for three days. The root cause was a condition where two edge-case flags collided. The line that contained that condition was executed in tests, but none of the tests checked the combination that actually caused the bug. Line coverage said 'covered.' But the test didn't assert anything meaningful about that condition. Luna: So line coverage measures that the line ran, not that it was tested properly. That's the classic distinction. Lucas: Right. And it's not just line coverage. Branch coverage is a step up — it checks whether both the true and false branches of a condition are taken. But even branch coverage can miss things. For example, a condition with multiple sub-expressions — like if A and B — branch coverage might only test the cases where A is true and B is true, and where A is false. But what about when A is true and B is false? Branch coverage doesn't necessarily cover that combinatorially. Luna: That's where condition coverage comes in. It checks each sub-expression independently. But even then, you can have 100 percent condition coverage and still miss a bug caused by the order of evaluation or a side effect. Lucas: Exactly. So the question becomes: what metric actually tells you your tests are good? My answer is mutation testing. Mutation testing introduces small changes — 'mutants' — into your code, like flipping a less-than to a greater-than, or deleting a line. Then it runs your test suite. If your tests fail, the mutant is killed. If they pass, the mutant survives, and you've found a gap in your tests. Luna: So it's like a test of your tests. Instead of asking 'how many lines executed,' it asks 'if I break something, will you catch it?' Lucas: Precisely. And the metric is mutation score percentage — the percentage of mutants killed. I've seen teams that had 90 percent line coverage but only a 60 percent mutation score. That's a huge gap. The mutation testing revealed that their tests were hitting the code but not really checking the logic. Luna: But mutation testing is famously slow, right? Running your suite against hundreds or thousands of mutants can take hours. Lucas: It can be. But modern tools have gotten smarter. Tools like Stryker for JavaScript, PIT for Java, or Mutatest for Python use techniques like incremental mutation, only mutating changed files, or running a subset of tests. I've seen teams integrate mutation testing into their CI pipeline for pull requests, targeting only the changed lines. That keeps the runtime under a few minutes. Luna: So the feedback loop stays fast enough for developers to act on it. Lucas: Exactly. And once you have a mutation score, you stop obsessing over line coverage. You set a mutation score target — say, 80 percent — and let line coverage be whatever it is. I've seen teams drop line coverage from 95 to 70 percent because they deleted useless tests that were inflating the number, and actually added tests that killed mutants. Luna: That sounds like a cultural shift. It's no longer about chasing a number; it's about asking 'did we actually test the logic?' Lucas: Right. And that leads to better conversations during code review. Instead of 'our coverage is 85 percent, we're good,' it becomes 'here are the surviving mutants — do we care about these cases?' Luna: And sometimes the answer is no. Some mutants are equivalent — they produce the same behavior as the original code. Or they test something trivial. Mutation testing forces you to prioritize. Lucas: That's a great point. Equivalent mutants are a real issue. Tools try to detect them automatically, but it's not perfect. Still, the signal from mutation testing is way more honest than coverage percentage. I'd rather have a 70 percent mutation score with a clear list of unchecked scenarios than a 95 percent line coverage that hides everything. Luna: So practically, if a team is still using line coverage as a gate, what's your advice to switch? Lucas: Start with a pilot on one service or module. Run mutation testing on that module for a sprint. Compare the mutation score to the line coverage. I guarantee you'll find surprises. Then present those findings to the team. Show them a mutant that survived and caused a real bug. Once they see the value, they'll want to adopt it. Luna: And gradually expand. That's how most good testing practices take hold. Lucas: Exactly. And speaking of practices that make a real difference — this podcast is ad-free and listener-supported. If today's conversation gave you something usable, you can support us at buy me a coffee dot com slash fexingo. That's buy me a coffee dot com slash f-e-x-i-n-g-o. We keep the show free because of listeners like you, so thanks for considering. Luna: Yeah, every bit helps us keep diving into topics like this. Now, back to mutation testing — one concern I've heard is that mutation testing encourages writing tests that are too tightly coupled to the implementation. Lucas: That's a real risk. If you write tests that simply assert the specific implementation rather than the behavior, then killing mutants becomes easy — but your tests become brittle. The key is to design mutation tests to focus on behavioral mutants, not implementation details. For example, mutating a private helper method might not matter if the public behavior is tested well. Luna: So you want to target mutations that change observable outcomes. Like an API response, a database write, or a side effect. Lucas: Right. And tools allow you to configure which types of mutants to generate. You can disable mutants that change constants or remove method calls if those don't align with your testing goals. It's not a silver bullet; it's a tool that requires thought. Luna: But it's a much better conversation starter than line coverage. Instead of arguing about whether we need 80 or 90 percent, you're arguing about which scenarios matter. Lucas: Exactly. And that's where risk-based testing comes in. You can prioritize killing mutants that correspond to high-risk scenarios — like payment logic, authentication, or data integrity. Low-risk mutants, like a display formatting change, you might accept. Luna: So the mutation score becomes a risk indicator. A high score means you've covered the important logic. Lucas: Yes. And that's way more useful than a blanket coverage number. I'd love to see more teams adopt mutation testing not as a replacement for coverage, but as a replacement for the conversation about coverage. Luna: It shifts the focus from quantity to quality. And in testing, that's the real win. Lucas: Exactly. So next time someone on your team proudly announces 'we have 95 percent coverage,' ask them 'what's your mutation score?' You might get a lot of blank stares. But that's the start of a better testing culture. Luna: And probably a better product too. Lucas: I think so. I'm Lucas, that's Luna, and this has been Software Testing with Fexingo.