Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Coverage Metrics in CI Are Misleading
Transcript
- Lucas: You're looking at your CI dashboard, green checkmark, 87 percent line coverage. Ship it. And then three weeks later the thing breaks in production because the path that mattered was never tested. Luna: Right. Coverage as a vanity metric. I've been there. It's so easy to chase that number and forget what it actually tells you. Lucas: Exactly. And I think the gap is especially dangerous because coverage in CI is often treated as a gate. If you're below some threshold — say, 80 percent — the pipeline fails. But that threshold has almost nothing to do with whether your tests are good. Luna: So what does coverage actually measure? I mean, the tool tells you which lines got executed during the test run. That's it, right? Lucas: That's it. Line coverage says 'this line of code was touched by at least one test.' It doesn't tell you whether the test checked the outcome, or whether it exercised the edge cases. A test can call a method, hit every line, and never assert a single meaningful thing. That still counts as covered. Luna: So you could have a test that runs a function, doesn't check the return value, and your coverage tool says 'green'? That seems broken. Lucas: It happens all the time. In fact, I talked to a team at a mid-sized SaaS company — they had 90 percent line coverage across their core service. Their CI would reject any PR that dropped below 85. They were proud of it. Then they had a production incident where a user's billing data got corrupted because a certain conditional path — something like 'if the subscription status is cancelled and the user has a pending credit' — that path wasn't tested. The test that 'covered' those lines only ran the happy path. The variables were set up so the condition never evaluated to true. Luna: Ouch. So the coverage number gave them false confidence. They thought they were protected, but the most critical business logic path was never exercised. Lucas: Exactly. And the fix wasn't to raise the threshold to 95 percent. The fix was to understand what kind of coverage actually correlates with bug detection. There's research on this — studies by Google and Microsoft have shown that line coverage alone has a very weak correlation with defect detection. Branch coverage is slightly better. But even branch coverage can be gamed. Luna: What's the difference between line and branch coverage again? Lucas: Line coverage asks 'did this line execute?' Branch coverage asks 'did we take both the true and false paths of every conditional?' So for an if-else, branch coverage requires that both branches are exercised. That's better, but still not enough. Because you might cover both branches but never check that the output is correct. Or you might cover both branches but only with trivial inputs. Luna: So what's the metric that actually matters? Is there one? Lucas: There isn't a single magic number, but there's a technique that correlates much better with real bug finding: mutation testing. You've probably heard us talk about it before. The idea is simple — the tool introduces small changes into your code, like flipping a less-than to a greater-than, or deleting a line. Then it runs your tests. If your tests pass despite the mutation, that means the bug was silent. Your tests didn't catch it. The mutation 'survives.' The score — the percentage of mutations your tests kill — gives you a much more honest picture of test effectiveness. Luna: Right. We did an episode on mutation testing a while back. And I remember the example: if you have a test that checks that a function returns a number, but you never verify that the number is correct, a mutation that changes the calculation might pass unnoticed. So mutation testing would flag that as a survived mutant. Lucas: Exactly. And the really interesting thing is that mutation scores tend to be much lower than line coverage. I've seen projects with 85 percent line coverage and a 40 percent mutation score. That's a huge gap. It means more than half of the injected bugs wouldn't be caught. And that's the real risk. Luna: So why isn't everyone doing mutation testing in CI? It sounds strictly better. Is it too slow? Lucas: Speed is the big one. Mutation testing can be computationally expensive. Running your full test suite against hundreds or thousands of mutated versions of your code can take hours. That's not feasible for a CI pipeline that needs to give feedback in minutes. But the landscape is changing. Tools like Stryker, PIT, and mutmut have made big strides in performance by using incremental analysis and only mutating code that's actually covered. Some teams run mutation tests on a smaller subset — say, the most critical modules — or schedule them as a nightly job instead of blocking the pipeline. Luna: So it's not a drop-in replacement for coverage. It's a complementary signal. You still want some coverage metrics, but you need to interpret them correctly. Lucas: Right. The mistake is treating coverage as a goal. A better approach is to use coverage as a heuristic for finding untested areas, not as a quality gate. You look at the uncovered lines and ask 'is this a gap we should fill?' And for the covered lines, you use mutation testing to verify that the tests are actually effective. Luna: I've also seen teams use something called 'diff coverage' — only measuring coverage on the lines changed in a pull request. That seems more focused than looking at the entire project. Lucas: Diff coverage is great because it ties the metric to the change. It prevents the situation where you have a high overall coverage but new code is untested. Services like Codecov and Coveralls support that. But even diff coverage has the same limitation: it measures execution, not assertion quality. Luna: Speaking of quality, I recently read a post by a Netflix engineer about their testing strategy. They don't have a coverage gate at all. Instead, they rely on a combination of property-based testing, integration tests, and a team culture where developers are expected to write meaningful tests. The emphasis is on test design, not test quantity. Lucas: That's the right mindset. And it connects to something we've talked about before: the difference between 'test coverage' and 'test effectiveness.' Coverage is a measure of what you've looked at. Effectiveness is a measure of what you've found. You want the latter. Luna: So if a team is currently using a coverage gate in CI, what's the first thing they should do? Lucas: I'd say audit the coverage number. Run a mutation test on a critical module. See if the mutation score aligns with the coverage percentage. Chances are, it won't. Then use that as a talking point to shift the conversation from 'we need 90 percent coverage' to 'we need tests that actually catch bugs.' And then gradually introduce mutation testing into your workflow, even if it's just a periodic check. Luna: And don't panic if your mutation score is low. It's a starting point. You can improve over time. Lucas: Exactly. And there's another angle: using coverage data to detect 'test flakiness' or 'test smell.' If a test that's supposed to cover a complex logic path suddenly stops hitting that path due to a code change, that's a valuable signal. Coverage drift can indicate that a test is no longer testing what it's supposed to. Luna: That's a good point. I hadn't thought of coverage as a diagnostic tool beyond the percentage. It's more like a map: it shows you where you've been, but it doesn't tell you if you found anything useful. Lucas: Right. And if you're doing that kind of analysis, you're already ahead of most teams. The real danger is the false security of a high number. I've seen companies ship confidently with 90 percent coverage and then scramble when a bug slips through. The coverage number didn't lie — it just didn't tell the whole story. Luna: It's like having a security camera that covers 90 percent of your building, but it only records in low resolution and the footage is never reviewed. You feel safe, but you're not. Lucas: That's a great analogy. And it leads to the practical takeaway: don't set a coverage threshold without understanding what it means. And if you do have one, pair it with mutation testing or at least with a process of test review. Because the goal isn't to hit a number — it's to prevent bugs. Luna: Speaking of preventing bugs, this kind of conversation is exactly why listener support matters. It keeps us free to dig into topics like this without worrying about ad revenue. If today's episode gave you a useful perspective, you can support the show at buy me a coffee dot com slash fexingo. Lucas: Yeah, that really does help us keep the episodes focused and ad-free. And it's a small way to make sure we can spend time on the details that matter. Luna: Alright, back to the practical side. So let's say you've decided to run mutation testing on a critical module. What does that look like in practice? What tool would you recommend? Lucas: For Python, mutmut is pretty straightforward. For Java, PIT is the standard. For JavaScript, Stryker. They all integrate with CI. The setup is usually just adding a script that runs the mutation tool on a specific subset of your code. The first run might take a while, but you can configure it to only mutate code that's actively being changed. Luna: And you'd want to set a baseline score, then track it over time. Just like you would with coverage, but with more meaning. Lucas: Exactly. And here's a tip: start with the code that handles money, authentication, or any critical business rule. Those are the areas where a missed mutation could be catastrophic. Then expand to less critical modules. Over time, you build a culture of test effectiveness, not just test coverage. Luna: I think the key takeaway is that coverage is a tool, not a target. It's useful for finding blind spots, but it shouldn't be the sole measure of quality. Lucas: Right. And that's the message I hope teams take away. Next time you see that green checkmark and that 87 percent, ask yourself: '87 percent of what? And does it actually matter?' Because the answer might surprise you.