Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Coverage Metrics Lie About Code Quality
Transcript
- Lucas: There's a number that gets thrown around in engineering teams like it's sacred: 80 percent test coverage. 90 percent. Sometimes even 95. And the implication is always the same — high number equals good code. Luna: Right, and I've definitely been on teams where that number is a badge of honour. You hit the target and everyone high-fives. Lucas: Exactly. But here's the uncomfortable truth: code coverage, as most teams measure it — line coverage — is one of the most misleading metrics in software. It tells you which lines of code were executed during a test run. It does not tell you whether those lines were actually verified. Luna: So you get a green checkmark on lines that might not even have a single assertion? Lucas: Correct. A 2024 study by researchers at the University of Zurich looked at open-source JavaScript projects with high coverage — above 80 percent. They found that nearly 40 percent of the covered lines had zero assertions. The test just ran the code and didn't check anything. So the coverage number was inflated by what's essentially a no-op. Luna: Forty percent. That's huge. So teams are celebrating a metric that's basically air. Lucas: And it gets worse. Line coverage also doesn't capture the difference between a simple path through a function and a complex one. Consider an if-else statement. If your test only exercises the true branch, you get 100 percent line coverage for that function — but the false branch might never run. That's a structural coverage gap, and line coverage doesn't penalise it. Luna: So you need branch coverage or decision coverage to catch that. Lucas: Right. Branch coverage tells you whether each possible path — true and false — was taken. But even then, you can have full branch coverage and still miss bugs. Because coverage is about executing code, not about asserting the right behaviour. The classic example is a function that calculates a discount. You can cover both the discount branch and the no-discount branch, but if your assertion only checks that the return type is a number, you miss the case where the discount is negative. Luna: I remember a fintech startup I worked with briefly — they had 90 percent line coverage on their payment processing module. But a production bug let through a transaction where the amount was zero. The test executed the line that subtracted the fee, but the assertion only checked that the balance changed, not that it changed by the right amount. Lucas: Perfect example. The coverage number was high, but the assertion was weak. That's why some teams have started measuring assertion coverage — the ratio of assertions to lines of code. It's a crude proxy, but it's better than nothing. The Zurich study found that projects with higher assertion density had significantly fewer post-release defects. Luna: Yeah, I've seen teams push for 'one assertion per test' as a rule. But that can lead to tests that just check a variable is not null, which is still weak. Lucas: Right. The quality of the assertion matters more than the quantity. A single assertion that checks the exact expected value against the actual output is worth ten assertions that just check existence. And there's another layer: mutation testing. That's where you introduce small faults into the code — changing a plus to a minus, flipping a condition — and see if your tests catch them. Luna: That really puts coverage in its place. If your tests don't detect a mutation, they're not really testing the logic. Lucas: Exactly. Mutation testing gives you a survival score — the percentage of mutations your tests kill. That's a much better indicator of test quality than line coverage. But it's computationally expensive, so most teams don't run it regularly. A pragmatic approach is to use mutation testing on critical modules, like payment or authentication, while accepting line coverage as a loose sanity check on the rest. Luna: So if I'm on a team that's fixated on hitting 90 percent line coverage, what's the first thing I should push back on? Lucas: I'd start by looking at the tests behind the number. Run a report that shows the assertion to line ratio. If it's below, say, 0.1 assertions per line, your coverage is probably hollow. Then look at branch coverage for the most complex functions. Most tools like Istanbul or JaCoCo can show that. If your branch coverage is significantly lower than line coverage, you've found your weak spot. Luna: And then the real conversation becomes: are we testing the right things? Not just hitting a number. Lucas: Exactly. Coverage is a means to an end — the end is catching bugs and preventing regressions. If you're religious about the number but your tests don't catch real defects, you've wasted effort. There's a well known case from a large e-commerce platform that had 95 percent line coverage on their checkout code. They shipped a change that reversed the order of two arguments in a function. The test still passed because it was using a mock that returned a fixed value. The coverage was perfect. The bug hit production and took down the checkout flow for two hours. Luna: And the mock was the real culprit there — it isolated the unit test from the actual integration. Lucas: Right. That's another way coverage can lie. If you mock heavily, your unit tests cover the logic of your code in isolation, but you miss the interaction bugs. Integration tests or contract tests often have lower coverage but higher value. So a team that only looks at unit test coverage is blind to whole categories of defects. Luna: I've started recommending that teams track two metrics: coverage of critical paths — the ones that handle money, user data, core workflows — and mutation survival on those same paths. That gives a much truer picture. Lucas: And that's a great approach. It focuses effort where it matters. For the other 80 percent of the codebase, you can use line coverage as a loose guardrail — say, below 60 percent might indicate untested code. But the obsession with hitting a single high number across the board is counterproductive. Luna: It also creates perverse incentives. Developers start writing tests that just touch lines to bump coverage, rather than writing meaningful tests. Lucas: Absolutely. I've seen tests that call a function but never store the result. That's the definition of a no-op test. And coverage tools count it. Some teams even auto-generate tests that simply invoke every method with default parameters — coverage goes up, but the tests are worthless. Luna: So what's the alternative? If you're starting a new project, what would you measure from day one? Lucas: I'd measure branch coverage on the critical modules, and I'd enforce a minimum assertion density. I'd also run mutation tests on every pull request for those critical modules — even if it takes a few extra minutes in CI. And I'd use a dashboard that shows not just coverage percentage, but also the defect detection rate over time. If your coverage goes up but your bugs don't go down, you're wasting effort. Luna: That's a more honest metric. And it aligns with the real goal: shipping reliable software, not hitting a number. Lucas: Speaking of shipping reliable software — and this ties back to the value of focusing on what actually matters — this show exists because of a community that values thoughtful discussions like this one. A small group of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's what keeps us ad-free and able to dive deep into topics like test coverage. If you find these conversations useful, that's a way to make sure they continue. Luna: Yeah, it's a simple model, but it works. No sponsors, no tracking — just people who appreciate the content and want to support it. Lucas: Exactly. So back to coverage — one more point that often gets overlooked is the difference between structural coverage and data coverage. Even if you have 100 percent branch coverage, you might be testing with only one set of input values. If your function handles a range of inputs — say, interest rates from 0 to 20 percent — but your tests only use 5 percent, you've covered the code paths but not the data space. Luna: That's a good point. So you need boundary value analysis and equivalence partitioning. Test at the edges. Lucas: Right. There's a known bug in a financial calculation where the interest rate was parsed as integer instead of float. All tests used whole numbers, so coverage was high. But a user input of 5.5 percent caused a truncation bug. That's a data coverage failure. Luna: And a good reminder that coverage is just one tool in the toolbox. It's not a magic number. Lucas: Exactly. So next time someone on your team proclaims 'we have 90 percent coverage,' ask them: coverage of what? With what assertions? At what boundaries? And if they can't answer, the number is probably lying to you.