Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Is a Liability Without Code Coverage
Transcript
- Lucas: Luna, let me ask you something. If I told you a team had 85 percent code coverage on their main payment service, would you feel good about deploying to production? Luna: Honestly? I'd want to know what kind of coverage. Line coverage? Branch coverage? And what the tests actually assert — not just that they ran. Lucas: Exactly. And that's the trap so many teams fall into. They set a coverage target — 80 percent, 90 percent — and then they celebrate when they hit it. But the number itself can be dangerously misleading. Luna: I've seen it firsthand. A fintech startup I consulted for had 85 percent line coverage on their payment processing module. They were proud of it. Then they shipped a bug that charged customers twice for the same transaction. Lucas: And the tests didn't catch it. Luna: They didn't. Because the coverage was mostly in the 'happy path' — the main flow where everything works. The edge case where a network timeout happened mid-processing? That path had zero test coverage. Lucas: Right. And line coverage — which is what most tools report by default — just tells you which lines of code were executed during the test run. It doesn't tell you whether every logical branch was tested. It doesn't tell you whether the test actually verified the correct outcome. Luna: So you can have 100 percent line coverage and still miss a major bug. Lucas: Absolutely. There's a famous example from a study of open-source Java projects — several had over 90 percent line coverage but still had critical defects. The tests were exercising the code, but they weren't checking the right things. Luna: What's the alternative? Branch coverage, I assume? Lucas: Branch coverage is better. It measures whether each possible branch of a control structure — if-else, switch cases — has been executed. That catches a lot more. But even branch coverage has blind spots. For example, it doesn't measure condition coverage — whether each sub-condition in a compound boolean expression has been tested both ways. Luna: So you have line coverage, branch coverage, condition coverage — and then there's mutation testing, which I think is the most honest metric of all. Lucas: Mutation testing is brutal. It introduces small changes into your code — swapping a plus for a minus, flipping a boolean condition — and then checks whether your test suite catches the change. If the tests still pass, you have a gap. Luna: It's like a stress test for your tests. And the results are often humbling. I remember a team at a European bank that ran mutation testing on a legacy risk-calculation module. They thought their coverage was great — 90 percent line coverage. Mutation testing revealed that only about 40 percent of the mutated code was actually detected. Lucas: Forty percent. That's terrifying. And that's not an outlier. Multiple studies have found that line coverage correlates poorly with mutation score. You can have high line coverage and low defect detection. Luna: So what's the practical takeaway for our listeners? Should they stop measuring coverage entirely? Lucas: No, but they should stop treating coverage as a goal in itself. The goal is catching bugs and preventing regressions. Coverage is just one proxy. And the type of coverage matters enormously. Luna: What would you recommend as a first step for a team that's currently focused on line coverage? Lucas: First, switch your coverage reporting to branch coverage as the primary metric. Most tools support it — JaCoCo, Istanbul, Cobertura. It's a simple configuration change. Then, start running mutation testing periodically — maybe once per sprint on critical modules. Luna: And what about coverage targets? Should teams have them at all? Lucas: Targets are fine as a floor, but they shouldn't be a ceiling. The real discipline is looking at uncovered code and asking: 'Is this okay? Or does this need a test?' A team I worked with at a logistics company did a code review specifically for test gaps. They'd ship every pull request with a report showing uncovered branches. Over three months, their production incident rate dropped by 60 percent. Luna: That's a great example. It's not about hitting a number — it's about visibility and conversation. Lucas: Exactly. And that's where I think the industry is slowly moving. We're seeing more teams adopt mutation testing in CI pipelines, and more tools making it practical. Stryker for JavaScript, PIT for Java, Cosmic Ray for Python. It's still slower than traditional coverage, but it's getting faster. Luna: I want to circle back to the fintech startup I mentioned. After that double-charge bug, they overhauled their testing approach. They adopted branch coverage as their primary metric, added integration tests for the payment flow, and started running mutation tests on their core transaction logic. Lucas: And did it help? Luna: Within six months, they had zero payment-related incidents in production. Their coverage percentage actually went down slightly — because they were measuring branch coverage instead of line coverage — but their defect detection went way up. Lucas: That's the paradox. A lower number that actually means something is worth more than a high number that gives false confidence. Luna: Yeah. And I think that's the core message for today. Don't let your test suite become a liability by chasing the wrong metric. Lucas: Speaking of what's worth it — if today's episode was useful to you and you want to keep it ad-free, buy me a coffee dot com slash fexingo helps. Luna: It really does. A few dollars a month from listeners who find value here is what keeps these conversations going. Lucas: We don't run ads — that's a deliberate choice. If you want to support that, buy me a coffee dot com slash fexingo is the place. Luna: All right. So back to the practical side — what about teams that are already using mutation testing? Are there any emerging approaches beyond that? Lucas: There's something called 'test amplification' — tools that automatically generate new test cases based on existing ones. They analyze how the code changes and suggest tests that cover new branches. It's still early, but promising. Luna: And there's also the whole area of property-based testing, where you define properties that should hold for any input, and the framework generates random inputs to verify them. Lucas: Right. QuickCheck in Haskell, Hypothesis in Python — those tools can uncover edge cases you'd never think to write a test for. They complement coverage metrics really well. Luna: So the future of QA isn't about hitting a coverage number. It's about having a diverse set of techniques that probe the code from different angles. Lucas: Exactly. Line coverage is fine as a floor — but if you're not measuring branch coverage, and you're not doing some form of mutation testing or property-based testing, you're flying blind. Luna: And the cost of flying blind in software is just too high. Bugs erode trust, cost money, and in some cases — like medical devices or autonomous vehicles — can be catastrophic. Lucas: Right. So let's leave listeners with a simple action item. Open your CI pipeline today. Check what coverage metric you're using. If it's line coverage, switch to branch coverage. Then run mutation testing on one critical module this week. See what your tests actually catch. Luna: And when you see the results, you might find that your test suite is less reliable than you thought. But that's a good thing — because now you know where to improve. Lucas: Exactly. Knowledge is the first step. And that's what we're here for.