Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Coverage Alone Fails Your QA Strategy
Transcript
- Lucas: You know that warm feeling you get when your CI pipeline shows 95 percent line coverage and every test passes? I think that feeling is actually dangerous. Luna: Dangerous because it's fake confidence? I've definitely shipped code thinking 'we're covered' only to get a ticket two weeks later about something we missed. Lucas: Exactly. And it's not just you. I want to talk about why test coverage — the metric we all use to feel good — is often the wrong thing to optimize for. And I have a concrete case. Luna: Okay, I'm listening. What happened? Lucas: Last year, a mid-size fintech company — let's call them PayFlow — shipped a payment calculation update. Their test suite had 92 percent line coverage. All green. Two weeks later, they found out that a specific edge case in rounding was miscalculating transaction fees on a subset of international payments. Luna: So 92 percent coverage and they still had a rounding bug that cost them money. How did that happen? Lucas: The coverage was real — those lines were executed. But the tests never actually verified the arithmetic. They checked that the function ran without crashing and returned a number. They didn't check that the number was correct for the specific rounding rule. Luna: Right, so the tests covered the code path but not the behavior. That's a classic distinction that gets lost in the coverage number. Lucas: It's the difference between 'we executed this line' and 'we proved this line does the right thing under all relevant conditions.' And once you start looking for it, you see this everywhere. Luna: I remember reading about a study — I think it was from Google or maybe Microsoft — that found no correlation between high line coverage and low defect density in production. Lucas: There's a well-known paper from the University of Texas that showed exactly that. They analyzed five large open-source projects and found that coverage percentages didn't predict bug-finding effectiveness. A test suite with 70 percent coverage that focuses on risky paths can outperform a 95 percent coverage suite that tests trivial getters and setters. Luna: So how do you know if your coverage is meaningful? I mean, every team I've been on sets a coverage target. Usually 80 percent. Sometimes it goes up to 90. Lucas: And that target instantly becomes a ceiling. Developers write tests that bump the number without actually testing risky logic. They add a test that calls a getter and asserts it returns the value they just set — that's coverage, but it's not value. Luna: Right, you're describing gaming the metric. I've done it. Everyone has. Lucas: Of course. And the team lead sees green and thinks 'we're safe.' So the real question is: what should you measure instead? Or at least, what should you measure alongside coverage? Luna: I've heard of mutation testing — where you introduce small faults into the code and see if the tests catch them. Does that actually work? Lucas: It works, but it's computationally expensive. The idea is that if your tests survive a mutation — meaning they don't fail when the code is subtly broken — then you have a test that doesn't actually verify the behavior. Mutation testing gives you a 'test effectiveness score' that's much closer to the truth than line coverage. Luna: But is it practical for a team that's already struggling to maintain their existing test suite? It sounds like it could double the CI time. Lucas: It can. For PayFlow, they ran mutation testing only on the modules that handled financial calculations — maybe 15 percent of their codebase. That was enough to catch the rounding bug before it went live. So you don't have to do it everywhere. You target the high-risk areas. Luna: So a risk-based approach. That makes sense. But let's go back to the coverage metric itself. What would you tell a team that's trying to set a coverage policy? Lucas: I'd say: stop treating coverage as a gate. Don't block a merge because coverage dropped from 82 to 81. Instead, look at uncovered lines and ask 'does this line matter?' If it's a logging statement or a trivial return, fine. If it's a branch in a pricing algorithm, that's a red flag. Luna: So it's about visibility, not enforcement. You want to see where the gaps are, not just the aggregate number. Lucas: Exactly. And one thing I've seen work well is a 'coverage review' during code review — not automated, but human. The reviewer looks at the diff and asks 'are there any paths in this change that aren't tested?' That catches way more than a coverage bot. Luna: That only scales if you have reviewers who are disciplined. But I think it's a good practice. Let me ask about another angle: does the type of coverage matter? Like branch coverage versus line coverage? Lucas: Huge difference. Line coverage just tells you if a line executed. Branch coverage tells you if both the true and false paths of a conditional were executed. Branch coverage is stricter and more meaningful. Most teams track line coverage because it's easier, but branch coverage is a better signal. Luna: So if I'm an engineering manager and I want to improve my test suite's effectiveness, what's the first thing I should do? Lucas: Switch your CI dashboard to show branch coverage instead of line coverage. Then run a mutation test on the module with the highest bug rate historically. I guarantee you'll find at least one test that passes but shouldn't. Luna: I like that it's specific. Not 'improve quality' but 'change the metric and run one targeted experiment.' Lucas: Right. And then share the results with the team. When they see that a test they thought was solid actually isn't testing anything, the conversation shifts from 'how do we hit 90 percent' to 'how do we make our tests actually catch bugs.' Luna: That shift in conversation is the real win. I wonder how many teams out there are still proudly displaying their 95 percent line coverage badge and have no idea they're leaking defects. Lucas: A lot. And the irony is that the teams with lower coverage but better test design often ship fewer bugs. Because they focus on the risky paths. Luna: So coverage as a vanity metric is almost worse than no metric, because it gives false comfort. Lucas: That's the argument. And look, I'm not saying ignore coverage entirely. I'm saying don't worship it. Use it as one signal among many, and always pair it with a qualitative review of what the tests actually check. Luna: I think a lot of engineers have experienced the PayFlow scenario — the bug that coverage didn't catch. But they don't always have the vocabulary to articulate why it happened. So this conversation is useful. Lucas: Honestly, if today's conversation gave you something you can take back to your team — maybe you're that manager who's going to switch to branch coverage on Monday — and it was worth the time, that's exactly the kind of thing listener support keeps this show ad-free and focused. If you've ever felt like the show helps you think more clearly about this stuff, you can support us at buy me a coffee dot com slash fexingo. Luna: Yeah, and it's genuinely a small thing that makes a big difference for us. Keeps us independent and able to pick topics like this without any sponsor pressure. Lucas: Alright, back to the practical side. Let me give you a framework I've used with teams to audit their test suite's effectiveness without running mutation testing everywhere. Luna: Good, I want to hear this. Lucas: Pick one module — preferably one that's had a production bug in the last six months. List every test in that module. For each test, write down one sentence describing what behavior it verifies. If you can't write that sentence, the test might not be testing anything meaningful. Luna: So you're basically doing a test inventory with a behavioral annotation. That sounds time-consuming but valuable. Lucas: It's a one-time exercise, maybe half a day for a medium-size module. What you'll likely find is that 20 percent of the tests cover 80 percent of the risky behavior. The rest are either redundant or trivial. Luna: And then you can delete or refactor the trivial ones, and add tests for the uncovered risky paths. Lucas: Exactly. And the team walks away with a much clearer picture of what their test suite actually does. The coverage number might even drop after you delete useless tests, but the suite becomes more effective. Luna: I love that. It's counterintuitive — you're making the number go down to make quality go up. Lucas: That's the kind of metric inversion that good engineering teams embrace. The goal isn't a high number. The goal is catching bugs before customers do. Luna: And that's really the bottom line. So if someone's listening and they're responsible for a test suite, what's the one thing you'd want them to do this week? Lucas: Do the inventory exercise on the module that scares you the most. The one that keeps you up at night. I promise you'll learn something. Luna: I think I'll do that on our own test suite this week. Even if it's painful. Lucas: Let me know what you find. And for everyone else, if you try it, drop us a note — we'd love to hear what uncovered paths you discovered. Luna: That's it for today. We'll be back next time with another angle on making software testing actually work.