Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Needs Mutation Testing
Transcript
- Lucas: Let me start with a question. You've got a test suite with ninety percent line coverage. All green. You ship to production. And then a bug slips through — something obvious, like a payment gets rounded incorrectly. How does that happen? Luna: Everyone nods along, right? 'Great coverage.' But coverage only tells you which lines the test executed. It doesn't tell you whether the test actually checked anything meaningful. Lucas: Exactly. And that's the gap we're looking at today — why your test suite might be lying to you, and how mutation testing exposes the lie. I've been digging into this technique that injects faults into your code on purpose to see if your tests catch them. Luna: So it's like a controlled experiment. You break the code, run the tests, and if the tests still pass, you know you have a blind spot. Lucas: Right. Let me give you a concrete example. Imagine a payment function that calculates the total after tax. The code is simple: multiply price by quantity, add tax, round to two decimals. Standard stuff. Your unit test calls it with a few values — 10 dollars, two items, 8 percent tax — asserts the result is 21.60. That test passes. Luna: And coverage says you hit every line. The multiplication, the addition, the round call. Lucas: Now mutation testing comes in. It creates a mutant — say, it removes the rounding step entirely. Or it changes the multiplication to addition. If your test still passes after that change, the mutant survives. That's a red flag. Luna: So in our example, if the mutant removes rounding, does the test catch it? If the input values happen to produce a nice even number, the test might pass either way. Lucas: Exactly. And that's the real-world scenario. In a payment system, rounding errors can compound. A few cents per transaction, times millions of transactions — that's real money. Mutation testing forces you to write tests that actually verify the behavior you care about, not just that the code ran. Luna: I remember a story from my last startup. We had a module that calculated subscription fees. All tests green, coverage in the high nineties. Then a customer complained about being overcharged by 5 cents. We traced it to a rounding function that truncated instead of rounding half-up. The test asserted the truncated result — so the test was technically correct, but the specification was wrong. Lucas: That's a classic: the test matches the buggy implementation. Mutation testing would catch that because if you mutate the rounding logic, the test might still pass for the wrong reason. It exposes what we call 'coincidental correctness.' Luna: So how do teams actually adopt this without blowing up their CI pipeline? I've heard mutation testing can be slow — like orders of magnitude slower than running the test suite once. Lucas: It can be. The naive approach — generate a mutant for every line, run the full test suite for each — takes forever. But there are practical strategies. First, start small. Pick one critical module, like payment processing or authentication. Run mutation testing only on that module. Luna: And you'd want to use a modern tool like PIT for Java or Stryker for JavaScript. They have incremental modes that skip mutants that are mathematically equivalent to the original code. Lucas: Right. Equivalent mutants are a headache — they produce the same output for all inputs, so killing them is impossible. Good tools filter those out. Also, you can set a mutation score threshold, say 80 percent, and fail the build only if you drop below that. Not a hard gate on day one. Luna: And you mentioned false positives — some mutants are technically killable but not worth the effort. Like changing a string literal in a log message. Who cares? Lucas: That's where you configure exclusions. Most tools let you ignore mutants in logging statements, getters, setters, or trivial boilerplate. The goal isn't perfection — it's finding the real gaps in your test logic. Luna: I think the cultural shift is the hardest part. Developers are used to coverage as a vanity metric. Mutation testing feels like someone deliberately breaking your code and blaming you for not catching it. Lucas: That's a nice way to put it. But the mindset should be: this helps us find the weak spots before customers do. It's a debugging tool, not a performance review. I've seen teams that run mutation testing as part of code review — the reviewer runs it on the new module and checks that surviving mutants are discussed. Luna: So it becomes a conversation starter. 'Hey, why did this mutant survive? Is the assertion too loose? Should we add an edge case?' That's much more productive than arguing over coverage percentages. Lucas: Exactly. And tools are getting faster. With incremental analysis and parallel execution, you can run mutation testing on a module in under a minute. Some teams even run it nightly on the full codebase, then triage the results the next morning. Luna: If today's tech conversation gave you something usable, we'd love it if you supported the show. We deliberately keep these episodes free of ads — just the content, no sponsors. If that model resonates, you can find us at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small way to keep the conversation honest. And it directly supports the research time that goes into episodes like this one. Luna: So back to mutation testing — one pattern I've seen work well is to combine it with property-based testing. You define properties that should hold — like 'the total is always positive' or 'the discount never exceeds 30 percent' — and let the tool generate random inputs. Lucas: That's a powerful combo. Property-based testing finds edge cases you didn't think of. Mutation testing tells you whether your test actually checks those properties. They reinforce each other. Luna: And then there's the question of cost. If you're in a regulated industry like banking or medical devices, the cost of missing a bug is enormous. Mutation testing becomes a no-brainer. Lucas: Absolutely. In those contexts, a few extra minutes of CI time is trivial compared to a compliance failure. I've seen financial institutions require mutation testing for any code that touches customer money. Luna: What about false positives from infrastructure — like flaky tests that cause mutants to be marked as killed when they shouldn't be? Lucas: That's a real problem. If your test suite has flaky tests, mutation testing will give you noisy results. Best practice: stabilize your test suite first — fix flaky tests, make them deterministic — before introducing mutation testing. Otherwise you're just amplifying the noise. Luna: So the order matters. First, get reliable tests. Second, add coverage. Third, introduce mutation testing to validate the tests themselves. Lucas: That's a solid progression. And the nice thing about mutation testing is that it gives you a single number — the mutation score — that is actually meaningful. Unlike coverage, which can be gamed by writing useless tests, the mutation score forces you to write meaningful assertions. Luna: But it's not a silver bullet. Mutation testing won't catch missing features or integration bugs between modules. Lucas: No tool is. But it fills a specific gap: it tells you when your tests are too weak to detect real faults. And that's a gap that most teams don't even know they have. So the next time you see a hundred percent coverage and a bug in production, ask yourself: would mutation testing have caught it? Luna: I'm going to try it on that subscription module I mentioned earlier. I bet the mutation score is lower than I think. Lucas: Let me know what you find. And if listeners try it, I'd love to hear about their first surviving mutant. It's a humbling experience.