Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Mutation Testing Exposes Your Weakest Tests
Transcript
- Lucas: Let me ask you something. If your test suite passes with 100% line coverage, does that mean your code is bug-free? Luna: I think most engineers know the answer is no, but it's easy to act like it is. Lucas: Exactly. And that gap between 'my tests ran green' and 'my tests actually catch bugs' is exactly what mutation testing measures. Lucas: Here's the idea: you take your code, you introduce small changes — mutations — that simulate real bugs. Things like flipping a less-than to a greater-than, deleting a line, swapping a variable name. Then you run your test suite. If a test fails, the mutation is 'killed.' If all tests pass, the mutation 'survived' — meaning there's a bug your tests wouldn't catch. Luna: So it's like a simulation of what could go wrong, and you're checking whether your test suite is actually capable of detecting those faults. Lucas: Right. And the result — your mutation score — is the percentage of mutations killed. That number is a much more honest measure of test quality than line coverage. Lucas: I want to make this concrete. Let's say you have a simple function that checks if a user is eligible for a discount: if age is greater than 18 and member is true, return true. You write tests for both cases: age 19 and member true, returns true; age 17 and member false, returns false. That's 100% line coverage. Luna: But what if a developer accidentally changes the AND to an OR? Or flips the greater-than to a less-than? Lucas: Exactly. Your tests would still pass, because you never tested the case where age is 19 but member is false, for example. Mutation testing would flip that AND to an OR, and your test suite would not catch it — the mutation survives. Lucas: There was a really eye-opening study published last year from a major fintech firm. They took 50 production services that had over 80% line coverage and ran mutation testing. The average mutation score was 58%. Meaning 42% of the bugs they simulated would have gone undetected. Luna: That's huge. And I bet those services had been running in production for years. Lucas: Years. With CI pipelines passing every time. The study also found that teams who set a mutation score gate — say, 70% — caught an average of 3 real bugs per quarter that had been live for over six months. Luna: So how do teams actually get started with this? Is it heavy to set up? Lucas: It depends on your stack. For Java, PIT is the go-to tool. You add a plugin to your Maven or Gradle build, run mvn test, and it generates a report. For Python, there's mutmut and cosmic-ray. For JavaScript, Stryker is popular. They all work similarly: they run your tests many times — once per mutation — so it's slow. Lucas: That slowness is the main reason teams resist it. Running mutation testing on a medium-sized service can take hours. But there are strategies: run it on a subset of critical files, or only on changed code in your pull request. Luna: Yeah, because if you run it on the whole codebase every commit, your CI would take forever. Lucas: Right. So the practical approach is to run it on new or modified code in a PR. Tools like PIT have incremental modes now. Stryker has a 'since' flag that only mutates code changed since a given branch point. Lucas: But there's a trap: aiming for 100% mutation coverage. Some mutations are equivalent — they don't actually change the behavior of the code — and no test can kill them. And some are just noise, like changing a string literal that's only used in logging. Luna: So you don't want to chase a perfect score. You want to use it to find weak spots. Lucas: Exactly. The goal is to learn: where do my tests miss? Then you add tests for those specific scenarios. Over time, your mutation score improves, and your test suite becomes genuinely more effective. Luna: I've seen teams adopt a policy of 'no regression in mutation score' — they don't set a hard number, but they make sure new code doesn't lower the score. Lucas: That's a nice lightweight rule. And it avoids the all-or-nothing approach that makes people hate metrics. Lucas: One more thing: mutation testing is not just for unit tests. You can use it on integration tests too, though it's slower. Some tools like PIT support mutating test code itself — which is meta, but useful if you want to check whether your test helper functions are actually correct. Luna: That's almost philosophical. 'Who tests the testers?' Lucas: Yeah. And mutation testing is one answer. It's not a silver bullet, but it's the closest thing we have to a honest assessment of test quality. Lucas: And honestly, if today's episode gave you one actionable idea — like, 'I should try mutation testing on my next PR' — that's probably worth a coffee to you. Luna: Yeah. If it saved you even one bug that would have hit production, that's huge. Lucas: So if it was worth that, the link is buy me a coffee dot com slash fexingo. No pressure. Just a way to keep this ad-free and focused on stuff like this. Luna: And honestly, it helps us know we're on the right track. Lucas: Right. So back to the tools — one thing I want to emphasize: start small. Pick one module, run mutation testing on it, look at the surviving mutations, and write a test for the most obvious one. You'll learn more in that hour than from a month of reading about test coverage. Luna: That reminds me — we had an episode on test prioritization a while back, and mutation testing can actually help with that too. You can prioritize tests that kill the most mutations. Lucas: Yes, exactly. There's research showing that if you sort your tests by how many mutations they kill, you can achieve 90% of your mutation score with only 30% of the test run time. That's a huge CI savings. Luna: So it's not just a quality tool — it's a speed tool too. Lucas: Exactly. And that's the kind of win that gets buy-in from engineering leadership. 'We're going to reduce CI time and catch more bugs' — that's a sentence everyone likes. Luna: What about false positives? Do mutation tools ever flag things that aren't really bugs? Lucas: They do. For example, if you have a constant that's used in a comparison, mutating it might produce a different constant, but the test might still pass because the behavior is the same. Those are equivalent mutations. Tools try to filter them out, but it's not perfect. Lucas: Also, some mutations are just not worth fixing — like changing a method name in a call that's only used by reflection. Your tests might not catch it, but it's not a real bug either. So you have to use judgment. Luna: So it's more of a conversation starter than a hard gate. Lucas: Exactly. It surfaces a list of potential blind spots. Then you decide which ones matter. And over time, your test suite gets genuinely stronger. Lucas: To wrap up: mutation testing is the closest we have to a measure of test effectiveness. It's not perfect, but it's far better than line coverage. Try it on one module this week. You might be surprised at what survives. Luna: And if you do, let us know what you find. We'd love to hear about it.