Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Mutation Testing Catches Bugs Your Tests Miss
Transcript
- Lucas: So you run your test suite, it passes, you see 95 percent code coverage, and you feel good about shipping. But here's the uncomfortable truth: coverage tells you which lines of code executed, not whether your tests actually verified the behavior on those lines. Luna: Right. It's like checking that every room in a house has a smoke detector, but never testing if the detector actually goes off when there's smoke. Lucas: Exactly. And that's where mutation testing comes in. Instead of just measuring coverage, mutation testing deliberately introduces small changes — mutants — into your source code. Then it reruns your tests to see if they detect the change. If a mutant survives, you have a blind spot. Luna: And if today's conversation gave you something usable, honestly, if it was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Keeps the show ad-free and lets us keep digging into these topics. Lucas: Yeah, and we really appreciate that. So let's dig into an example. I pulled an open-source Python library that validates date strings — something like parsing '2024-02-30' and returning whether it's a valid date. Luna: Classic edge case. February 30th doesn't exist. Lucas: Right. The original test suite had 100 percent statement coverage. But when I ran PIT — the mutation testing tool for Java and Python — it generated a mutant that changed a less than or equal comparison to a strict less-than. That mutant survived. Luna: Oof. So the test never verified that boundary condition. February 28th versus 29th versus 30th — that's a classic off-by-one. Lucas: Exactly. The test was asserting that a valid date returns true, but it never checked what happens at the exact boundary. The mutant survived because no test failed when February 30th was incorrectly accepted. And that's the power of mutation testing — it finds holes that coverage metrics don't see. Luna: So how does this work practically? You run mutation testing as part of your CI pipeline? Lucas: You can, but it's computationally expensive. For each mutant, you have to recompile and rerun the relevant tests. A large codebase can generate thousands of mutants, and running them all can take hours. Most teams run mutation testing on a subset — say, the module with the highest churn or most critical logic. Luna: So it's not something you run on every commit. Maybe nightly or before a release. Lucas: Yeah, exactly. And there are tools that make it faster. PIT for Java, Stryker for JavaScript and TypeScript, Cosmic Ray for Python. They all work on the same principle: introduce a mutation, run the tests, report survivors. Luna: What about false positives? I've heard of 'equivalent mutants' — changes that don't actually alter the observable behavior of the program. Lucas: That's the biggest challenge. An equivalent mutant is syntactically different but semantically identical. For example, refactoring 'if ' to 'if ' — the mutant doesn't change behavior, so it should survive. But you don't want to chase those. Tools can sometimes detect them, but it's not perfect. Luna: So when you see a surviving mutant, you have to manually inspect whether it's equivalent or a real gap. Lucas: Right. And that requires domain knowledge. But even with that overhead, mutation testing is one of the most effective ways to assess test quality. Studies show that a mutation score above 80 percent correlates strongly with lower post-release defect density. Luna: And what's a realistic goal? 100 percent mutation coverage seems impossible because of equivalent mutants. Lucas: 80 to 90 percent is a solid target for critical modules. Below 70 percent, you likely have significant gaps. And the nice thing is, once you've written tests that kill all non-equivalent mutants, you have a very robust suite. It's a forcing function for thinking about edge cases. Luna: I've also seen teams use mutation testing to guide test-driven development. Write a test, then run mutation testing to see if the test is strong enough. Lucas: That's a great approach. Write your test first, then run mutation testing on the implementation. If your test doesn't catch a mutant, you know you need to refine the test. It's a feedback loop that builds confidence. Luna: Let's talk about integration with CI. How do you handle the time issue without blocking developers? Lucas: One pattern is to run a fast subset — maybe just the modules changed in the commit. Another is to run mutation testing only on the pull request, and fail the build if the mutation score drops below a threshold. But you need to be careful about flakiness. Luna: Flaky tests cause false positives in mutation testing too, right? Lucas: Absolutely. If a test fails intermittently, it might kill a mutant even though the mutant isn't actually detected. That adds noise. So you want a stable test suite before introducing mutation testing. Otherwise, you'll waste time investigating non-issues. Luna: So the prerequisite is a reliable test suite with minimal flakiness. Lucas: Exactly. Once you have that, mutation testing can dramatically improve test quality. I've seen teams go from 70 percent mutation score to 85 percent in a couple of sprints, and their bug rate dropped noticeably. Luna: What about legacy codebases with no tests? Can mutation testing still help? Lucas: It can, but it's painful. If you have zero coverage, mutation testing will generate thousands of surviving mutants. Better to start by writing tests for the most critical paths, then gradually introduce mutation testing as you improve coverage. Luna: That makes sense. So the takeaway: don't rely on code coverage alone. Use mutation testing to validate that your tests actually test the right things. Lucas: Exactly. Coverage tells you what code ran. Mutation testing tells you what code was verified. And that's a much more useful metric for shipping with confidence. Luna: Great episode. Thanks, Lucas. Lucas: Thanks, Luna. And thanks to our listeners — we'll see you next time.