Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Mutation Testing Reveals Your Tests Are Lying
Transcript
- Lucas: So, Luna, imagine you're testing a banking app's withdrawal function. You write a test that deposits a hundred dollars, withdraws fifty, and asserts the balance equals fifty. The test passes. You're confident. But what if I told you that same test would also pass if you accidentally deleted the line that subtracts the withdrawal? Luna: That's terrifying, but also sounds like the test is just checking the wrong thing. How would you even know? Lucas: That's exactly the problem mutation testing solves. It deliberately introduces small changes—mutations—into your source code, like flipping a less-than to a greater-than, or deleting a line. Then it runs your tests. If a test fails, we call that a 'killed mutant'. If all tests still pass, the mutant survived—and that means your test suite missed a bug that's now sitting in production. Luna: So mutation testing doesn't test your code—it tests your tests. That's a subtle but important flip. Lucas: Exactly. And the numbers are sobering. A 2021 study by Papadakis and colleagues analyzed over five thousand open-source Java projects. They found that on average, about 20 to 30 percent of mutants survived in supposedly well-tested codebases. That means one in four of your passing unit tests might not be testing what you think they're testing. Luna: Twenty to thirty percent is huge. I mean, I've seen code coverage reports at 90 percent and felt great. But that's just line coverage—it doesn't tell you if the test actually validates the behavior. Lucas: Right. Line coverage tells you that line executed. It doesn't tell you the test checked the result. Mutation testing goes deeper. It asks: 'If I break this line, does your test catch it?' If not, that line effectively has no test. Luna: Let me push back a little. Mutation testing sounds expensive. You're generating hundreds or thousands of mutated versions of your code, and running the full test suite against each one. That's gotta be slow. Lucas: It is slow. Traditional mutation testing can take hours for a large project. But tools like PITest for Java and Stryker for JavaScript have gotten smarter. They use techniques like 'selective mutation'—only mutating operators that are most likely to catch real bugs—and 'incremental analysis' to avoid re-running tests on code that hasn't changed. Luna: Okay, so it's not as impractical as I thought. Still, for a team shipping daily, is the trade-off worth it? Lucas: I'd argue it's worth the investment on critical paths—payment processing, authentication, anything where a bug means lost money or trust. One case study I like is from a company called 'Siemens'—they applied mutation testing to their medical device software and found that tests they thought were robust actually missed 15 percent of the injected faults. Luna: Fifteen percent missed faults in medical software. That's the kind of find that justifies the compute cost. Lucas: And you don't have to do it on every build. Many teams run mutation testing on a schedule—maybe nightly or before major releases. Or you can target it at the most changed modules. Luna: So you're setting a mutation score threshold, like 'we want at least 80 percent of mutants killed'? Lucas: Exactly. A mutation score is simply the ratio of killed mutants to total mutants. A score of 85 percent means 15 percent of your bugs would slip through. You can set a bar and enforce it in CI. The tool fails the build if the score drops below, say, 70 percent. Luna: That's a concrete quality gate. I like it. But I wonder—does mutation testing ever give false positives? Like, what if the mutant is functionally equivalent to the original code? Luna: For example, if you have an if-statement that checks 'x > 0' and the mutation flips it to 'x >= 0', but due to the data flow, the behavior is identical—no test can distinguish them. Lucas: That's a known challenge. Tools call those 'equivalent mutants'. They can inflate your mutation score artificially. Some tools try to detect them with compiler optimizations or simple data-flow analysis, but it's an open problem. In practice, teams often accept a small percentage of equivalent mutants and treat the score as an approximation. Luna: So it's not perfect, but it's still a huge improvement over just coverage numbers. Lucas: Absolutely. And I want to be clear—mutation testing isn't a replacement for other testing techniques. It's a complement. You still need integration tests, end to end tests, property-based tests. But mutation testing gives you a unique insight: it tells you where your unit tests are weak. Luna: Weak in the sense of 'this line has a test that passes but doesn't actually validate anything'. Lucas: Right. And that's the silent confidence trick. You think you're covered, but you're not. Mutation testing exposes that. Luna: I'm sold. But let's talk practical adoption. If a team wants to start, where do they begin? Do they need to set up a new tool, or can they integrate into existing frameworks? Lucas: Most mutation testing tools plug into your existing test runner. For Java, PITest integrates with JUnit, Maven, Gradle. For JavaScript, Stryker works with Jest, Mocha, Jasmine. For Python, there's MutPy or Cosmic Ray. You typically add a plugin or a dependency, then run a command like 'mvn org.pitest:pitest-maven:mutationCoverage'. Luna: So it's a one-liner to get started. That's low friction. Lucas: Low friction to run, but the output can be overwhelming. You'll get a report showing which mutants survived and which lines. The key is to prioritize. Don't try to kill every mutant on day one. Focus on the core business logic, the parts where a bug would have the highest impact. Luna: And then over time, you can expand the coverage. I imagine that also helps with the performance issue—you're not mutating the entire codebase at once. Lucas: Exactly. Start small, learn what the tool tells you, and build a culture of test quality. One thing I've seen is teams who start mutation testing often find they have a lot of 'happy path' tests but no edge-case tests. The tool forces you to think about what your test is really asserting. Luna: Speaking of forcing thinking—this whole conversation reminds me why we do this show ad-free. We don't have sponsors pushing us to keep things shallow. If today's tech conversation gave you something usable, and you want to support that choice, you can find us at buy me a coffee dot com slash fexingo. It's a small way to keep the signal strong. Lucas: Yeah, we'd rather you get value from the content than sit through ads for software you don't need. So if this episode clicked for you, that link is buy me a coffee dot com slash fexingo. Back to mutation testing—there's another angle I want to hit. Luna: What's that? Lucas: Mutation testing also reveals something about test design. A common pitfall is over-mocking. If your test mocks every dependency, mutations in the real implementation might not even affect the test because the mocked path is different. So your mutation score can expose that your tests are too isolated to catch real integration bugs. Luna: That's a great point. So mutation testing indirectly encourages better test architecture—fewer mocks, more real interactions. Lucas: Exactly. And it also catches 'test pollution'—where a test passes only because a previous test left some state. If a mutant survives because the test ran in a specific order, that's a red flag. Luna: I want to get into a specific example. Can you walk us through a simple one? Like that banking withdrawal we started with. Lucas: Sure. Let's say you have a function 'withdraw' that checks if balance is sufficient, then subtracts. You write a test: deposit 100, withdraw 50, assert balance equals 50. That test passes. But now, mutation testing generates a mutant that removes the line 'balance -= amount'. The mutant version never subtracts. If your test still passes, the mutant survived—meaning your test didn't actually verify the subtraction happened. Luna: So the test should have checked more than just the final balance. It should have also verified the account's transaction log, or checked that the balance changed by exactly fifty. Lucas: Right. In this case, a better test would read the balance before and after, and assert the difference. Or, if the function returns a receipt, check that receipt. The point is, mutation testing tells you your assertion is too weak. Luna: That's a concrete improvement you can make immediately. I love that. Lucas: And that's why I think mutation testing is one of the most underused techniques in software testing. It shifts your mindset from 'did the test pass?' to 'would the test catch a real mistake?'. Luna: It's a form of humility for testers. You have to accept that your tests might be lying to you. Lucas: Exactly. And once you accept that, you can start fixing them. So, if you're listening, and you've never tried mutation testing, I'd say pick one critical module, run PITest or Stryker on it, and see how many mutants survive. The first time, you might be shocked. But then you get better. Luna: And you'll never look at a green test suite the same way again. Lucas: That's the goal. Thanks for sticking with us on this deep dive.