Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Mutation Testing Reveals Your Tests Are Lying
Transcript
- Lucas: You run your test suite, you see green across the board, you merge. We've all been there. But here's the uncomfortable truth: a passing test suite doesn't mean your code is correct. It means your test suite didn't find a bug — not that one doesn't exist. Luna: Right, and that gap between 'tests pass' and 'code is correct' is exactly what mutation testing tries to measure. Lucas: Exactly. Mutation testing is a technique where you intentionally introduce small changes — mutations — into your source code. Things like flipping a less-than to a greater-than, deleting a line, changing a constant value. Then you run your tests against the mutated code. If a test fails, the mutation is 'killed.' If all tests still pass, the mutation 'survived' — and that means your tests didn't catch the change. That's a gap. Luna: So it's like you're injecting faults on purpose to see if your safety net actually catches them. Lucas: Precisely. And the output is a mutation score: the percentage of mutations your tests killed. A high score means your tests are sensitive to small logic errors. A low score means you have tests that pass for the wrong reasons — maybe they never actually assert the critical behavior, or they test a stub instead of the real logic. Luna: I've seen teams celebrate 90 percent code coverage, only to run mutation testing and find that a third of their mutations survive. That's a sobering moment. Lucas: It is. And it's more common than people think. Let's take a concrete example. Imagine you have a JavaScript function that applies a discount to a cart total. Something like: if the cart total is over one hundred, apply a ten percent discount. You write a test that passes a cart of one hundred fifty dollars and asserts the result is one hundred thirty-five. That test passes. But what if the developer accidentally wrote 'greater than or equal' instead of 'greater than'? You'd never catch it with that one test. A mutation test flips the operator, and if the test still passes, you know your test isn't sensitive enough to catch off-by-one errors. Luna: And that's a really common scenario. Edge cases around boundaries are where bugs live. Lucas: So tools like Stryker for JavaScript, Pitest for Java, and mutmut for Python automate this. They generate hundreds or thousands of mutations, run your test suite against each one, and report which survived. The key insight is that mutation score is a much stronger metric than code coverage. Code coverage tells you what code was executed. Mutation score tells you whether your tests actually verify the behavior. Luna: But it's computationally expensive. Running your test suite once is fast. Running it a thousand times — even with optimizations — can blow up your CI pipeline. Lucas: That is the biggest practical challenge. Stryker and Pitest both use techniques to speed things up: they run tests in parallel, they only run tests that cover the mutated code, they do incremental mutations. But still, a large codebase can take hours. The typical recommendation is to run mutation testing on a subset — the most critical modules, or the code that changes most often. Or to run it on a scheduled basis, not on every single commit. Luna: So it's a trade-off. But when you do find a surviving mutation, what's the typical fix? Lucas: Often it means your test is too weak. Maybe it only checks the happy path. Maybe it mocks out the logic you're trying to test. The fix is to add a more specific assertion. For the discount example, you'd add a test with a cart total of exactly one hundred dollars, and another with ninety-nine ninety-nine, to force the boundary to be tested. Sometimes the fix is to remove a mock and test the real integration. Mutation testing forces you to write better tests. Luna: I've seen teams adopt a rule: no new code is merged unless the mutation score for that module is above 80 percent. That's a high bar, but it catches real bugs. Lucas: Yeah, and it's a bar that changes the way you write tests from the beginning. You stop writing tests that just hit lines and start thinking about what could go wrong. It aligns really well with property-based testing and fuzzing, which we've talked about before. Luna: How does it relate to fuzzing? I see them as complementary but different. Lucas: Fuzzing throws random inputs at your code to find crashes or unexpected behavior. Mutation testing changes the code itself and checks if your tests catch the change. They're both about finding gaps in your verification, but from opposite directions. Fuzzing explores the input space; mutation testing explores the code-space of potential bugs. Together, they give you a much stronger safety net. Luna: So mutation testing is really a test of your tests. Not of your code. Lucas: Exactly. It's a meta-test. And the results can be humbling. I remember a team I worked with that had a critical payment module with 95 percent line coverage. They ran mutation testing and found that 60 percent of the mutations survived. Most of the tests were mocking the payment gateway so heavily that they never actually exercised the logic. They had to rewrite almost every test. Luna: And I bet that effort paid off when they deployed to production and didn't break anything. Lucas: It did. They went from a hotfix a week to zero payment-related incidents in the next quarter. That's the kind of impact mutation testing can have. But the tooling still has room to grow. One issue is equivalent mutants — mutations that are semantically identical to the original code, so they can never be killed. For example, changing 'i++' to '++i' in a loop where the order doesn't matter. Those inflate your score and require manual filtering. Luna: Right, that's a known pain point. Some tools try to detect equivalent mutants automatically, but it's an undecidable problem in general. Lucas: So you have to accept some noise. But even with that, the signal is still incredibly valuable. The industry is moving toward making mutation testing more practical. For example, Stryker now has a 'incremental' mode that only mutates changed files since the last run. That cuts the runtime significantly. And CI services like GitHub Actions have prebuilt actions for Stryker that cache results. Luna: That sounds like it could fit into a real development workflow without grinding everything to a halt. Lucas: It can. And I think we'll see mutation testing become as common as code coverage in the next few years. It's just a better metric. If you care about quality, you should care about whether your tests actually catch bugs, not just whether they run. Luna: And on that note — you know, keeping this show ad-free and diving into topics like this is only possible because some listeners chip in monthly. It's a small group, but it makes a real difference. Lucas: Yeah. If you find these deep dives into software testing useful, and you want to keep them free from ads and sponsors, you can support the show at buy me a coffee dot com slash fexingo. No pressure, just putting it out there. Luna: And we really appreciate those who do. It lets us spend time on these topics instead of chasing ad revenue. Lucas: So back to mutation testing — one more practical tip: if you're trying it for the first time, don't run it on your whole codebase. Pick one module that's critical and has a decent test suite. Run Stryker or Pitest on just that module. Look at the surviving mutants. You'll likely find gaps you didn't expect. Fix those tests, and then decide if the insight is worth scaling up. Luna: That's a good starting point. And if the mutation score is already high, you can have more confidence in that module's tests. Lucas: Exactly. Mutation testing doesn't replace good test design, but it reveals where your design is weak. And that's a superpower.