Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Mutation Testing Finds Bugs Your Tests Miss
Transcript
- Lucas: Say you have a function that calculates a discount on an invoice. Your test suite covers every line, every branch — 100 percent coverage by every metric. And then someone accidentally changes a plus sign to a minus sign in production. Your tests pass. Your customer gets a negative total. This exact scenario is why mutation testing exists. Luna: Wait — mutation testing. That's where you intentionally break your code to see if your tests catch the break, right? Lucas: Exactly. You introduce small syntactic changes — mutants — into your source code, run your tests, and see how many mutants survive. A surviving mutant means your test suite didn't detect the change, which is a red flag that your tests are not as thorough as you think. It's like a fire drill for your test suite. Lucas: And if today's fire drill gave you something useful, here's the thing: this show stays ad-free because listeners who find value in it choose to support it. If you got something out of this conversation, you can keep it going at buy me a coffee dot com slash fexingo. No pressure — just a simple way to make sure conversations like this keep happening. Luna: Yeah, it's a small gesture that makes a big difference for the show. So, back to mutation testing — how does it actually work in practice? Lucas: Let's walk through a concrete example. You're testing a function that applies a ten percent sales tax. The code is: 'return price * 1.1'. Your test passes with 'price = 100' and expects 110. Line coverage is 100 percent. Now a mutation tool changes the '1.1' to '1.0' — a tiny mutation. If your test still passes, that mutant survived. It means your test didn't verify that the tax rate is actually ten percent — it only checked that the function returns something vaguely proportional. Luna: So a surviving mutant tells you exactly where your test lacks depth. It's not just about covering lines — it's about covering the logic. Lucas: Right. Traditional coverage metrics measure whether a line was executed. Mutation testing measures whether a line was meaningfully exercised. There's a big difference. And it's not a theoretical difference — it finds real bugs. I talked to a team at a payments company that had 95 percent line coverage on their transaction processing module. They ran mutation testing and found that twelve percent of mutants survived. Two of those survivors corresponded to actual bugs in production that had been causing rounding errors for months. Luna: That's a powerful story. But I imagine running mutation testing on a large codebase is slow — you're essentially compiling and testing many versions of the code. Lucas: It can be. Mutation testing is computationally expensive. But there are practical strategies. First, you don't run it on every commit for the whole codebase. You run it on changed files or on critical modules. Tools like PIT for Java, Stryker Mutator for JavaScript, and MutMut for Python have built-in optimizations like selective mutation — they only generate mutants for lines that are actually covered by tests. That cuts the runtime significantly. Luna: And you can set a mutation score threshold for your CI pipeline — fail the build if the score drops below, say, 80 percent. Lucas: Exactly. And you can incrementally increase that threshold over time. Start with a baseline, run mutation testing periodically — maybe once a week — and focus on improving the score for the modules that matter most. The key is not to treat mutation score like a pass-fail gate on every commit, but as a trend you track over time. Luna: I've seen teams combine mutation testing with code coverage in their dashboards. They show line coverage in green but overlay a mutation score heat map so you can see which parts of the code are actually well-tested. Lucas: That's a smart visualization. It makes the gap tangible. I think the real value of mutation testing is that it forces you to think about test quality instead of test quantity. You stop writing tests just to hit a coverage number and start writing tests that actually challenge the code. Luna: So for someone new to mutation testing, what's the first step? Lucas: Pick one critical module — something with business logic, not just CRUD. Run a mutation testing tool on it and look at the surviving mutants. Read each one and ask: 'Why didn't my test catch this?' Then write one additional test per surviving mutant that targets that specific behavior. You'll learn more about your code and your tests in an afternoon than you would from a month of chasing coverage percentages. Luna: I like that. It's a surgical approach. And it scales — once you've done it for one module, you have a template for the rest. Lucas: Exactly. And over time, your mutation score becomes a leading indicator of test suite health. If you see a sudden drop, it might mean a new feature shipped without proper tests. It's a much earlier warning sign than waiting for a production incident. Luna: Mutation testing sounds like one of those practices that seems overkill until you actually try it and realize what you've been missing. Lucas: I think that's the right take. It's not for every project on day one. But for any codebase where correctness matters — payments, healthcare, autonomous systems — it's becoming table stakes. And for teams that adopt it, the question quickly shifts from 'Should we do mutation testing?' to 'How did we ever trust our tests without it?' Luna: Alright, Lucas. I think we just convinced a few people to fire up Stryker this afternoon. Lucas: I hope so. And if you try it, let us know what surviving mutants you find — we'd love to hear the stories.