Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Mutation Testing Found a Critical Bank Bug
Transcript
- Lucas: So there's this story from early 2020. A mid-sized fintech company — I won't name them, but their core banking platform processed a few million transactions a day — rolled out what looked like a minor update to their transaction routing logic. Luna: Minor update — that's almost always the setup for a major incident. Lucas: Exactly. The change was one line in a Java method that handled rollback logic if a transfer failed mid-process. The existing test suite had 87 percent line coverage on that class. Every test passed. But after deployment, any transaction that triggered a specific failure path — say, a timeout from the downstream ledger service — would leave the database in an inconsistent state. Luna: So the tests said everything was fine, but the production system was silently corrupting data? Lucas: Correct. It took them about six hours to catch it, during which roughly twelve thousand transactions had partial state written. The total cost was around nine million dollars in reversals, lost business, and regulatory fines. And here's the kicker: a mutation testing tool would have caught that bug in under two minutes during the build. Luna: Alright, I want to unpack that. Mutation testing — we've mentioned it on earlier episodes but never really drilled into it. So walk me through how it would have flagged this bug. Lucas: Sure. The core idea is simple: mutation testing automatically introduces small, intentional changes — mutants — into your production code. Then it runs your existing test suite against each mutant. If a test fails, the mutant is killed — meaning your tests are sensitive enough to detect that change. If the mutant survives, you have a gap in your tests. Luna: So in this fintech case, what kind of mutant would have been introduced? Lucas: The bug was that the rollback method had a condition that checked for a specific error code, but it was using an OR where it should have used an AND. A mutation tool would flip that operator — change the OR to an AND, or vice versa. That's a standard mutant type called an operator replacement. If the tests still passed after that change, the mutant survives, and you have a red flag. Luna: And in this case, the tests would have passed because none of them actually exercised the full failure path. So the mutant survives, and the developer sees 'oh, my tests aren't catching this logical error'. Lucas: Precisely. And that's the real value of mutation testing. It doesn't just tell you which lines of code are executed — it tells you whether your tests actually verify the behaviour that matters. Line coverage can be 100 percent and still miss a bug because the test might call the method but never assert on the correct outcome. Luna: I remember a study I saw — I think it was from the University of Sheffield — where they compared mutation score to line coverage across several open-source projects. The mutation score was, on average, about 30 percentage points lower. So projects that bragged about 90 percent line coverage often had mutation scores around 60 percent. Lucas: Right. And a 60 percent mutation score means that for 40 percent of the logical changes you could make to the code, your tests don't notice. That's a lot of blind spots. The fintech team's tests had 87 percent line coverage, but their mutation score was probably under 50 percent on that particular method. Luna: So how do you practically get started with mutation testing? I know some teams are intimidated by the runtime — doesn't it take forever to run all those mutants? Lucas: It can, but modern tools have gotten a lot smarter. For Java, there's PIT — stands for PIT Mutation Testing. It uses bytecode manipulation and incremental analysis. On a typical microservice with a few thousand lines of code, a full mutation run might take 10 to 15 minutes. But you can also run it only on changed code in your pull request, which cuts that down to under a minute. Luna: And for other languages? Python, JavaScript? Lucas: There's MutPy for Python, Stryker for JavaScript and TypeScript — Stryker actually supports.NET too. They all work on the same principle: introduce mutants, run tests, report survivors. The key is to integrate it early, ideally in CI, and to set a threshold — say, 80 percent mutation score — that the build must meet. But you have to be careful not to chase a perfect score. Luna: Why not? Wouldn't 100 percent mutation coverage be ideal? Lucas: In theory, but some mutants are equivalent mutants — they produce the same behaviour as the original code, so no test can kill them. And some mutations are in trivial code, like getters and setters, where killing them doesn't improve test quality. The industry consensus is to aim for 80 to 85 percent mutation score for critical code paths, and to treat surviving mutants as a signal to investigate, not a failure by itself. Luna: So it's more of a heuristic than a hard pass-fail gate. Lucas: Exactly. And the biggest win is that it forces developers to write better assertions. I've seen teams who started using mutation testing end up with more meaningful unit tests — fewer tests that just call a method and check it doesn't throw an exception, and more tests that validate specific outputs under specific conditions. Luna: Let's go back to that fintech case for a second. If they had mutation testing in their CI pipeline, what would the workflow have looked like? The developer makes the change, pushes, the build runs — and then what? Lucas: The mutation tool would have generated, say, 50 mutants for that method. Forty-nine of them would have been killed by the existing tests. But the one that flipped the logical operator — that mutant survives. The tool highlights it in the report. The developer sees 'survived mutant: changed OR to AND in line 47'. That's a huge clue that the test doesn't actually verify the logic of that condition. Luna: So the developer adds a test that specifically triggers the failure path and asserts the correct rollback state. Then they rerun the mutation tests, and that mutant is now killed. The build passes. Lucas: Right. And that new test would have caught the bug before it ever reached production. The whole cycle adds maybe 15 minutes to the developer's workflow, but it prevents a nine-million-dollar incident. The return on investment is absurdly high. Luna: It's one of those practices that sounds expensive upfront but actually pays for itself on the first major bug it catches. Lucas: Absolutely. And it's not just for financial software. Any system where correctness matters — medical devices, autonomous vehicles, e-commerce checkout flows — mutation testing can catch the kinds of logical errors that traditional testing routinely misses. I've seen it find bugs in string formatting, date calculations, even cache invalidation logic. Luna: Cache invalidation — that's famously one of the two hard things in computer science. Lucas: Along with naming things and off-by-one errors. And mutation testing catches off-by-one errors beautifully, because a mutant that changes 'less than' to 'less than or equal' will survive if your test data doesn't hit the boundary condition. Luna: Alright, so let's say a team is convinced. What's the first step? Should they run mutation testing on their entire codebase overnight? Lucas: No, that's a recipe for frustration. Start with one module — ideally one that's had bugs in the past. Run the mutation tests on that module, look at the survivors, and write tests to kill them. That builds confidence and familiarity. Then gradually expand to more modules. Also, make sure you set a time limit — most tools let you cap the run at, say, 10 minutes, so it doesn't blow up your CI pipeline. Luna: And what about test quality? Do you need high-quality tests first for mutation testing to be useful? Or does it work even with mediocre tests? Lucas: It works with mediocre tests — in fact, that's where it adds the most value. If your tests are already strong, the mutation score will be high, and you won't get much actionable information. But if your tests are weak, the mutation score will be low, and every surviving mutant is a clear, specific opportunity to improve. It's a diagnostic tool. Luna: So it's almost like a test for your tests. You write tests for your code, then mutation testing tests whether those tests are good enough. Lucas: That's exactly the framing I use. It's meta-testing. And it's one of the few techniques that directly measures test quality rather than just quantity. Line coverage tells you how much code you touched. Mutation testing tells you how well you verified it. Luna: There is a practical challenge though: mutation testing can generate a huge number of mutants — sometimes thousands — and reviewing all survivors can be overwhelming. How do teams handle that? Lucas: Most tools let you filter by mutant type. You can focus on operator mutations, which tend to be the most revealing, and ignore trivial ones like constant changes in string literals. Also, you can set a threshold — if your mutation score is above, say, 80 percent, the build passes. You only need to investigate the survivors if the score drops below that threshold. And over time, as your tests improve, the number of survivors shrinks dramatically. Luna: So it's a practice that compounds. The more you do it, the better your tests get, and the faster future runs become. Lucas: Exactly. And it changes how developers think about testing. I've had engineers tell me that after using mutation testing for a month, they started writing tests with the mutation tool in mind — they'd think 'would this test kill a mutant that flips this condition?' And that mental shift is worth more than any single tool. Luna: I think that's the real takeaway here. It's not just about adding a tool to your CI pipeline. It's about developing a deeper sense of what makes a test valuable. Lucas: Yeah. And if today's deep dive into mutation testing gave you something concrete to apply — a new technique, a new way to evaluate your tests, maybe even a specific tool to try — that's exactly the kind of resource we aim to provide with this show. It stays ad-free because of listener support, so if you found value, buy me a coffee dot com slash fexingo helps keep these conversations going. Luna: It's a small way to say 'this was worth my time' — and it genuinely makes a difference in keeping the show independent. Lucas: So back to the practical side: one thing I want to emphasize is that mutation testing doesn't replace your existing test practices. It complements them. You still need unit tests, integration tests, end to end tests. But mutation testing adds a layer of scrutiny that those other techniques don't provide. Luna: And it does it automatically, which is the beauty of it. Once set up, it runs in the background and flags problems without requiring manual review of every test. Lucas: Right. The same way a linter catches formatting issues automatically, mutation testing catches logical gaps automatically. It's a force multiplier for quality assurance. Luna: So for anyone looking to level up their testing game, mutation testing should definitely be on the radar. Start small, focus on critical modules, and let the survivors guide your test improvements. Lucas: Exactly. And remember that fintech story: a one-line change, 87 percent line coverage, all tests passing, and a nine-million-dollar bug. Mutation testing would have turned that into a two-minute feedback loop. That's the kind of edge it gives you. Luna: Alright, I think we've given everyone plenty to think about. Thanks for joining us on Software Testing with Fexingo. Lucas: See you next time.