Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How A-B Testing Your Tests Prevents False Confidence
Transcript
- Lucas: A few weeks ago I was talking to a QA lead at a fintech that processes about forty million dollars in transaction volume a month. They had twelve thousand automated tests, they ran them every commit, and they genuinely believed their pipeline was rock solid. Then they ran a simple experiment and discovered their test suite was missing nearly a quarter of real defects. Luna: A quarter? That's terrifying. How do you even measure that without already knowing what the bugs are? Lucas: You inject them. They used what's sometimes called 'mutation testing lite' — they intentionally seeded defects into their codebase, things like off-by-one errors, swapped boolean conditions, missing null checks. The kind of bugs that actually slip into production. Then they ran their suite and tracked how many it caught. Luna: So basically an A-B test for your test suite. One version of the code with the bug, one without, and you check if your tests can tell the difference. Lucas: Exactly. And the results were sobering. Their existing suite caught only seventy-seven percent of the seeded defects. That means twenty-three percent sailed through green. The team had been shipping with full confidence, but their tests were giving them a false all-clear. Luna: What was the root cause? Were the tests themselves poorly written, or was it more about what they weren't covering? Lucas: Both, but the bigger issue was something called 'oracle decay.' Over time, as the codebase evolved, the expected outputs embedded in their test assertions had drifted out of sync. A test that once verified the right behavior now passed for the wrong reasons — because the assertion had become too loose or was checking a field that no longer mattered. Luna: So the tests were still green, but they were testing the wrong thing. I've seen that happen in every team I've worked with. It's insidious because nobody questions a passing test. Lucas: Right. The fintech actually built a small experiment framework around this. They set up a canary deployment that ran two parallel test suites — their standard one and an augmented version with what they called 'oracle rejuvenation' checks. Every two weeks, they'd bake in a fresh set of seeded defects and measure the detection delta. Luna: And I'm guessing over time that delta told them exactly which parts of their test suite had gone stale. Lucas: Yeah, they could pinpoint it to specific modules. The payments pipeline module, for example, had a detection rate that dropped from ninety-two percent to sixty-eight percent over three months. The team had been refactoring that module aggressively but hadn't updated the corresponding tests. The experiment surfaced it immediately. Luna: That's the kind of thing most teams only discover after a production incident. I love that they made it a proactive measure. Lucas: And the cost of running this kind of A-B test is surprisingly low. For their setup, it added maybe fifteen minutes to the CI pipeline. They used a small sidecar container that injected the mutations at the bytecode level, so it didn't require any changes to the application source or the test code itself. Luna: Did they ever find a bug that was actually live in production? I mean, not a seeded one, but a real defect that the A-B test helped uncover. Lucas: Yes, about six weeks in, the oracle rejuvenation checks flagged a module where the expected output for a currency conversion rounding rule was outdated. The assertion was still checking for four decimal places, but the business had moved to five. The test passed because both the code and the test were wrong in the same way. The A-B test caught it because they seeded a different rounding error and the test didn't catch that either — which told them the assertion was broken. Luna: So the test was silently validating the old behavior, and the code had been updated to match the old test. That's a classic coupling problem. The test oracle and the implementation were coupled to the same stale requirement. Lucas: Exactly. And that's why simply adding more tests doesn't fix it. You need a mechanism to independently verify that your tests still test the right thing. The A-B framework provides that independent signal. Luna: I want to talk about the practical side. If our listeners want to try this themselves, where do they start? Lucas: Start small. Pick one module — the one you're least confident about, or the one with the highest churn. Use an open-source mutation testing tool like PIT for Java, or Stryker for JavaScript. Run it on a weekend, see your baseline mutation score. Then set a goal to improve it by ten percentage points over the next sprint. Luna: And don't chase a hundred percent, right? Because some mutations are equivalent to the original behavior — they can't be detected by any test. Lucas: Right. Equivalent mutations are a real thing. The goal isn't perfection, it's visibility. You want to know your blind spots. And you want a process that tells you when those blind spots are growing. Luna: Before we go deeper, I want to point out — conversations like this are exactly why we keep the show ad-free and focused on practical insights. A couple of dollars a month from listeners genuinely makes that possible. If you've gotten something useful out of today's episode, you can support the show at buy me a coffee dot com slash fexingo. It's a small thing that adds up. Lucas: Yeah, completely agree. It keeps us independent and lets us tackle topics like this without worrying about sponsors. Appreciate everyone who chips in. Luna: So back to the A-B framework — one thing I found really clever in their approach was they didn't just measure pass/fail. They also tracked the time it took for a test to detect a mutation. Some tests caught bugs in under a second, others took nearly a minute. Lucas: That's a great point. Detection latency matters because in a CI pipeline, slow tests get skipped or parallelized differently. If your fastest tests are also your most reliable, you want to know that. The fintech discovered that the slowest tests in their suite were also the ones with the lowest mutation detection rates — they were slow because they were doing too much setup and not enough actual assertion. Luna: So they were both slow and ineffective. That's a double hit. Did they refactor those tests or replace them? Lucas: They rewrote about a third of them, focusing on making each test assert exactly one behavior. The mutation score for that module jumped from sixty-eight percent to eighty-nine percent, and the total execution time actually dropped because the new tests didn't have all that heavy setup. Luna: That's a case where A-B testing your tests pays for itself in CI minutes alone. Lucas: Exactly. And the cultural shift was maybe the biggest win. The team started treating their test suite as something that needed to be validated, not just relied on. They scheduled a monthly 'test health review' where they looked at mutation scores, detection rates, and oracle freshness. Luna: That's the kind of discipline that separates teams who just have tests from teams who have trustworthy tests. Lucas: Right. And it's not about blaming anyone. The fintech lead told me his team had been working hard, writing tests, following best practices. The problem was systemic — they had no mechanism to detect when their tests were lying to them. The A-B framework gave them that mechanism. Luna: I think the key takeaway for our listeners is: don't assume your tests are telling the truth just because they're green. Build a way to check. It doesn't have to be complex — a canary, a mutation run, a periodic oracle audit. But build it. Lucas: Yeah. And if you're not sure where to start, pick one test, one assertion, and ask yourself: if this assertion were wrong, would I know? That question alone changes how you look at your suite. Luna: Great note to end on. Lucas, as always, thank you for the deep dive. Lucas: Thanks, Luna. And thanks to our listeners for spending time with us. We'll be back next episode with another angle on reliable software.