Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Differential Testing Catches Regressions Without Tests
Transcript
- Lucas: Let me start with a problem. Say you've got a core library — maybe a JPEG encoder — and you rewrite it for performance. You know the old version works. But how do you prove the new version produces the same output for every valid image? Writing a test case for every pixel pattern is impossible. Luna: Right, because the space of possible inputs is huge. And you can't just compare to a golden image for every case. Lucas: Exactly. This is where differential testing comes in — sometimes called diff testing or, in academic circles, metamorphic testing. The idea is simple: you take a random or fuzzed input, feed it to both the old and new versions of the same function, and compare the outputs. If they differ, you've found a regression — no hand-written assertions needed. Luna: I've seen this used in compilers. The LLVM project runs a tool called 'Cargo-diff' that tests Rust compiler optimizations. But for a JPEG encoder, how do you know what 'same output' means? Binary identical? Lucas: That's the nuance. You define a 'metamorphic relation' — a property that should hold between old and new versions. For a lossless encoder, the output should be bit for bit identical. For a lossy encoder, maybe you compare pixel-by-pixel difference thresholds. Or you re-encode the output and check that the second encode is similar to the first. Lucas: The beauty is you don't need to write test cases. You just need a way to generate valid inputs and a comparator. Tools like AFL or LibFuzzer can generate random inputs that respect the input format — say, valid JPEG headers. Then you run both versions and diff the results. Luna: Okay, but generating valid JPEG images randomly isn't trivial. And if your comparator flags a difference, how do you know it's a real bug versus a legitimate change, like an optimization that reorders arithmetic? Lucas: That's the biggest challenge: flaky diffs. If you optimize a function, the output might be mathematically equivalent but numerically different due to floating-point non-associativity. You have to design your comparator to tolerate those differences — say, comparing within an epsilon, or using a robust hash that accounts for permutations. Lucas: But when it works, it catches bugs that no unit test would. There's a famous case: the Mozilla project used differential testing on their JPEG decoder in Firefox and found a whole class of crashes that only happened on certain malformed images. Their test suite had passed, but diff testing found edge cases. Luna: So it's especially valuable for 'oracle-less' problems — where you can't easily say what the right answer is, but you know two implementations should agree. Like comparing two different sorting algorithms: they should produce the same sorted list. Lucas: Right. Or comparing a database query optimizer — the old query plan and the new plan should return the same rows. Or a math library: sin from old and new versions should match within tolerance. The applications are everywhere. Luna: I remember a team at Stripe used diff testing to validate their idempotency key logic. They ran the same request through two versions of their API and checked that the response and side effects were identical. Found a bug where a race condition caused different error codes. Lucas: That's a great example. And it shows that diff testing isn't just for low-level code. You can apply it to any system where you have two versions that should behave identically — even at the integration level. Luna: But let's talk about cost. Running thousands of random inputs through two versions and comparing output — that's CPU time. Especially if the function is expensive, like a video encoder. Is it practical for CI? Lucas: You don't run it on every commit. You run it as a nightly or weekly job, or on a dedicated fuzzing cluster. The key is to minimize the input set. You can use coverage-guided fuzzing — like AFL — to find inputs that exercise new code paths, so you're not wasting cycles on duplicate paths. Lucas: And you can prioritize. Start with a small corpus of known edge cases. Then incrementally add random mutations. Over time, you build a library of inputs that have historically triggered differences — and you keep re-testing those. Luna: Honestly, this episode has given me a concrete technique I want to try. If today's conversation was worth a coffee to you, you know the deal — buy me a coffee dot com slash fexingo. Keeps the show ad-free and helps us dive into more of these practical testing topics. Lucas: Yeah, listener support is what lets us stay independent and focus on the techniques that actually matter. No sponsor pressure, no fluff. Luna: And speaking of practical — let's talk about setting up your first diff test. Where do you start, Lucas? Lucas: Start with a core function that has a clear metamorphic relation. For example, a serialization library: serialize an object, deserialize it, and resealize — the second output should match the first. Write a script that generates random objects, runs the old code and new code, and diffs the serialized bytes. Lucas: Then iterate. When you find a difference, inspect it. If it's a true bug, write a unit test for that specific input. If it's a tolerable difference, update your comparator. Over a few weeks, you'll have a differential test suite that catches regressions your unit tests never would. Luna: And tools? I know of 'cargo-diff' for Rust, but what about other ecosystems? Lucas: For Python, there's 'hypothesis' — a property-based testing library — which has a 'diff' strategy. For Java, there's 'DiffTest' from the Google testing-on-the-toilet series. And for any language, you can hack together a simple script with AFL, a custom comparator, and a shell loop. Luna: One risk I see: if your comparator is too loose, you'll miss bugs. Too tight, you'll drown in false positives. How do you tune it? Lucas: Start strict, then relax based on evidence. For a lossless encoder, start with bitwise equality. When you find a false positive — say, a timestamp embedded in metadata — exclude that field from comparison. Document the exclusions. Over time, your comparator becomes a formal specification of what 'equivalent output' means. Lucas: And remember, diff testing complements, not replaces, other testing. If you have a unit test that checks a specific transformation, keep it. Diff testing catches the things you didn't think to test. Luna: So for a team with legacy code and no test coverage, this could be a quick win. Run old vs. new after a refactor and see if anything breaks. Lucas: Exactly. I've seen teams adopt diff testing as a safety net during major rewrites. They set up a pipeline that runs both versions in parallel and alerts on differences. It gives them confidence to refactor boldly. Luna: Alright, let's get technical for a moment. What about non-determinism? If your code uses randomness or concurrency, two runs of the same version might produce different outputs. How do you handle that? Lucas: Good question. For deterministic code, no issue. For non-deterministic, you need a way to make it deterministic for testing. Seed the random number generator. Serialize concurrent operations — or use a model checker to explore interleavings. But that's a whole other episode. Lucas: The short answer: diff testing works best on pure functions. If your function has side effects, you have to replicate the state — which is harder. But for many core algorithms, purity is common. Luna: I think the takeaway is: next time you're about to refactor a critical piece of code, before writing a hundred test cases, try running both versions on a bunch of random inputs and see if they agree. It's a cheap way to find the cracks. Lucas: And if you're curious to try it, start with a small library — maybe a date parser or a JSON validator. Two hours of setup could save you days of debugging later. That's the beauty of differential testing.