Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Testing Fuzz Generators Catches Edge Case Exploits
Transcript
- Lucas: So — we talk a lot on this show about writing better tests. Unit tests, integration tests, fuzz tests. But here's a question I've been sitting on: who tests the fuzz generator? Luna: Wait — you mean the tool that throws random data at your code? That thing needs testing too? Lucas: Exactly. And the answer is, most teams don't. They assume if the fuzzer runs without crashing, it's doing its job. But a fuzz generator can have bugs — silent ones that reduce coverage without you knowing. Luna: Okay, I need an example. What kind of bug are we talking about? Lucas: Let's use Google's oss fuzz as a case study. It has a library of seed inputs — small files that kickstart the fuzzer. If a seed is malformed or gets corrupted during processing, the fuzzer might only explore a narrow path. Or worse, the generator's mutation logic could have an off-by-one error that never produces certain byte sequences. Luna: So the fuzzer is running, green checkmarks everywhere, but it's basically blind to whole classes of input. Lucas: Right. And the scary part is, you only find out when a bug slips to production. So the idea is: treat your fuzz generator like any other piece of critical software. Write tests for it. One approach is mutation coverage of the generator code itself. Luna: Mutation coverage — we talked about that for production code in episode 80. How does it apply here? Lucas: You take the generator's source, inject small mutations — change a bit in a boundary condition, flip a logic operator — and then check whether your test suite catches the difference. If your tests don't detect a mutated generator that's producing narrower ranges, your test suite has a blind spot. Luna: So you're testing the test generator. Meta, but I buy it. What's a concrete technique? Lucas: One I like is called 'generator differential testing'. You run the original generator and a mutated version side by side, feeding the same seed corpus. Then you compare the sets of outputs they produce. If the mutated version fails to produce certain edge cases — say, a string with a null byte in the middle — you know the mutation broke something. Luna: And if your tests don't flag that difference, you've found a hole in your test suite. Lucas: Exactly. There's a paper from 2023 at the ICST conference — they did this on libFuzzer and AFL. They found that about 12 percent of mutation-injected bugs in the generator went undetected by the standard test suites. That's a lot. Luna: Twelve percent blind spots. So what did they recommend? Lucas: They proposed augmenting the test suite with a set of 'generator invariants'. For example, if your fuzzer is supposed to produce inputs with specific properties — like valid JSON or non-negative integers — you write a property-based test that checks every output against those invariants. Luna: So it's like contract testing for the fuzzer itself. If the generator suddenly starts emitting NaN values where it shouldn't, the property test fails. Lucas: Right. And you can automate that in CI. Run the generator on a fixed seed suite, run the property checks, and block the build if any invariant is violated. That catches regressions before they hit your fuzzing pipeline. Luna: I'm thinking about the teams I've worked with. Most of them use a single fuzz harness, maybe with a few hand-picked seeds. They never audit the generator. Lucas: That's the norm. And it's understandable — the fuzzer is a tool, not the product. But if the fuzzer is silently missing a code path, it's worse than having no fuzzer at all, because you have false confidence. Luna: So what's the minimal thing a team can do next week? Not a full research project. Lucas: Two steps. First, write a simple property test that checks your fuzzer's output against a known format. For example, if you're fuzzing a parser that expects UTF-8 strings, verify every generated input is valid UTF-8. Second, create a 'golden seed' set — say ten carefully crafted inputs that should each trigger a specific code path. Run them through the fuzzer and assert they're still in the generated corpus after mutation. Luna: So if the fuzzer's mutations accidentally discard a golden seed, you know something changed. Lucas: Exactly. And that's a low-effort way to catch generator drift. One team at Mozilla did this for their JavaScript fuzzer — they found that an update to the random-number generator had shifted the distribution, making it far less likely to produce deep recursion cases. Luna: Which is exactly the kind of bug that leads to stack overflow exploits. Lucas: Right. So the broader point is: fuzzing is powerful, but it's not fire and forget. The generator is software, and software has bugs. Treating it as a black box is a risk. Luna: I want to dig into one more thing — the seed corpus itself. You mentioned corrupted seeds. How do you test for that? Lucas: Great question. A seed corpus can degrade over time — files get truncated, encoding gets mangled, or a new version of the generator expects a slightly different format. One technique is to compute a checksum or hash of each seed file and store it in a manifest. Then in CI, you verify every seed matches its hash. If a seed got corrupted, the build fails. Luna: That's basically data integrity testing for your test data. I like it. Lucas: And you can extend it: for seeds that are supposed to trigger specific coverage events, you can run a coverage tool like gcov or llvm-cov and assert that the seed actually hits the expected lines. It's a bit heavier, but for critical seeds it's worth it. Luna: Okay, so let me tie this back to something we've said before. Testing is a stack. You test your code, you test your tests, and now you're saying test your test tools. Lucas: Exactly. The stack goes deeper than most people think. And the return on investment is real — catching a generator bug early can save weeks of chasing phantom failures. Luna: Speaking of return on investment — if today's episode gave you something useful, a practical idea you might actually use, this show stays ad-free because of listener support. You can help keep it that way at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small gesture that goes a long way for us. And we genuinely appreciate everyone who's chipped in. Luna: Alright, back to the tech. Lucas, you mentioned Mozilla's JavaScript fuzzer. Are there any open-source tools that help with generator testing? Lucas: Yes. There's a tool called 'FuzzIntrospector' from the oss fuzz team. It analyses a fuzz target and reports on coverage, but it also has a mode that evaluates the generator's behaviour — like whether it's hitting all the branches it should. And there's 'FuzzTest' from Google, which is a C++ framework that includes built-in property testing for fuzz generators. Luna: So teams don't have to build everything from scratch. They can adopt existing tooling. Lucas: Right. And the key is to integrate it into your CI pipeline. Run a nightly job that does generator differential testing or property checks. If it fails, you get an alert before the generator silently regresses. Luna: One last thing — what about fuzz targets that wrap external APIs? Like, if you're fuzzing a REST client? Lucas: That's trickier, because the generator might need to produce valid HTTP requests. You'd want to write property tests that verify, say, every generated request has a well-formed URL and headers. And you'd want to seed it with real API examples. But the same principles apply. Luna: So the takeaway is: don't trust your fuzzer blindly. Test the tester. Lucas: Exactly. And that's the kind of meta-thinking that separates a mature QA practice from one that's just going through the motions. Luna: Great stuff. Next episode, I want to talk about something we touched on — how to measure the health of a fuzzing campaign over time. Lucas: Love it. We'll dig into metrics like coverage growth rate and crash uniqueness. Until then, keep testing the tests.