Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Fuzzing Unlocks Edge Cases Your Tests Miss
Transcript
- Lucas: Picture this: you've written a thousand unit tests. Your code coverage is at 85 percent. Every function has a happy-path test and a couple of obvious edge cases. You deploy to production, and within an hour, a user sends a string with a single unexpected character — and the entire service goes down. Luna: Yeah, that's the classic blind spot. You test what you think of, and the universe thinks of something else. Lucas: Exactly. And that's the problem that fuzz testing — or fuzzing — was built to solve. Instead of writing test cases by hand, you let a tool generate thousands or millions of random inputs and throw them at your program. If any input causes a crash, an assertion failure, or a memory leak, you've found a real bug that your manual tests missed. Luna: And this isn't some niche academic technique. I mean, Google's oss fuzz has been running continuously since 2016 and has found over 30,000 bugs in critical open-source projects — things like OpenSSL, FFmpeg, libpng. Luna: A 33 percent crash rate on standard Unix utilities. That's wild. Lucas: It set the stage for decades of work. The key insight is that human beings are terrible at imagining all the weird inputs a program might receive. Fuzzers don't have that bias — they just generate noise and see what breaks. Luna: So how does a modern fuzzer actually work? I know there are different flavours — dumb fuzzing, coverage-guided fuzzing, grammar-based fuzzing. Lucas: Yeah, let's break those down. A dumb fuzzer — also called a black-box fuzzer — just generates random bytes and feeds them to the program. It's simple and fast, but it often wastes time on inputs that get rejected at the parsing layer. A coverage-guided fuzzer, like American Fuzzy Lop — AFL — or libFuzzer, instruments the code to track which branches are being executed. When it finds an input that triggers a new code path, it saves that input and mutates it further. Luna: So it learns as it goes. It's not just throwing spaghetti at the wall — it's optimising to explore deeper logic. Lucas: Exactly. And then you have grammar-based fuzzers, which understand the structure of the input — like a valid JSON or an HTTP request — and generate random but syntactically valid data. That's useful when you want to test the business logic rather than the parser. Luna: I've heard that Microsoft found a decade-old bug in their Windows TCP/IP stack using fuzzing. Can you tell that story? Lucas: Yeah, that's a great example. In 2019, Microsoft's internal fuzzing team — part of their Security Response Center — was running coverage-guided fuzzers on the Windows networking stack. They found a bug in the handling of IPv6 fragmentation that had been present since Windows 7. It was a race condition that only triggered under very specific packet sequences. Manual testing had never caught it because the timing window was so narrow. Luna: And I assume that bug had been shipping for over a decade in every version of Windows. Lucas: Exactly. It took millions of fuzzed inputs and thousands of CPU hours to find, but once it was found, the fix was a few lines of code. That's the power of fuzzing — it surfaces bugs that human testers would never think to look for. Luna: If today's tech conversation gave you something usable — maybe a new approach to edge cases, or a technique you want to try — that's exactly the kind of content we aim to bring every episode. And listener support is what keeps this show ad-free and focused on the substance. If you find value in what we do, you can support us at buy me a coffee dot com slash fexingo. Lucas: Yeah, and it genuinely makes a difference. Every contribution goes straight into production time, research, and keeping the podcast independent. No pressure, but if you've been meaning to, that's the place. Luna: So back to fuzzing — how do you integrate it into a typical CI pipeline without slowing everything down? Lucas: Great question. Most teams run fuzzers as a separate CI job, not on every commit but on a schedule — nightly or weekly. Google's oss fuzz is a good model: it runs continuously on a cluster of machines and reports bugs to the project maintainers. For smaller teams, you can use tools like cargo-fuzz for Rust, Jazzer for Java, or Atheris for Python. They integrate with your existing test framework. Luna: And what about the argument that fuzzing is overkill for a startup that's just trying to ship a prototype? Is it worth the setup cost? Lucas: I'd argue it's even more valuable for startups. A single security bug can tank a young company's reputation or get you kicked off a platform. And the setup cost is minimal — you can write a basic fuzzer in about 50 lines of Python using the atheris library. The time investment is maybe an afternoon. Compare that to the cost of a production incident. Luna: Fair point. So it's not about covering every line with fuzz tests — it's about targeting the high-risk areas: input parsers, network handlers, authentication logic. Lucas: Exactly. You don't fuzz your entire codebase. You fuzz the boundaries where external data enters your system. That's where the most dangerous bugs live. Luna: I want to talk about something that often gets overlooked: the output checking problem. A fuzzer can tell you when the program crashes, but how do you catch logical bugs — where the program doesn't crash but produces the wrong result? Lucas: That's where assertions and oracles come in. You embed assertions in your code — things like 'this function should always return a positive number' or 'the sum of these two values should not exceed X'. When the fuzzer generates an input that violates an assertion, it reports a failure even if the program didn't crash. Some teams also use differential testing: run the same input through two implementations and compare the outputs. Luna: Differential testing is clever. Like, if you have a reference implementation in Python and a new version in Rust, you can fuzz both and flag any discrepancy. Lucas: Right. That's how the CertiK team found bugs in blockchain smart contracts — they fuzzed the contract against a formal specification. But you don't need formal methods to benefit from output checking. Even a simple sanity check like 'the result should parse back correctly' can catch subtle issues. Luna: I've also seen fuzzing used for performance regression. You can set a threshold on function execution time and flag inputs that cause unusually slow processing. Lucas: Yeah, that's a growing area — performance fuzzing. Tools like google/benchmark can be paired with fuzzers to detect algorithmic complexity attacks. For example, a hacker could send a specially crafted input that causes a hash table to degrade from O to O. A performance fuzzer would catch that slowdown before it hits production. Luna: Let's talk about the ecosystem. What are the go-to tools right now for a team that wants to start fuzzing tomorrow? Lucas: I'd start with libFuzzer if you're working in C or C++ — it's part of the Clang compiler suite and incredibly fast. For Rust, cargo-fuzz wraps libFuzzer and is dead simple. For Java, Jazzer is the standard — it's also built on libFuzzer under the hood. For Python, Atheris from Google is solid. And if you want a managed service, Google's oss fuzz is open to critical open-source projects, and you can also run your own instance using ClusterFuzz. Luna: And there's a new class of coverage-guided fuzzing tools that use symbolic execution — like Angr or SAGE from Microsoft — but those are heavier and more research-oriented. Lucas: Right. For most teams, the lightweight tools are more than enough. The barrier to entry is lower than ever. I'd say any team shipping software that handles untrusted input should have at least one fuzzer running. Luna: Is there a risk of false positives? I've heard fuzzers can report the same bug multiple times, or flag issues in third-party libraries that you can't fix. Lucas: That's a real concern. Modern fuzzers have deduplication — they track the stack trace of each crash and group identical ones together. For third-party library bugs, you can either suppress those signals or report them upstream. oss fuzz actually has a triage team that does that work for open-source projects. The false positive rate for crashes is very low, but for assertions it can be higher if your assertions are poorly designed. Luna: So the key is to write good assertions and have a triage process. That's not unique to fuzzing — it's true for any testing approach. Lucas: Exactly. And the return on investment is enormous. I've seen teams adopt fuzzing and discover within the first week critical bugs that had been in production for years. It's one of those techniques that once you start using, you can't imagine not using it. Luna: I'm curious about the regulatory angle. Do you think fuzzing will ever be mandated for safety-critical software — like medical devices or autonomous vehicles? Lucas: It's already happening in some domains. The FDA's guidance on premarket submissions for medical device software includes recommendations for robust testing of inputs, and fuzzing is increasingly mentioned. For automotive, the ISO 26262 standard doesn't explicitly require fuzzing, but the industry is moving toward it as a best practice. I wouldn't be surprised if, within five years, fuzzing becomes a de facto requirement for any software that could cause physical harm. Luna: That would be a huge shift. But it makes sense — why would you trust a life-critical system that hasn't been tested against random inputs? Lucas: Right. And the same logic applies to financial systems, authentication services, any software that handles money or identity. The cost of a bug in those systems is measured in dollars and trust. Fuzzing is a cheap insurance policy. Luna: Alright, so this has been a deep dive into fuzzing. Let's leave the audience with one actionable takeaway — what's the first step for a team that wants to try this? Lucas: Pick one input parser — say, a JSON endpoint in your API. Write a simple fuzzer that generates random JSON payloads and sends them to your endpoint. Use a library like Jazzer or Atheris. Run it for an hour. I guarantee you will find at least one crash or unexpected behaviour. And once you see that first bug, you'll be hooked. Luna: And then you'll wonder how you ever shipped software without it. Lucas: Exactly. That's the fuzzing moment.