Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Tests Need Fuzzing for Input Validation
Transcript
- Lucas: So you've built a solid test suite. Unit tests pass, integration tests are green, your visual regression checks are in good shape. Then a user types 'Robert'); DROP TABLE Students;--' into a form field and your app goes up in flames. Luna: Classic Bobby Tables. That's a SQL injection attack, but honestly, it could be any kind of malformed input that your tests never thought to throw at the system. Lucas: Exactly. And that's the gap we're going to talk about today: fuzz testing. Fuzzing is the practice of feeding a program a stream of random, malformed, or unexpected inputs and monitoring for crashes, hangs, or memory errors. Luna: It's not new — the concept goes back to the late 80s at the University of Wisconsin-Madison. But it's having a real renaissance now, especially with tools that integrate into CI pipelines. Lucas: Right. And the reason it's so important is that conventional test cases — even good ones — tend to be polite. They test what the developer expected. Fuzzing tests what the developer didn't expect. And that's where the scary bugs live. Luna: Can you give me a concrete example where fuzzing would have caught something that regular testing missed? Lucas: Sure. Let's talk about Shellshock. In 2014, a vulnerability was found in the Bash shell — the command-line interpreter used on most Linux and Unix systems. The bug allowed attackers to execute arbitrary commands by passing a specially crafted environment variable. It was a huge deal — CVSS score of 10 out of 10, affected millions of systems. Luna: And it had been in the code since version 1.03, which came out in 1989. So for 25 years, nobody noticed. Lucas: Right. And here's the thing: a simple fuzzer that threw random strings at the environment variable parser would have triggered a crash — or at least anomalous behavior — very quickly. Because the bug was in how Bash handled trailing characters after a function definition. Fuzzing would have found it in minutes. Luna: That's a powerful example. So what does a modern fuzzing setup actually look like? Because I think some teams hear 'fuzzing' and imagine a lab with a million machines running 24/7. Lucas: It can be that intense, but it doesn't have to be. For most teams, you can start with something like libFuzzer — it's an in-process, coverage-guided fuzzing engine that's part of the LLVM compiler toolchain. You write a small harness that calls your function with a random input, compile with address sanitizer, and run. Luna: Coverage-guided — that means it tracks which code paths each input explores, and then mutates inputs to try to reach new paths, right? Lucas: Exactly. So instead of purely random noise, the fuzzer learns. It generates inputs that penetrate deeper into the code. And it's surprisingly efficient. A single core can run thousands of test cases per second. Luna: And then you integrate that into your CI pipeline. So every time you push code, the fuzzer runs for a set amount of time — say, five minutes — and if it finds a crash, it reports it. Lucas: That's the ideal. Google's oss fuzz project does this at scale for hundreds of open-source libraries. They've found over 20,000 bugs since 2016. But even a small team can benefit from running libFuzzer on their most critical input-parsing code. Luna: Let's talk about what kinds of bugs fuzzing is particularly good at finding. I'm thinking buffer overflows, format string vulnerabilities, integer overflows... Lucas: All of the above. Plus use-after-free errors, null pointer dereferences, infinite loops. Basically anything that can cause a crash or undefined behavior. And the beauty is that you don't need to know what the input should look like — you just need to define what's 'interesting' input for your program. Luna: So for a JSON parser, you'd feed it not just valid JSON, but truncated JSON, deeply nested JSON, JSON with unicode, with binary data, with huge numbers... Lucas: Right. And the fuzzer will quickly find edge cases like a string that's exactly 65536 characters long causing a buffer overflow, or a deeply nested structure blowing the stack. These are exactly the kind of inputs that a human developer might not think to test, but an attacker will. Luna: What about the cost? I imagine running fuzz tests in CI could slow things down. Lucas: It can, but you can mitigate that. You don't need to fuzz every commit. You can run fuzzing nightly, or on a schedule. And you can use a small seed corpus — a handful of valid inputs — to give the fuzzer a head start. The key is to make it part of your quality process, not a one-off exercise. Luna: There's also a growing ecosystem of cloud-based fuzzing services that handle the scaling for you. You just provide the harness and they run it on a fleet of machines. Lucas: Yeah, companies like Code Intelligence and Synopsys offer those. But honestly, for many teams, a simple local setup is enough to catch the low-hanging fruit. And that low-hanging fruit can be the difference between a vulnerability disclosure and a quiet Wednesday afternoon. Luna: Let me ask — how does fuzzing compare to other techniques we've covered on this show, like property-based testing? Lucas: Good question. Property-based testing, which we did an episode on, generates random inputs but checks that certain properties hold — like 'the sort function always returns a list of the same length.' Fuzzing is more focused on finding crashes and memory errors. They're complementary. You can use both. Luna: And fuzzing is especially powerful when combined with sanitizers — AddressSanitizer, UndefinedBehaviorSanitizer, MemorySanitizer. Those tools instrument the code to detect errors during execution, and the fuzzer triggers them. Lucas: Exactly. A fuzzer without a sanitizer might only find input that causes a visible crash. With a sanitizer, it can detect subtle memory corruption that might not crash immediately but could be exploited later. Luna: I want to talk about one more thing: fuzzing for non-crash bugs. Like logical errors that cause wrong output but don't crash. Can fuzzing catch those? Lucas: It's harder, but yes — if you can define a property that the output must satisfy, you can check it inside the fuzzer harness. For example, if you're fuzzing a compression library, you can check that decompressing the output gives back the original input. That's a property-based fuzzing hybrid. Luna: So it's not just about crashes. It's about any violation of a contract. Lucas: Right. And that's where a well-written harness can really shine. You're essentially encoding your assumptions about the system and letting the fuzzer test them millions of times. Luna: One thing that's come up in our listener questions — and this ties back to what we were just saying — is how to convince your team to invest time in setting up fuzz testing. Especially if they feel like their existing test suite is good enough. Lucas: It's a common pushback. But the counterargument is straightforward: conventional tests only cover the paths you thought of. Fuzzing covers the paths you didn't. And for security-critical code — parsers, network protocols, authentication — the cost of not fuzzing can be astronomical. Look at the Equifax breach, which was caused by a known vulnerability in Apache Struts that a fuzzer might have caught. Luna: Yeah, that's a compelling argument. And I'd add that fuzzing can be surprisingly fun. The first time you see a fuzzer find a crash in your own code, it's both terrifying and exhilarating. Lucas: Absolutely. It's like having a million monkeys typing at your code, but the monkeys are smart and have debuggers. Luna: So for listeners who want to start today — what's the quickest path? Lucas: Pick one function that parses user input. Write a simple harness using libFuzzer or your language's equivalent. Compile with AddressSanitizer. Run it for an hour. See what happens. I guarantee you'll find something interesting. Luna: And if nothing else, you'll have a new appreciation for the chaos that users — and attackers — can throw at your software. Lucas: On that note — and I don't want to break the flow here — if today's conversation gave you something usable, it's worth mentioning that this show stays ad-free because of listeners who chip in a couple of dollars a month at buy me a coffee dot com slash fexingo. Genuinely, that's what keeps these episodes coming. Luna: Yeah, it's a small thing that makes a real difference. And we're grateful for every bit of support. Lucas: So back to the technical side — one more angle I want to touch on: fuzzing for APIs. This is a big deal for microservices. Luna: Oh, good. Because APIs are a prime target for malformed inputs — JSON with extra fields, missing fields, wrong types, huge payloads. Lucas: Right. There are tools like schemathesis that auto-generate test cases from OpenAPI specs. They fuzz your API endpoints by sending requests that violate the schema in various ways. And they find real bugs — like endpoints that crash when sent an empty string instead of an integer. Luna: And those bugs are exactly the kind that slip through unit tests because the unit test mocks the request and the mock never sends an empty string. Lucas: Exactly. Fuzzing at the API level gives you a different perspective. It's not about code coverage — it's about behavior under adversarial conditions. Luna: So to sum up: fuzzing is a powerful addition to your testing toolkit, especially for input validation, security, and stability. It's not a replacement for other testing, but it's a complement that finds bugs you didn't know existed. Lucas: And it's more accessible than ever. Open-source tools, cloud services, CI integration — there's no excuse not to try it. Start with one component, see what you find, and scale from there. Luna: If you've never seen a fuzzer in action, I'd encourage you to give it a shot this week. It might change how you think about testing. Lucas: And if you do find something — especially something that would have been a production incident — drop us a note. We'd love to hear about it. For now, thanks for listening.