Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Property-Based Testing Finds More Bugs Than Examples
Transcript
- Lucas: So, we've spent a lot of episodes talking about how to write better tests—shift-left, mutation testing, visual regression. But there's a fundamental question we haven't touched: what if the way we write tests is the problem? Luna: You mean like, the whole approach of writing specific examples and asserting specific outcomes? Lucas: Exactly. Most test suites are built on example-based testing. You pick a few inputs, you know the expected output, and you assert that your code returns that output. But here's the thing—there are infinitely many possible inputs. Example-based testing only covers the ones you thought of. Luna: And bugs often live in the inputs you didn't think of. I've definitely shipped code that passed all my unit tests but broke the first time a user did something I didn't anticipate. Lucas: That's exactly the gap property-based testing aims to close. Instead of writing 'when I pass 42, I expect 84', you write a property like 'for any integer input, the function always returns a positive number'. Then the testing framework automatically generates hundreds or thousands of random inputs and checks that the property holds for all of them. Luna: So the machine does the exhaustive—or at least stochastic—exploration for you. That sounds powerful, but also kind of abstract. Can you give me a concrete example? Lucas: Sure. Let's say you're building a sorting function. An example-based test might check that sorting returns. A property-based test would check that for any list of integers, the output is sorted, has the same length, and contains the same elements as the input. The framework runs that against thousands of random lists. Luna: And it would catch a bug where the function accidentally drops a duplicate? Because the length property would fail. Lucas: Exactly. Or if it modifies the input list in place—that would break the element count property. The framework finds the minimal failing case, which is huge for debugging. Luna: I've heard of QuickCheck for Haskell and Hypothesis for Python. Are these mature enough for production use? Lucas: Absolutely. QuickCheck has been around since the early 2000s. Hypothesis is actively maintained and used at companies like Stripe and Mozilla. There's also ScalaCheck for Scala, JSVerify for JavaScript, and property-based testing libraries in Rust, Go, and Java. Luna: Let's talk about Stripe. I read that they use Hypothesis to test their payment routing logic. That's a system where incorrect behavior can directly lose money. Lucas: Right. Payment routing involves complex rules: which acquirer to use, currency conversion, fee calculations. Stripe's team wrote properties like 'the total amount charged should always equal the sum of amounts sent to the acquirer and the fees retained'. Then they let Hypothesis generate random transaction graphs. Luna: And did it find anything? Lucas: It found a bug where a specific rounding edge case—a one-cent discrepancy—occurred only when a transaction was split across three acquirers with different rounding modes. That example would have been nearly impossible to write by hand. Their example-based test suite had over twelve thousand cases, and none of them caught it. Luna: So the property-based test found a one-in-a-million bug that had been in production for months. That's the kind of story that makes you want to rewrite your entire test suite. Lucas: And that's the thing—you don't have to rewrite everything. Property-based testing complements example-based tests. You keep your happy-path examples for documentation and readability, and add properties for the core invariants that must never break. Luna: But there's a learning curve. Writing a good property requires thinking about what must always be true, which is harder than 'given input X, output Y'. Lucas: It's a shift in mindset. You start by identifying invariants. For a function that reverses a string, the property is 'reversing twice gives the original string'. For a sort function, it's 'the output is sorted and the multiset of elements is unchanged'. Once you get the hang of it, it becomes natural. Luna: What about stateful systems? Like a database or a file system? Lucas: Great question. There's a technique called stateful property-based testing. You model the system's state abstractly—say, a set of key-value pairs—and define operations like 'put', 'get', 'delete'. The framework generates random sequences of operations and checks that the real system's state matches the model. It's incredibly effective at finding race conditions and consistency bugs. Luna: I can see how that would catch things like lost updates or phantom reads. Is this used in production at scale? Lucas: Yes. FoundationDB, which is a distributed database, uses stateful property-based testing extensively. They model the entire database as a simple key-value store and run random concurrent operations against it. They've found bugs that would never surface in a controlled test harness. Luna: So the testing framework essentially becomes a fuzzer with an oracle. It generates chaotic inputs and checks against a known correct model. Lucas: Exactly. And the model doesn't have to be perfect—it just has to capture the essential correctness properties. Over time, you refine the model as you discover new edge cases. Luna: One concern I have is test run time. If you're generating thousands of random inputs per test, doesn't that slow down your CI pipeline? Lucas: It can, but you control the number of examples. You can run a small set on every commit and a large set nightly. Also, tools like Hypothesis automatically shrink failing cases and cache results, so the most impactful tests run first. In practice, many teams find the trade-off worth it for the bugs they catch. Luna: I want to try this myself. What's a good way to start? Lucas: Pick a small, pure function in your codebase—something that takes inputs and returns outputs without side effects. Write a property like 'the output is always in the expected range' or 'the function is idempotent'. Run it with a library like Hypothesis for Python. You'll likely be surprised by what it finds. Luna: And if it doesn't find anything, that's also useful—it gives you confidence that your function is robust for a broad set of inputs. Lucas: Right. And that confidence scales. Example-based testing gives you confidence about the cases you imagined. Property-based testing gives you confidence about the cases you didn't imagine. Luna: Speaking of confidence—one thing that keeps this show going is knowing that listeners find these deep dives useful. We don't run ads, we don't have a corporate sponsor, and we want to keep it that way. Lucas: Yeah, it's something we're proud of. A small but generous group of listeners supports us through buy me a coffee dot com slash fexingo. That's what makes it possible to produce episodes like this without any outside pressure. Luna: It's not a big ask—just if you've gotten value from the show and want to keep it ad-free, that's the way. And it genuinely makes a difference. Lucas: Anyway, back to property-based testing. I think the key takeaway is that it forces you to think about what your code must guarantee, not just what it should do for a few examples. That's a higher level of specification. Luna: It's like writing a contract instead of a list of scenarios. And that contract can be tested automatically, at scale. Lucas: Exactly. And once you have that contract, you can refactor with confidence. If your code still satisfies the property, you know you haven't broken the core behavior—even if you completely rewrite the implementation. Luna: That's a powerful tool for agile teams that need to move fast without breaking things. Lucas: One more thing: property-based tests can also serve as living documentation. The properties capture the intended behavior in a precise, machine-checkable way. New team members can read the properties to understand what the system is supposed to do. Luna: Better than a wiki that's always out of date. Lucas: Way better. So, if you're tired of flaky tests and want to systematically cover the infinite space of inputs, start with a small property today. Your future self will thank you. Luna: And if you want to hear more about advanced testing techniques, let us know. We've got a whole backlog of topics. Lucas: Next time, we'll look at contract testing for microservices—another approach that shifts the testing paradigm. See you then.