Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Property-Based Testing Finds Bugs Example Tests Miss
Transcript
- Lucas: So you write a function, you test it with a handful of examples — say a sorted list, an empty list, a list with duplicates — and they all pass. You ship it. A week later, a user reports that sorting a list with a million elements and a specific pattern crashes. Sound familiar? Luna: Way too familiar. And that's the core problem with example-based testing — you only test what you think to test. Lucas: Exactly. Today we're talking about property-based testing — a technique where instead of writing specific examples, you define general properties that should always hold, and let a computer generate thousands of random inputs to check those properties. Luna: It's not new — QuickCheck for Haskell has been around since 1999 — but it's finally getting serious traction in mainstream languages like Python, Java, and JavaScript. Lucas: Let's start with a concrete case. Imagine you've written a sorting function in Python. You test it with a list of five integers, an empty list, a list with one element, and a list with duplicates. All pass. But there's a bug — say it fails when the list has an odd number of elements with certain values. Luna: Right, and you'd never catch that with those four examples. But with property-based testing, you'd write one property: 'the output should be sorted, and it should be a permutation of the input.' Lucas: Then the framework — like Hypothesis in Python — generates random lists of random lengths, runs your function, and checks that property. If it finds a failing case, it shrinks it — tries to find the smallest possible failing input. Luna: And that shrinking part is key. When I first used QuickCheck, it found a bug in my JSON parser — but the failing input was a 10-kilobyte blob. Shrinking reduced it to a single malformed character. Lucas: That's the killer feature. Without shrinking, you'd be debugging a haystack. With shrinking, you get the needle. So property-based testing doesn't just find more bugs — it makes them easier to fix. Luna: So what kinds of properties do you actually write? Not everything has a simple mathematical invariant. Lucas: True, but there are patterns. For any transformation, you can often define 'idempotence' — applying the operation twice should give the same result as applying it once. Or 'round-tripping' — serialize then deserialize should give you back the original. Luna: Oh, round-trip properties are huge for data pipelines. We once found a bug where a timestamp format conversion lost milliseconds on the second pass because of a rounding error. The example-based tests never covered a full cycle. Lucas: Another common one is 'inverse' — if your function has a logical inverse, like encrypt and decrypt, then encrypt) should equal x. Or 'invariant under transformation' — like a sorting function's output length should always equal its input length. Luna: But what about more complex systems — like a microservice that processes orders? You can't just throw random data at it and expect it to survive. Lucas: You can, actually — if you define valid input generators. Hypothesis lets you compose generators: valid user IDs, order amounts within a range, timestamps in the past. You model the domain so the generated data is realistic but still explores edge cases. Luna: So it's like fuzzing, but with structure. Lucas: Exactly. Fuzzing usually expects crashes. Property-based testing expects logical correctness. And they complement each other well. Luna: I want to talk about a real-world success story. In 2020, the team behind Hypothesis published a case study where they applied property-based testing to the Python standard library's 'statistics' module. They found over 30 bugs, including a critical one in the median function that returned wrong results for even-length lists with certain repeated values. Lucas: And those bugs had survived years of traditional testing. That's the power of exploring the input space systematically rather than manually. Luna: But it's not a silver bullet. Property-based tests can be slow — generating thousands of cases takes time. And if your properties are poorly defined, you can get false positives or, worse, false negatives. Lucas: Right. A common mistake is writing a property that's too weak — like 'the output is not None' — which passes even if the function returns the wrong thing. You need properties that actually capture correctness. Luna: And there's the learning curve. Writing good generators and properties requires thinking in abstractions, not examples. It's a different mindset. Lucas: But once you get it, it's addictive. I've seen teams adopt property-based testing for their core logic and reduce regression bugs by over 50 percent within a quarter. Luna: And for microservices, contract testing with property-based approaches is especially powerful. Instead of fixing a few example requests, you define properties for the API's behavior — like 'every request with valid authentication returns either a 200 or a 4xx, never a 5xx'. Lucas: So how does a team start? I'd recommend picking a small, self-contained module — a parser, a validator, a transformation function — and writing one property for it. Use the library's built-in strategies to generate inputs. Luna: Hypothesis in Python has strategies for integers, strings, dates, nested structures. You can compose them with 'one_of' or 'lists_of'. It's quite elegant. Lucas: And if you're in the JVM world, there's jqwik for Java, and ScalaCheck for Scala. For JavaScript, fast-check is very popular. Luna: I'd also say: don't try to replace all your example tests overnight. Use property-based testing as a complement, not a replacement. Example tests document expected behavior for humans; property tests find the edge cases. Lucas: Great point. And once you have a solid property, you can even run it in production as a canary — generate random inputs and check invariants on a small percentage of traffic. Luna: That's actually something we've talked about in other episodes — testing in production. Property-based testing makes that safer because you're checking invariants, not specific outputs. Lucas: Let's zoom out for a second. The bigger idea here is that testing is about exploring state spaces. Humans are terrible at imagining all the ways software can break. Computers are great at generating variations. Luna: Property-based testing is one of the most practical ways to get computers to do that work for you. And it's not just for functional programmers anymore. Lucas: If today's conversation gave you something usable — maybe you're thinking about writing your first property test — that's exactly why we make this show. We deliberately don't run ads on these episodes. The only reason we can keep doing that is the listeners who support the show directly. The link is buy me a coffee dot com slash fexingo. Luna: Yeah, it's a tiny gesture that keeps this whole thing ad-free and focused on the content. We appreciate everyone who chips in. Lucas: Anyway, back to the topic — let's talk about a common pitfall: flaky properties. If your property depends on random data that isn't reproducible, you can get failures that don't reproduce — that's a nightmare. Luna: Most frameworks handle that by seeding the random generator. Hypothesis gives you a 'hypothesis.example' decorator to replay specific failures. You just save the seed and the failing input. Lucas: And that leads to another best practice: when a property fails, don't just fix the bug — add the failing case as a traditional example test. That way, future developers see exactly what broke. Luna: Exactly. So property-based testing discovers the edge case, and example testing documents it. They work together. Lucas: Let's look at one more advanced use case: stateful testing. Hypothesis can test entire workflows — like a database or a cache — by generating sequences of operations and checking that invariants hold after each step. Luna: That's how they found bugs in the Python standard library's 'lru_cache' decorator. They modeled the cache as a state machine and found a race condition. Lucas: Stateful testing is still niche, but it's incredibly powerful for concurrent systems. If you're building a distributed system, it's worth exploring. Luna: So to wrap up: property-based testing is not a magic wand, but it's a tool that can dramatically improve the quality of your tests by shifting your focus from specific examples to general invariants. Lucas: And the best part is, the barrier to entry is lower than ever. Pick your language, install the library, write one property. You'll be surprised what it finds. Luna: I'd love to hear from listeners who try it — what bugs did you catch that your example tests missed? Lucas: Until next time, keep your invariants invariant.