Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Property-Based Testing Finds Bugs Example Tests Miss
Transcript
- Lucas: So let's say you write a test for a function that validates a trade. You feed it a normal trade, it passes. You feed it a trade with zero quantity, it fails the way you expect. You feel good, right? But what if the bug only shows up when the notional is exactly one point three seven times the sum of allocations, and the currency is Swiss francs, and the timestamp falls on a leap day? Luna: Right, that's the classic example-based testing problem. You test what you think of, not what you don't think of. Lucas: Exactly. And that's the gap property-based testing is designed to fill. Instead of writing a test case with specific inputs, you define a property that should hold for any valid input — an invariant — and then you let a tool generate hundreds or thousands of random inputs to try to break it. Luna: So it's like fuzz testing, but with a brain. You're not just throwing random bytes at the system. You're saying 'for all x that satisfy these conditions, the result should always be true.' Lucas: That's exactly the framing. The classic example is a sorting function. You could write a test that says 'sort this list of five integers' and assert the output is sorted. But with property-based testing, you say: for any list of integers, the output should be sorted, and it should contain the same elements as the input. The tool generates a hundred random lists, some empty, some with duplicates, some with negative numbers. If any of those break your function, you've found a bug you never thought to test. Luna: And the tool doesn't just find the failure. It shrinks the input to the smallest case that still fails, right? So you get a minimal reproduction. Lucas: That's the killer feature. The shrinking step. In the sorting example, if your function fails on a list of a thousand random numbers, the tool will try to simplify it — removing elements, reducing values — until it gets down to maybe a list of two numbers that reveals the actual logic error. Without that, you'd be debugging a haystack. Luna: I've seen property-based testing catch things that had been in production for months. A colleague at a payments company had a rounding function that looked correct. They wrote a property: 'for any transaction amount, rounding then un-rounding should return the original amount up to a certain precision.' The tool found a case where it didn't. Lucas: That's a beautiful property. And it's the kind of invariant that's hard to think of when you're writing example tests. You test the happy path and maybe the off-by-one. But you don't test the combinatorial explosion of possible amounts. Property-based testing forces you to think at a higher level: what are the laws that my system must obey? Luna: So how do you start? Do you need to rewrite your entire test suite? Lucas: Absolutely not. The best approach is to pick one module, one function, one property that you suspect has edge cases. Write a single property, run it, see what happens. The most popular tools are Hypothesis for Python, QuickCheck for Haskell and Java via jqwik, and for JavaScript there's fast-check. They all have shrinking built in. Luna: And what about the learning curve? The team I worked with found it hard to write good properties at first. They kept writing properties that were too weak — like 'the result is not null' — which the tool passes trivially. Lucas: That's the number one mistake. A property that only checks something obvious is a waste of compute. The property needs to capture a real business invariant. For example, in a trading system: 'for any valid trade, the sum of the allocations across all legs must equal the trade notional.' That's a concrete constraint that, if violated, means money is misallocated. Luna: And the tool will generate trades with random allocations, some that sum correctly, some that don't. But you have to set up the generator to only produce valid inputs according to your schema, otherwise you're testing the validation, not the business logic. Lucas: Right. You need separate generators for valid and invalid inputs. You test the business logic with valid inputs and assert the property holds. You test validation with invalid inputs and assert it throws. Mixing them is where flakiness comes from. Luna: Speaking of flakiness — property-based tests can be flaky if the property depends on something nondeterministic, like system time or random number generation inside the function under test. Lucas: Exactly. That's a common pitfall. You have to control the randomness. Most frameworks let you seed the generator so that failures are reproducible. Hypothesis, for example, prints the seed that caused the failure. You can replay it deterministically. Luna: So it's not just about catching bugs. It's about documenting the rules of your system in a way that's executable. That's powerful. Lucas: It's a shift in mindset. Instead of asking 'what test case should I write?', you ask 'what must always be true about this system?' That question alone improves the design of your code, because vague invariants become obvious. Luna: If today's conversation made you think about your own test suite in a new way, that's exactly why we keep these episodes free and ad-free. We deliberately don't run ads on these episodes. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. Lucas: Yeah, we hear from listeners that the no-ads format lets them focus on the content without interruption. So if it's valuable to you, that's the place to show it. Luna: Okay, back to properties. I want to talk about stateful testing — where the property involves a sequence of operations, not just a single function call. Lucas: That's where property-based testing really shines for systems with state, like a database or a key-value store. You define a model of what the state should be — say, a Python dictionary — and then you generate random operations: insert, delete, update. After each operation, you assert that the real system's state matches the model. Luna: That catches concurrency bugs, race conditions, and weird interleavings that you'd never think to test manually. The tool will find the sequence that breaks your system and shrink it to maybe three operations. Lucas: And it's not just for storage systems. I've seen teams use it for testing REST APIs. They model the expected resource state, generate random calls — POST, GET, PATCH, DELETE — and check that the API never violates invariants like 'a deleted resource returns 404' or 'creating a resource returns a 201 with a location header.' Luna: What about performance? Generating thousands of random inputs sounds expensive. Lucas: It depends. For a single property, running a few hundred iterations takes maybe a second in Python. In compiled languages like Java, it's even faster. You don't run it on every save. You run it in CI on a schedule, or before a release. The cost is negligible compared to catching a production bug. Luna: And the payoff is that you get a test suite that doesn't just document what you knew — it discovers what you didn't know. Lucas: Exactly. And that's the real argument for property-based testing. It turns your tests from a safety net into a bug-finding engine. The question isn't whether you can afford to run it. It's whether you can afford not to.