Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Property-Based Testing Finds Bugs Example-Based Tests Miss
Transcript
- Lucas: You've probably written a test that passes on every input you thought of, only to have a user find a bug with some weird combination you never considered. That's where property-based testing comes in. Luna: I've definitely been there. Example-based tests feel safe but they only cover what you explicitly imagine. Lucas: Exactly. Property-based testing flips the approach. Instead of writing 'given input X, expect output Y,' you define a property that should hold true for all valid inputs, and the testing tool generates hundreds or thousands of random inputs to check it. Luna: So instead of saying 'if I add 5 and 3, I get 8,' you'd say 'addition is commutative' and let the computer throw random numbers at it? Lucas: That's the idea, though in practice properties are more specific to your business logic. Let's take a concrete example. Say you have an e-commerce function that applies a discount: ten percent off for orders over fifty dollars, but with a maximum discount cap of twenty dollars. Luna: Okay, that sounds straightforward. Lucas: An example-based test might check: order total of sixty dollars gets a six dollar discount. Order total of two hundred dollars gets a twenty dollar discount because of the cap. Looks fine. But what if the order total is exactly fifty dollars? Or what if the quantity causes a fractional cent rounding issue? Luna: Right, those edge cases are easy to miss. Lucas: With property-based testing, you'd write a property like: 'the discount should never exceed twenty dollars, and the final price should never be negative.' The tool — say Hypothesis in Python — generates random inputs, including edge values like zero, negative numbers, very large numbers, and weird decimals. Luna: And it would catch the case where the input is negative? That's an obvious bug, but it also catches subtler ones. Lucas: Absolutely. A team I worked with found a bug where for certain order totals just above the threshold, a floating-point rounding error caused the discount to be one cent more than the cap. The example test passed because it used clean numbers like 50.00 and 200.00. The property test ran with 57.83 and caught it. Luna: That's a great example. It reminds me of a fintech startup I talked to. They had a currency conversion function that worked fine in all their manual tests, but when they ran property tests on transaction fees, they found a rounding bug that had been in production for six months. Lucas: Did they share what the bug was? Luna: It was a loss of precision when converting from a small currency to a large one — like converting a few hundred yen to dollars. The rounding was truncating instead of rounding to the nearest cent, so every transaction lost a fraction of a cent. Over thousands of transactions, it added up. Lucas: And the property they wrote was probably something like 'the converted amount should be within one cent of the expected conversion using a high-precision calculator.' Luna: Exactly. They defined an invariant: the output should be a valid monetary amount with two decimal places, and the rounding error should never exceed half a cent. The property test found the truncation immediately. Lucas: Now, one challenge with property-based testing is writing good properties. It's not always obvious what the invariant is. For a sort function, it's easy — the output should be in order and contain the same elements. But for business logic like discount calculations, you have to think abstractly. Luna: It's a different mindset. Instead of 'this specific input gives this output,' you ask 'what should always be true about the output?' Lucas: Right. And there are patterns. For example, if you have a function that processes a list, the property might be that the length of the output equals the length of the input. Or if you have a function that formats a date, the property is that the output can be parsed back into a valid date. Luna: Those are called metamorphic relations, right? Where you can relate multiple outputs to each other. Lucas: Exactly. Another common one is idempotence: running the function twice should give the same result as running it once. That catches state-related bugs. Luna: I imagine the tool shrinks the failing input too, so you don't get a huge random mess to debug. Lucas: That's a key feature. When Hypothesis finds a failing input, it automatically shrinks it to the simplest input that still fails. So if a bug only occurs with numbers above a million, it'll find the smallest number that triggers it. That makes debugging much easier. Luna: So you're not left staring at a random 47-line input. Lucas: Exactly. And the shrinking is smart — it tries to reduce the input in ways that preserve the failure. It's one of the reasons property-based testing is practical, not just academic. Luna: That sounds like something worth trying. But teams are busy — how do you start without it feeling like a big overhead? Lucas: Start small. Pick one function that has clear properties — maybe a validation function, like checking email format. Write a property that says 'every output should match a standard email regex.' Run it, see what happens. Once you get comfortable, replace one example-based test per sprint with a property-based equivalent. Luna: And you don't have to throw away your existing tests. Property tests complement them. Lucas: Exactly. Example tests are great for documenting expected behavior — 'this is what we designed.' Property tests are great for exploring the unexpected. They're like a fuzzer but with intent. Luna: It's a bit like the difference between checking your work by re-reading it versus getting someone else to read it fresh. Lucas: That's a good analogy. And the tools are mature now. Hypothesis for Python, QuickCheck for Haskell, fscheck for.NET, and even property-based libraries for JavaScript like fast-check. There's no excuse not to experiment. Luna: I've heard some teams worry that property-based tests are slow because they run hundreds of inputs. Lucas: They can be slower than a single example test, but you control the number of examples. Hypothesis defaults to a hundred, which is usually enough. And you can cache results or run them in CI on a schedule rather than on every commit. Luna: That's a good compromise. Lucas: Plus, the bugs they find are often the ones that would take hours to track down in production. A few extra seconds in CI is a cheap price. Luna: And speaking of cheap, a couple of dollars a month is genuinely what keeps these shows going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: Yeah, it really does make a difference. Keeps us ad-free and focused on the content. Luna: So back to property-based testing — one thing I wonder about is how you handle stateful systems, like a database or a queue. Lucas: That's where stateful property-based testing comes in. Tools like Hypothesis have a stateful testing module where you define actions — like 'add item,' 'remove item,' 'get count' — and the tool sequences them randomly, checking invariants after each step. Luna: So it's like chaos engineering for your data layer. Lucas: Exactly. For example, you might define an invariant: 'the count of items should always equal the number of added items minus removed items.' The tool will interleave add and remove operations in random orders, including duplicates, empty removals, and so on. It finds race conditions and state leaks that unit tests never would. Luna: Have you seen teams use that in practice? Lucas: A payments team I consulted for used it on their transaction processing pipeline. They found a bug where if a refund was processed within the same millisecond as the original charge, the state machine would double-count the refund. That's a timing issue you'd never catch with example tests. Luna: And the bug had been there for months? Lucas: Over a year, actually. But it only triggered under very specific timing conditions, so it rarely happened. The property test triggered it within minutes. Luna: That's the power of random exploration. Lucas: Right. And once you start thinking in properties, you see opportunities everywhere. Even simple utility functions benefit. For example, a function that truncates a string to a maximum length — property: the output length is always less than or equal to the maximum, and the output is always a suffix of the input. Luna: That's a simple but powerful check. Lucas: And it's easy to write. The hardest part is the initial mental shift. Once you've written a dozen properties, it becomes natural. Luna: So for someone listening who wants to try it today, what's the first step? Lucas: Install Hypothesis if you're in Python, or fast-check for JavaScript. Pick a function you trust — maybe a date parser or a URL builder. Write a property that says 'the output should be parseable back to the original input' or 'the output should match a known format.' Run it. You'll either gain confidence or find a bug. Either outcome is valuable. Luna: And if you find a bug, you get to tell your team you found it with a hundred random inputs. Lucas: Exactly. And you'll have a minimal failing case to show them. It's a great conversation starter. Luna: Alright, I'm convinced. I'm going to try it on our invoice generator this week. Lucas: Let me know what you find. I'd bet you discover at least one edge case you hadn't thought of. Luna: I'll report back. Lucas: That's the thing about property-based testing — it doesn't replace your judgment, it amplifies it. You think about what should be true, and the machine does the brute-force checking. That combination is hard to beat. Luna: It's like having a tireless intern who tests every possible input while you sleep. Lucas: Exactly. And unlike interns, it doesn't complain about the boring work.