Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Testing Your Test Data with Property-Based Testing
Transcript
- Lucas: So, Luna, we've talked a lot about testing strategies in recent episodes, from mutation testing to contract testing, but today I want to dig into something that often feels like a secret weapon for the teams that actually use it. Property-based testing. Luna: Property-based testing. I've heard the term, but I have to admit, I'm a little fuzzy on what it actually means versus, say, just writing more test cases. Lucas: That's exactly the right question. So, the traditional way we test is example-based. You write a test, you feed it a specific input, you assert a specific output. That works, but you're only testing the cases you thought to write down. Property-based testing flips that. Instead of you choosing the inputs, the framework generates hundreds or thousands of random inputs for you, and you define the property that must hold true for all of them. Luna: Okay, so instead of testing that a specific addition works, you're testing that, for any two integers, the sum is commutative. Lucas: Exactly. And the beauty is, these frameworks shrink failing cases down to a minimal example, so you're not drowning in a sea of random noise. You get one tiny, clean input that breaks your code. Lucas: Let me give you a concrete example that I think really sells it. Imagine you're writing a sorting function. With example-based testing, you might test an empty list, a single element, a reverse-sorted list, a list with duplicates. But what about a list with a million elements? Or a list with negative zeros? Or Unicode strings? Luna: Right, and if you're a developer, you're probably not going to think of all those edge cases off the top of your head. Lucas: Exactly. And that's where property-based testing shines. You define properties like: for any list of integers, the sorted version has the same length as the original, its elements are in non-decreasing order, and it contains the same elements as the original. Run a thousand generated lists through that, and you'll catch subtle bugs you never would've thought to write manually. Luna: So it's like having a tireless intern who throws every possible input at your code, and then politely tells you which one broke it. Lucas: Ha, exactly. And for this episode, I want to focus on a particular case that really shows the value. A few years back, a payment processing team I know was dealing with a bug where a transaction dated December 31st would get processed as January 1st of the following year, but only for a specific timezone and only when the transaction happened at exactly 11:59 PM. They had tests for the dates, but they'd only tested a handful of calendar dates, not the full range of timezone offsets combined with leap years. Luna: Ooh, that's a classic timezone and date bug. So, they used property-based testing to catch that? Lucas: They did. They defined a property that said: for any valid timestamp, the local date representation should match the expected date in that timezone, accounting for daylight saving time. The framework generated random timestamps across multiple timezones, including the edge cases like the exact transition moments, and it found that specific December 31st case in seconds. Luna: That's a great story, but I imagine property-based testing can be intimidating to get started with. What tools are out there? Lucas: Yeah, the good news is, they're pretty mature now. There's QuickCheck for Haskell, which started it all, and then it's been ported to just about every language. In the Java world, you've got jqwik; in JavaScript, fast-check; in Python, Hypothesis is really popular. And they all work more or less the same way: you define a property, you tell it what kinds of inputs to generate, and it does the rest. Luna: Right, and I've actually used Hypothesis a bit. It's pretty straightforward once you get the hang of it. Lucas: Exactly. And the framework even has a feature that saves and replays failing tests, so you're not chasing flaky failures. Luna: That's handy. So, is property-based testing a replacement for example-based testing, or just a complement? Lucas: It's definitely a complement. Example-based tests are great for documenting intended behavior and for regression checks. Property-based tests are great for finding unexpected edge cases and for validating complex logic. I'd say every test suite should have some of both. Lucas: And honestly, if today's conversation gave you something usable, that's the link — buy me a coffee dot com slash fexingo. It's the smallest ask, and it keeps this show ad-free. Luna: Yeah, and it really does help us keep doing these deep dives. Every coffee counts. Lucas: Anyway, back to property-based testing. One thing that trips people up is thinking in properties, not examples. So let's talk about how to identify good properties. Luna: Okay, I need to hear this. Because I feel like I'd struggle to come up with properties. Lucas: There are a few common patterns. One is the round-trip property: if you encode something and then decode it, you should get back what you started with. Another is the inverse property, like sort and then reverse-sort. There's also the invariant property, like the length of a list after a filter operation. And there's the idempotence property, where running the same operation twice gives the same result as running it once. Luna: Oh, that's really helpful. So, for example, if I had a function that parses a date from a string, the round-trip property would be: parse the string, format the date, and you get the same string back. Lucas: Exactly. And that's a perfect property for that payment date bug. They could have defined a property that says: given any timestamp, converting to local time and then back to UTC should yield the original timestamp. That would've caught the timezone issue immediately. Luna: That's a beautiful way to think about it. So, what about performance? Does property-based testing slow down your test suite a lot? Lucas: It can, if you're not careful. That's why most frameworks let you set the number of generated cases. You might start with a hundred for a quick check, then run thousands in your CI nightly. And you can shrink the generated data to minimal examples, which helps with debugging but doesn't directly speed things up. Luna: Gotcha. So, is there a risk of generating inputs that are invalid for your domain? Lucas: There is, but most frameworks let you define custom generators and constraints. For example, if you're testing a function that only accepts positive integers, you can tell the framework to generate only positive integers. And you can also use filters to skip invalid inputs, though it's better to generate valid ones directly to avoid wasting cycles. Lucas: Let me give you one more real-world example. I remember a team that was building a shopping cart system. They had a function that calculated the total price of items, applying discounts. They had a few example-based tests: a cart with one item, a cart with a 10% discount, a cart with free shipping. But they missed a bug where if you had a negative quantity, the total price would go negative. Luna: How would property-based testing catch that? Lucas: They defined a property that said: for any list of items with positive quantities and prices, the total should be non-negative. The framework generated a list that included a negative quantity because they hadn't constrained the generator to only positive values, and it caught the bug right away. Luna: So, the framework generated an invalid input, and that actually led to finding a bug, but it's a bug in the sense that the function should have handled it gracefully, even if the input was invalid. Is that right? Lucas: Exactly. It's about robustness. Your function should either handle the invalid input gracefully or throw a clear error, not silently return a negative total. Luna: That makes a lot of sense. So, how can our listeners get started with property-based testing in their own projects? Lucas: I'd say start small. Pick a pure function that has clear properties, like a sorting algorithm or a string manipulation. Write a few example-based tests first to document the expected behavior, then add a property-based test that generates random inputs. Run it locally, see what it finds. You might be surprised at how many edge cases you've been missing. Lucas: And once you're comfortable, you can start applying it to more complex business logic, like the payment date validation we talked about. But the key is to practice thinking in properties. It's a bit of a mind shift, but it's incredibly powerful. Luna: I love that. So, if someone wants to learn more, are there any resources you'd recommend? Lucas: There's a great book called 'Property-Based Testing with PropEr, Erlang, and Elixir' by Fred Hebert, but it's not language-specific in its concepts. And the Hypothesis documentation is excellent if you're in Python. Also, just search for 'quickcheck properties' and you'll find plenty of blog posts. Luna: Perfect. Well, I think today's episode gave us a solid foundation. I'm definitely going to try adding a property-based test to my next project. Lucas: And that's the takeaway, really. The next time you're writing a test and you find yourself listing edge cases manually, stop and ask: what's the property here? Because the property will generate the edge cases for you.