Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Property-Based Testing Catches the Bugs You Miss
Transcript
- Lucas: So Luna, let me ask you something. When was the last time you wrote a test and thought, 'I'm pretty sure this covers the important stuff,' and then a bug slipped through anyway? Luna: Honestly? Last week. I had a test for a date formatter, covered the happy path, a leap year, even a time zone edge case. Then someone passed an invalid date string, and boom, it crashed. Lucas: Right, and that's the classic problem with example-based testing. You're testing the cases you thought of. But the universe has a way of throwing cases you didn't. That's where property-based testing comes in. Luna: We've touched on it in a few episodes, but I think we've never really given it a full episode. Let's fix that. Lucas: So the idea is simple: instead of writing a test with a specific input and expected output, you define a property that should hold for all inputs. Then a tool generates hundreds of random inputs and checks the property for each one. Luna: Give me a concrete example. I'm a visual thinker. Lucas: Okay, let's say you have a function that calculates the total price of a shopping cart. A traditional test might check that with two items at ten dollars each, you get twenty dollars. Good. But what if the function can accidentally return a negative total when the discount exceeds the sum of the items? Your example test would never catch that, unless you specifically thought of it. Luna: But a property-based test would check, for any list of prices and discounts, the total is always greater than or equal to zero. Lucas: Exactly. You define that property, and the tool generates random carts with random numbers, including extreme values, negatives, zeros, even empty carts. And if it finds a violation, it shrinks the input to the simplest case that still fails, so you get a minimal example to debug. Luna: That shrinking part is huge. It's what makes property-based testing practical. Otherwise you'd be staring at a huge random input and have no idea where to start. Lucas: Absolutely. The first time I used a property-based testing library, I was blown away by how quickly it found bugs that my example tests had missed for months. One time it found a bug in a sorting function where, for certain inputs, the output wasn't actually sorted. My example tests just didn't cover that combination. Luna: So how do you actually get started? What's the learning curve like? Lucas: There are libraries for almost every language. QuickCheck in Haskell started it all, but there's Hypothesis for Python, fast-check for JavaScript, and they all have a similar workflow. You define generators for your input data, write properties, and run the test. The library does the heavy lifting. Luna: And the properties, how do you come up with them? That seems like the hard part. Lucas: It's actually a shift in thinking. You start looking for invariants, things that should always be true. For a function that reverses a list, the property is that reversing twice gives you the original list. For an addition function, it's that addition is commutative: a plus b equals b plus a. Luna: So you're testing the essence of the function, not just a few cases. Lucas: Exactly. And that's why it's so powerful. You're not limited by your imagination. The tool will find edge cases you never thought of, simply because it can explore the input space much more thoroughly than you can by hand. Luna: I remember reading about a team that used Hypothesis to test their API's JSON parser. They found bugs that only occurred with deeply nested structures or weird Unicode characters. Those are things you'd never write a test for manually. Lucas: Right. And that's the beauty of it. But there's a learning curve. You have to be careful not to write properties that are too trivial, like 'the function returns an integer.' That's useless. You want properties that actually capture the intended behavior. Luna: So what's a good property versus a bad one? Give me some guidance. Lucas: A good property is one that would fail if the function had a bug. For example, for a function that filters a list by a predicate, the property could be that every element in the output satisfies the predicate, and every element in the input that satisfies the predicate appears in the output. That's strong. Luna: And a bad property? Like 'the output is a list'? Lucas: Exactly. That's a tautology. It's always true unless you have a severe type error. You want properties that are specific enough to catch real bugs. Luna: One thing I've seen is people using property-based testing to test their own test data generators. Like, you generate a random object, serialize it, deserialize it, and check that you get the same object. That's a great property. Lucas: Absolutely. That's a classic: round-trip property. It's used everywhere, from serialization to database writes. And it's a perfect example of a property that's hard to test with examples, because you'd have to think of all the different shapes your data can take. Luna: Now, I know a lot of listeners might be thinking, 'This sounds great, but how do I sell this to my team?' Or 'How do I integrate it into our existing test suite?' Lucas: That's a fair point. It's not about replacing your existing tests. It's about adding a new layer. You still need example tests for specific, known scenarios. Property-based tests complement them by covering the unknown. Luna: So it's like having a safety net underneath your tightrope. Lucas: Exactly. And you can start small. Pick a function that's pure, with clear inputs and outputs, and write a few properties for it. Run it in your CI pipeline. Once people see the value, they'll want to use it more. Luna: I also think it's a good idea to run property-based tests with a higher number of examples than the default. Sometimes the default is like a hundred, but you can bump it to a thousand or more for a more thorough search. Lucas: Good point. And you can also use the same library to generate more complex data structures, like trees or graphs, which can be really useful for testing algorithms. Luna: Before we go deeper, I just have to say, I'm really enjoying this conversation. It's one of those topics that makes you feel like you're leveling up as a developer. Lucas: I know what you mean. And honestly, if today's conversation gave you something you can use, you know what we always say. If it was worth a coffee to you, that's the link. Buy me a coffee dot com slash fexingo. Luna: That's the smallest ask we can make, and it genuinely keeps this show ad-free. No pressure, just if it felt useful. Lucas: Right, so back to it. Another powerful use case for property-based testing is testing your own test fixtures. Instead of manually crafting fake data, you can generate it on the fly, ensuring your tests always have fresh, varied data. Luna: That's a great point. And it ties into what we talked about in the data diversity episode. Property-based testing gives you diversity by default. Lucas: Exactly. And there's a common pitfall: people sometimes write properties that are too tightly coupled to the implementation. For example, if you test that a function calls a certain internal method, that's not a property; that's a mock. Property-based testing works best at the behavior level. Luna: So you're testing what the function does, not how it does it. Lucas: Exactly. And that makes your tests more robust to refactoring, which is a huge win. Luna: Now, a lot of teams worry that adding property-based tests will slow down their CI. Is that a real concern? Lucas: It can be, if you're not careful. Running a thousand iterations for a complex function can take a few seconds. But you can mitigate that by running the full suite nightly and a smaller set in CI. Or you can use seeds to reproduce failures deterministically. Luna: Ah, so you can save the random seed when a failure occurs, and then re-run with that seed to debug. That's clever. Lucas: Yeah, most libraries do that automatically. They show you the minimal failing example and the seed, so you can replay it. That's a huge time saver. Luna: So, for someone who's never tried property-based testing, what's the very first step you'd recommend? Lucas: Pick a small, pure function in your codebase that you're a little suspicious about, or one that's been buggy in the past. Write a property that captures its intended behavior. Run it. See what happens. I'm willing to bet you'll find a bug within the first few runs. Luna: That's a bold claim. But I've seen it happen. It's like the tool is reading your code's mind. Lucas: It's a shift in mindset, but once it clicks, it's hard to go back to pure example tests. You start seeing properties everywhere. Luna: I think our listeners are going to have a lot of fun with this. Thanks for breaking it down, Lucas. Lucas: Anytime. And remember, the goal isn't to replace your existing tests, it's to make your suite smarter. Happy testing.