Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Property-Based Testing Exposes Hidden Bugs
Transcript
- Lucas: So, Luna, have you ever written a test, watched it pass, and then a few weeks later the exact same function blows up in production with an input you never even thought of? Luna: Oh, all the time. I remember one time I tested a date parsing function with like five different formats, and then the API sent me an ISO string with timezone and my code just melted. Lucas: Right. And that's the classic problem with example-based testing. You write a handful of specific inputs and outputs, and you hope they cover the important cases. But there's an entire universe of inputs you never considered. That's where property-based testing comes in. Luna: I've heard the term, but I always thought it was some academic thing that's too abstract to actually use. Lucas: It sounds academic, but the core idea is actually simple. Instead of writing a test with a specific input and a specific expected output, you define a general rule—a property—that should hold true for any valid input. Then you let a tool generate hundreds or thousands of random inputs and check that the property holds for all of them. Luna: So instead of testing that 'two plus two equals four', you'd test that 'addition is commutative' and let the tool throw random numbers at it? Lucas: Exactly. And the beautiful part is that when one of those random inputs causes a failure, the tool doesn't just stop. It shrinks the input down to the smallest possible case that still triggers the bug. That's called shrinking, and it gives you a minimal reproduction case that's actually useful for debugging. Luna: Okay, that does sound powerful. But can you give me a real example where this caught something that regular tests missed? Lucas: Sure. I was talking to a QA engineer at a fintech company a few months ago. They had a currency conversion module that was thoroughly tested. Unit tests, integration tests, you name it. Everyone was confident it was solid. Luna: And then? Lucas: And then a trader noticed that a large conversion involving a tiny currency pair was off by a few cents. Not a huge deal on its own, but in high-volume trading, those cents add up. They traced it to a rounding issue. The module was using a floating-point representation for the exchange rate, and for certain extreme rates, the rounding error propagated through the calculation. Luna: Ah, the classic floating-point trap. Lucas: Right. Their test suite had cases with normal rates like one point two, zero point eight, that sort of thing. But when a property-based tester started generating random rates across a huge range, it found a failing case almost immediately. The shrinking process then reduced it to something like one point zero three four six times ten to the minus seven, which is a ridiculously small rate, but real in some markets. Luna: So the property they defined was something like 'for any two currencies, converting from A to B and then back to A should return the original amount'? Lucas: Exactly. That's the kind of invariant that should always hold, and it exposed the rounding flaw. Luna: That's a fantastic example. And it's not just about floating point. I imagine property-based testing can reveal edge cases in all sorts of logic, like string manipulation, list sorting, you name it. Lucas: Absolutely. For instance, you might test that reversing a list twice gives you back the original list. Or that sorting a list always produces a list where every element is less than or equal to the next one. Or that parsing a date and formatting it back yields the original string for any valid date. Luna: Those are all great properties. But I have to ask—how do you actually get started? Is it a huge overhead to write these properties? Lucas: It can be, if you try to convert your entire test suite overnight. But the practical approach is to start small. Pick your most critical modules, the ones where a bug would be catastrophic, and write properties for the core invariants. You don't have to abandon example-based tests entirely—in fact, they complement each other. Luna: So it's more of a layered defense. Lucas: Exactly. Example-based tests document specific behavior, like 'this function should return zero when the input is negative'. Property-based tests verify the broader rules, like 'this function should never return a negative result for any non-negative input'. Together, they give you a much stronger safety net. Luna: And the tooling has matured a lot, right? I know there's QuickCheck for Haskell, but what about more mainstream languages? Lucas: Oh, there are great libraries everywhere. There's Hypothesis for Python, fast-check for JavaScript and TypeScript, and property-based testing is built into many functional languages like Scala and Elixir. There's also jqwik for Java and FsCheck for.NET. So no matter your stack, you probably have a solid option. Luna: That's good to know. And I'm guessing the learning curve is real, but the payoff is worth it. Lucas: Definitely. I've seen teams that adopt property-based testing report a significant drop in production bugs related to edge cases. It's not a silver bullet, but it's a powerful tool that catches a whole class of bugs that humans are just terrible at predicting. Luna: You know, this conversation reminds me of something. We spend so much time on this podcast talking about testing techniques that genuinely help our listeners, and we do it without any ads. That's something I'm really proud of. Lucas: Yeah, we've always kept the show ad-free. It's our way of keeping the advice clean and focused. If you find these episodes useful and want to support that, there's a small way to help—you can visit buy me a coffee dot com slash fexingo. Just a simple way to keep things running. Luna: And honestly, even if you never do, we're just glad you're here. Now, back to property-based testing—one thing I'm curious about is how it handles stateful systems. Like, can it test more than just pure functions? Lucas: Great question. Yes, there are extensions for stateful testing. For example, you can model a system as a state machine and generate sequences of operations, checking that invariants hold after each step. That's especially useful for testing databases, caches, or even entire microservices. Luna: So you could test that a user's balance never goes negative after a series of deposits and withdrawals, as long as the total withdrawals don't exceed the balance. Lucas: Exactly. You'd generate random sequences of valid operations and verify that invariant after every single one. It's like chaos engineering for your business logic, but deterministic and fast. Luna: That sounds incredibly valuable. I'm starting to think we should integrate this into our own test strategy. Lucas: I'd say start with a small pilot. Pick one module that's complex but well-understood, write a few properties for its core invariants, and run them in CI. You'll quickly see if it's worth expanding. Luna: And for teams that are already overwhelmed with testing, is there a risk that property-based tests become flaky or slow? Lucas: They can be, if you're not careful. The key is to control the number of generated cases and use a fixed seed when you want reproducibility. Most libraries let you configure the sample count and the randomness source. And as for flakiness, since they're deterministic given a seed, they're actually less flaky than example-based tests that depend on external data. Luna: That's a good point. The determinism is a big plus for CI. Lucas: Yeah, and when a property test does fail, the shrinking gives you a precise, minimal case that you can add to your example-based suite as a regression test. So it's a self-improving system. Luna: That's a neat feedback loop. You get a bug, you shrink it, you add it to your regular tests, and your suite gets stronger over time. Lucas: Exactly. It's like having an automated QA engineer that never gets tired and always thinks of the weirdest inputs. Luna: So, to wrap up, what's the one takeaway you want listeners to remember about property-based testing? Lucas: I'd say this: if you're still testing with only hand-picked examples, you're leaving a lot of bugs on the table. Property-based testing forces you to think about the rules that should always hold, and that's a higher level of testing that catches the things you don't even know you don't know. Luna: That's a great way to put it. And I think it's a skill worth investing in, even if it feels a bit abstract at first. Lucas: Absolutely. Once you get the hang of it, you'll wonder how you ever tested without it.