Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Property-Based Testing Exposes Invisible Bugs
Transcript
- Lucas: So there's this bug that almost every payment system hits at some point — it's tiny, invisible, and if you only write example-based tests, you'll never catch it until it hits production. Luna: Ooh, I think I know where you're going. The rounding thing, right? Fractions of a cent? Lucas: Exactly. A payment processing API calculates fees as a percentage — say 2.9 percent plus thirty cents. For a $10 transaction, the fee should be $0.59. But what about $9.99? The math gives you 29.971 cents plus thirty cents, which is 59.971 cents. Do you round up or down? Floor? Ceiling? Banker's rounding? Luna: And each choice breaks some edge case. I've seen systems where over a million transactions, those fractions add up to thousands of dollars — either the platform loses money or users get overcharged. Lucas: Right. Now, a conventional test suite might check a handful of amounts — $5, $10, $100 — and everything looks fine. But the bug lives in the combinatorial explosion of inputs. That's where property-based testing comes in. Luna: Let me guess — instead of saying 'for this specific input, I expect this specific output,' you define a rule that should always hold true. Lucas: Exactly. You state a property — an invariant. For the payment example, the property might be: for any valid transaction amount, the fee should always be between zero and the transaction amount, and the total collected should equal the transaction amount plus the fee. Then the testing tool generates thousands of random amounts — including weird ones like $0.01, $999,999.99, negative numbers — and checks that the property holds. Luna: And it'll find the rounding edge case because it's going to generate something like $0.01 where rounding matters disproportionately. Lucas: Bingo. The framework even shrinks the failing input — it finds the simplest case that breaks the property. So instead of a $999,999.99 failing test, you get something like $0.01 with a specific rounding mode. That's the killer feature. Luna: I've used this in a scheduling app. We had a property: 'for any collection of timezone offsets and any meeting time, the meeting should always have the same UTC start time regardless of who creates it.' The property-based test caught a case where a half-hour timezone — like India Standard Time — was being treated as a full-hour offset in one code path. Lucas: That's a beautiful example. So you had a specific test for New York and London, but the half-hour zones would have slipped through until someone in Mumbai complained. Luna: Exactly. The property-based test generated random offsets from the IANA timezone database and found the mismatch in about five seconds. Lucas: The core insight is that example-based tests are necessary but not sufficient. They document expected behavior for known cases. Property-based tests explore the unknown. They're like a mathematician stating a theorem and then checking it with random proof assistants. Luna: So how does someone start? What's the easiest first property to write? Lucas: Pick a function that's pure — same input always gives the same output — and has a simple invariant. For example, a sorting function. The property is: for any list of integers, the sorted list should have the same length as the input, and every element should be less than or equal to the next. That's it. You can do it in Python with the Hypothesis library in about ten lines. Luna: And if it finds a failing case, you get a minimal reproduction. That's way better than a unit test that says 'expected but got ' because you learn the actual edge condition. Lucas: Exactly. And once you get comfortable, you can apply it to stateful systems — like a database migration. Property: after any sequence of inserts and deletes, the record count should match the number of inserts minus deletes. That catches race conditions and transaction rollback bugs. Luna: That's a lot more powerful than writing ten specific scenarios by hand. Lucas: It is. And it scales. The same technique works for APIs, UI state machines, and even hardware-software interfaces. QuickCheck, the original library, was invented for telecom systems in Erlang — they had to validate complex protocol state machines. It's been battle-tested for decades. Luna: I've heard people say property-based testing replaced their entire regression suite. Is that realistic? Lucas: Not entirely. You still need example tests for documentation and for known edge cases that the property generator might not hit — like null inputs or very large numbers. But property-based tests can cover combinatorial depth that would take thousands of manual examples. A good mix is maybe 80 percent property tests, 20 percent example tests. Luna: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that kind of support is what funds us making episodes like this one. Lucas: Yeah, it's a small group, but it keeps the show ad-free and lets us dive into topics like property-based testing without worrying about commercial breaks. We appreciate it. Luna: Anyway — back to properties. One practical tip: when you're starting, don't try to test everything. Pick one module, one function, one invariant. Run the property test as part of your CI. Once it's green, add another. Lucas: Great advice. And there's a hidden benefit: writing properties forces you to clarify your own understanding of what the code should do. You can't write a property if you're fuzzy on the invariant. It's like test-driven development on steroids. Luna: That's the thing — property-based testing isn't just a bug-finding tool. It's a design tool. When I wrote that timezone property, I realized our internal representation of time was inconsistent. We had two different data structures for storing offsets, and they didn't agree on half-hour zones. Lucas: That's the real win. You catch design flaws before they become bugs. And because the property is a precise, machine-checkable statement about your system, it becomes living documentation. A new developer can look at the property and understand the contract. Luna: So for someone listening who wants to try it today, what's the one concrete step? Lucas: Go to the Hypothesis documentation — it's for Python — and copy the example that tests a sorting function. Run it. Then change the function to have a subtle bug — like forgetting to handle negative numbers — and watch the test fail and shrink. That moment when you see the minimal failing input is when it clicks. Luna: I'd add: use it on a function that you know is correct first, just to see the green. Then introduce a bug. The shrinking is the magic. Lucas: Yeah. And if you're not in Python, there are ports for nearly every language — fast-check for JavaScript, ScalaCheck for Scala, fscheck for.NET. The concept is language-agnostic. Luna: So the takeaway: property-based testing fills the gap where example tests leave combinatorial blind spots. It's not a replacement — it's a complement. Lucas: And once you start thinking in invariants, you'll see opportunities everywhere. The payment rounding bug, the timezone mismatch — those are just the tip of the iceberg. There are a thousand similar bugs waiting in every codebase.