Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Property-Based Testing Finds Bugs You Never Thought Of
Transcript
- Lucas: You know that sinking feeling when a bug slips through code review, unit tests, integration tests — and then a customer reports it, and you immediately see what went wrong? Luna: Oh, absolutely. It's always something obvious in hindsight. Like a missing null check or an off-by-one. Lucas: Right. And often the root cause is an input pattern you never thought to test. You wrote tests for the happy path, maybe a few boundary values, but the real world sends data that doesn't fit your assumptions. That's where property-based testing comes in. Luna: Okay, I've heard the term, but let's ground it. What's the concrete difference from normal unit testing? Lucas: Normal unit testing is example-based. You write: 'given input X, I expect output Y.' You pick the examples. Property-based testing flips that — you define a property that should always hold true for any valid input, and then the tool generates thousands of random inputs to try to disprove it. Luna: So instead of saying 'this function returns 4 when I pass 2 and 2,' you say 'for any two numbers, addition is commutative — a plus b equals b plus a.' Lucas: Exactly. And the tool, like Hypothesis in Python or QuickCheck in Haskell, will throw random pairs at it — including zero, negative numbers, floats, very large numbers — and if any pair breaks the property, it reports the minimal failing input. That's the killer feature. Luna: I want a real example. Something messy that actually happened. Lucas: Sure. A team I know was building a CSV ingestion pipeline. They had unit tests for standard columns, tests for quoted fields, tests for empty rows. Everything passed. Then in production, a CSV from a client had trailing spaces on some values — 'USD ' instead of 'USD'. The parser silently dropped those rows because the column count didn't match. Luna: Ouch. Classic whitespace trap. And property-based testing would catch that how? Lucas: You'd define a property like: 'If I parse a CSV and then serialize it back, the round-tripped data should match the original.' That's an invariant. Then you ask Hypothesis to generate random CSVs — including ones with trailing spaces, missing headers, extra commas, weird quoting. It found the bug in under a minute. Luna: But wait — the round-trip property only works if your serializer produces the original format exactly. That's a strong assumption. Lucas: It is. You'd usually start with weaker properties. Like 'every parsed row has the same number of columns as the header.' Or 'for any row, re-parsing its string representation yields the same row.' Or 'if the CSV has N rows, the parser returns exactly N records.' Each property catches different kinds of failures. Luna: So you're not just writing one property. You're writing a small set of invariants that together define correctness. Lucas: Right. And the beauty is, you're forced to think about what must always be true. That mental shift is huge. Instead of 'does this test case pass?' you ask 'what am I assuming about the world?' And often your assumptions are wrong. Luna: Give me a non csv example. Something from, say, an API. Lucas: An API that accepts a date range. Property: 'the start date must always be before the end date.' Or 'if I request the same data twice within a minute, I get the same results.' Or for a pagination endpoint: 'the total count is the same regardless of page size.' Those are all properties. Luna: Okay, I see the appeal. But there's a cost. Running thousands of random inputs takes time. My unit tests finish in seconds. Lucas: That's the trade-off. Property-based tests are slower. You might run them in CI but not on every save. And debugging a failing test can be tricky — the input is generated, so you have to inspect the minimal counterexample. Hypothesis gives you a human-readable test case, but it still adds cognitive load. Luna: And flakiness? If the test passes most of the time but fails once in a thousand runs due to a rare edge case, you could get random CI failures. Lucas: That can happen. But Hypothesis actually has a feature called 'stateful testing' that replies the exact same sequence of inputs when it finds a failure. So it's reproducible. And you can seed the random generator to get deterministic runs. Luna: Okay, so it's not for everything. Where does it shine most? Lucas: Data pipelines, financial calculations, any system where you have a clear mathematical invariant. Think sorting — property: output is sorted, and output contains the same elements as input. Or a compression algorithm — property: decompressing a compressed input yields the original. Or a billing system — property: the sum of line items equals the total. Luna: Financial calculations are a great example. A rounding error that only occurs for specific currency amounts could hide for years. Lucas: Exactly. And property-based testing is especially good at finding those. It's like fuzzing but with a specification. Fuzzing throws random data at your program and checks it doesn't crash. Property-based testing checks that it doesn't crash AND that it behaves correctly. Luna: So the property is the oracle. Without one, you're just fuzzing. Lucas: Right. And writing a good property is hard. It takes practice. But once you get the hang of it, you start seeing properties everywhere. It changes how you design code — you write functions that are easier to describe invariants for. Luna: That's a nice side effect. Any gotchas for teams starting out? Lucas: Don't try to property-test everything. Start with one module that has clear inputs and outputs. Use Hypothesis's built-in strategies for common types. And pair it with example-based tests for the obvious cases. The combination is powerful. Luna: And I imagine the debugging story improves with practice. Once you've seen a few minimal counterexamples, you get faster at reading them. Lucas: Absolutely. The first time you see Hypothesis shrink a failing input from a thousand-character string down to just 'a ', you'll be hooked. It's like magic. Luna: Alright, I'm sold. I'm going to try it on our data validation layer. We've got a lot of 'must always be true' rules there. Lucas: That's a perfect candidate. And if you run into trouble, the Hypothesis documentation has great examples. Plus the community is very active. Luna: Speaking of things that are ad-free and community-supported — this podcast runs entirely on listener support. We don't have sponsors, no ads, just people who find value in these conversations. Lucas: Yeah, and it makes a real difference. If you've gotten something out of today's episode, consider throwing a couple of dollars our way at buy me a coffee dot com slash fexingo. It genuinely keeps these episodes coming. Luna: Even small contributions add up. It's what keeps us independent and focused on topics like this. Lucas: Anyway — back to property-based testing. One thing I want to emphasize is that it's not a replacement for traditional tests, but a complement. You still need example-based tests for clarity and regression. But adding a layer of property tests catches the things you didn't know you were missing. Luna: And it forces you to articulate your invariants. That alone improves code quality, even if you never run the test. Lucas: Exactly. So if you take one thing from today: next time you write a function, ask yourself 'what must always be true?' Then write that as a property. Luna: Good advice. And if you've tried property-based testing, we'd love to hear your stories — especially the weirdest bug it uncovered. Lucas: We're on socials at Fexingo. Until next episode, keep testing your assumptions.