Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Data Entropy Exposes Hidden Bugs
Transcript
- Lucas: So you run your test suite, it passes, you deploy — and then a customer calls because their payment was rounded down by a penny. That penny, across a million transactions, is ten thousand dollars. Luna: And the test suite said everything was fine. That's the classic gap — test data that doesn't reflect reality. Lucas: Right. And the gap has a name I want to talk about today: test data entropy. Specifically, how the randomness and variety in your test inputs can expose bugs that structured, curated test data misses. Luna: Entropy. Like the physics concept — disorder, randomness. So we're saying most test data is too ordered? Lucas: Exactly. Most teams build test data that's clean, predictable, and repeatable. First name 'John', last name 'Smith', amount '$100.00'. That's low entropy. And it's fine for basic sanity checks. But production data is messy — names with apostrophes and hyphens, amounts like $99.9999, timestamps from all time zones. Luna: And that mess is where bugs live. So how do you measure entropy in test data? Lucas: You can use a simple Shannon entropy calculation. Shannon entropy, named after Claude Shannon, measures the information content — essentially how unpredictable each byte is. For a set of test inputs, you calculate the probability distribution of characters or values. Luna: Give me an example. Low entropy versus high entropy. Lucas: Sure. Suppose you have a field for transaction amount. Low entropy: every value is a whole dollar amount, like 100, 250, 500. The characters are only digits and occasionally a decimal point. Shannon entropy might be around 2.5 bits per character. High entropy: amounts like 99.9999, 0.01, 12345.67, and also negative numbers, very large numbers, zero. The character set expands — more digits, minus signs, more decimal points. Entropy might be 4 bits per character or higher. Luna: And the bug lives in the high-entropy region. I'm thinking of a real case — a startup that processed payments, they had a rounding error that only appeared when the amount had more than two decimal places. Lucas: That's exactly the kind of story I wanted to bring up. I talked to a QA lead at a payment processing startup — let's call it PayFlow — back in 2024. They had a test suite with a thousand test cases, all using amounts like $10.00, $25.50, $100.00. Standard stuff. Their code used a decimal type internally, but a third-party API expected a float. The conversion truncated after two decimal places. Luna: So $99.9999 became $99.99? Or maybe $100.00? Lucas: It rounded down to $99.99. The difference was $0.0099 per transaction — less than a cent. But they processed about a hundred thousand transactions a day. That's almost a thousand dollars a day lost. Luna: And their test suite didn't catch it because every transaction amount in tests had exactly two decimal places. So the truncation was invisible. Lucas: Right. The bug existed for six months. Then a new QA engineer joined and, on a hunch, added random amounts generated with a script that produced values with up to six decimal places. The test failed immediately. Luna: That's the power of high-entropy test data. One script, one new test, and they found a bug that cost them roughly $180,000 over six months. Lucas: Exactly. So the question is: how do you systematically introduce entropy without drowning in noise? Because you don't want a test that fails randomly because of some weird edge case that doesn't matter. Luna: You need controlled entropy. Maybe you decide that for every test case, you have a small percentage of fields that use high-entropy data. Lucas: That's one approach. Another is to run a separate high-entropy test suite — not replacing your core tests, but complementing them. You take your existing test scenarios and parameterize them with data from a pool that has high entropy. Luna: So you don't change the logic of the test, just the inputs. And you measure the entropy of that input pool to ensure it's above a threshold. Lucas: Yes. And there's a tooling angle here. Some testing frameworks now have plugins that automatically generate high-entropy data. For example, property-based testing libraries like QuickCheck for Haskell or Hypothesis for Python are essentially entropy engines. They generate random inputs and check that certain properties hold. Luna: But property-based testing can be slow and hard to debug. A more lightweight approach is just entropy injection into existing parameterized tests. Lucas: Right. You don't have to rewrite your entire test suite. You can add a decorator or a data provider that mixes in high-entropy values for specific fields — like currency amounts, email addresses, zip codes. Luna: And you measure the entropy of that data? Like, you have a quality gate that says 'this data set must have Shannon entropy above 3.5 bits per character for string fields'. Lucas: Exactly. You can set a threshold. For numeric fields, entropy might be measured differently — based on the distribution of values, the presence of edge cases like zero, negative, very large, very small, NaN. The key is to be intentional. Luna: So let's talk about when entropy matters most. Financial calculations are one. What else? Lucas: Cryptographic functions — if you're hashing or encrypting, low-entropy test data might not exercise the algorithm properly. Also stateful systems like user sessions, where the ordering and timing of inputs matters. Luna: And any system that deals with user-generated content — because users will give you high-entropy data by default. Lucas: Right. The common thread is: if your production data has high entropy, your test data should too. Otherwise you're testing in a sanitized world that doesn't exist. Luna: I want to push back a little though. High-entropy tests can be flaky. If you generate random inputs, you might hit a bug that's actually a false positive — like a timing issue or a race condition that only appears with specific data. Lucas: That's a real concern. The solution is to separate your deterministic tests from your stochastic tests. Your deterministic tests — the ones that always pass — use curated data. Your stochastic tests — the ones that explore the input space — are allowed to fail, but you analyze the failures to decide if they're real bugs. Luna: So you don't gate your CI pipeline on stochastic tests passing every time. You run them as a separate stage, and only fail the build if a failure persists or matches a known pattern. Lucas: Exactly. Some teams run stochastic tests with a fixed random seed so they can reproduce failures. The seed is logged, and if a test fails, you can rerun with the same seed to debug. Luna: That's a good pattern. So entropy injection doesn't have to be chaotic. It can be controlled randomness. Lucas: Right. And the payoff is huge. At PayFlow, after they introduced high-entropy test data, they found three other bugs within the first month — a currency conversion error for certain exchange rates, a date handling bug for leap years, and a string truncation issue for long merchant names. Luna: All of which were in production for months. And they only found them because they added a little disorder to their tests. Lucas: So the takeaway is: audit your test data's entropy. If you're only using clean, curated data, you're leaving money on the table — literally, in some cases. Luna: And before we wrap, I want to mention something related. The reason we can have conversations like this — diving into niche testing techniques without worrying about sponsors — is because of listeners who support us directly. Lucas: Yeah, we keep the show ad-free on purpose. If you find these specific deep dives useful, you can support that choice at buy me a coffee dot com slash fexingo. It's not a membership, no perks, just a way to keep the signal clean. Luna: And it genuinely helps us justify spending time on episodes like this — entropy in test data — instead of chasing trending topics. Lucas: Okay, back to the tech. So if you want to try this yourself, start with one field. Pick a field that's numeric and has high entropy in production — like transaction amount, age, or distance. Generate a hundred random values that cover the real distribution, and inject them into one test. See what breaks. Luna: And measure the entropy before and after. You want to see at least a doubling in the Shannon score. Lucas: Then expand to other fields, one at a time. Within a sprint, you'll likely find a bug that's been there for months. And you'll wonder why you didn't do this sooner. Luna: I think that's a good challenge for our listeners. Next time your test suite passes but you suspect something's lurking, check the entropy of your test data. It might be too clean. Lucas: And that's the episode. Thanks for listening — see you next time.