Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Data Quality Is the Hidden Cost in CI Pipelines
Transcript
- Lucas: So, Luna, here's a question for you. When was the last time you debugged a test that failed only on certain days of the week, and the root cause turned out to be the data, not the code? Luna: Oh, that's painfully specific. I once had a test that passed on Monday, failed on Wednesday, and nobody could figure out why. Turned out the test data had a date-dependent field that expired mid-week. Lucas: Exactly. And I think that's the story we want to dig into today, because test data quality is one of the most underrated factors in how fast and reliable your release pipeline actually is. It's not as glamorous as writing clever test cases, but it can quietly sink your whole CI. Luna: And if today's conversation gives you a framework for spotting these problems in your own setup, that's exactly the kind of thing that makes this show worth your time. The reason we can keep doing this ad-free is listener support, so if you found today useful, you can buy us a coffee at buy me a coffee dot com slash fexingo. Lucas: Yeah, that genuinely helps us keep going. And like Luna said, we're not trying to sell you anything, just a way to keep the show alive if it's adding value. Lucas: So, let's get into it. There was a mid-sized fintech company, let's call them FinPay, that was struggling with a 30 percent flaky test rate. Developers were spending hours re-running builds, and trust in the suite was evaporating. Luna: Thirty percent is brutal. That's not just an annoyance, that's a cultural problem where people start ignoring red signals. Lucas: Right. And when they finally audited their test databases, they found three main issues. First, the schemas had drifted from production. Second, the data was full of PII, which created compliance headaches. And third, they were using a 'golden dataset' that had been hand-crafted years ago and had no relation to how the system actually behaved in production. Luna: The golden dataset trap. I've seen that. It looks comprehensive, but it's like testing against a museum exhibit instead of the real world. Lucas: Exactly. So FinPay's fix wasn't to write more tests. They did a three-week data hygiene sprint. They masked all the PII, they built a pipeline to refresh subsets of production data nightly, and they started generating synthetic data for the edge cases that production doesn't have enough of. Luna: And what happened? Did the flakiness actually drop? Lucas: It dropped by 40 percent within a month. And the cost savings were substantial. Developers stopped spending hours on re-runs, and the CI pipeline became a signal again, not a source of noise. Luna: So, what's the mental model you want listeners to take away? What's the first sign that test data is the problem? Lucas: The first sign is tests that fail in a non-deterministic way. If a test passes in isolation but fails when run with others, or fails only on certain days, or only in the CI environment, the code might be fine but the data is not. Another big clue is when a test starts failing after a production schema change, because your test data wasn't updated to match. Luna: And schema drift is so sneaky. You deploy a new field to production, but the test database doesn't have it, so tests might pass because they're not exercising the new code path, or they fail with a cryptic error. Lucas: Precisely. And that's why treating test data as a versioned artifact is so important. Just like you version your code, you should version your seed data and your data migrations. If a new column is added, the test data should be updated in the same commit, ideally in the same repository. Luna: So, what about the PII issue? How do you mask data without losing its realism? Lucas: That's the fun part. You can use deterministic masking, where you replace names and emails with fake but consistent values, so relationships still hold. For example, every occurrence of 'John Smith' becomes 'User 1234', but it's the same user across all tables. There are tools that do this automatically, but even a simple custom script can go a long way. Luna: And what about synthetic data? When is that better than production data? Lucas: Synthetic data is great for edge cases that production just doesn't have. Think about testing a payment system: you want to test a transaction with a penny, a transaction with a huge amount, a transaction with a specific currency. Production might not have those naturally, but synthetic data lets you create them on demand. Luna: Right, but you still need production data for realism, because synthetic data can miss the weird correlations that real users create. Lucas: Exactly. So the best approach is a hybrid. Use production subsets for broad coverage, and synthetic data for targeted edge cases. FinPay ended up with a nightly job that pulls a random but representative subset of production, masks it, and loads it into the test environment. That way, the test data is always fresh and close to what real users see. Luna: And that freshness is crucial, because stale data can cause all kinds of subtle issues. A test that expects a certain number of records might fail if the production data has grown or shrunk. Lucas: Right. And that's why you need to be careful with assertions that depend on exact counts. Better to assert on relative conditions, like 'at least one' or 'the count increased by one'. That makes tests less brittle to data changes. Luna: So, let's talk about the 'golden dataset' again. Why do teams fall into that trap? Lucas: Because it feels safe. You have a dataset that you know passes all tests, so you're afraid to change it. But it becomes stale, and it doesn't reflect production anymore. The fix is to treat test data as a living thing, not a fixed artifact. You need to regularly regenerate it, and you need to have a process for when it changes. Luna: And that requires a cultural change, which is often the hardest part. Lucas: Absolutely. But the payoff is huge. When FinPay made test data a first-class citizen, they not only reduced flakiness, but they also sped up their release cycle. They were shipping new features in a week instead of two, because they weren't constantly fighting the test suite. Luna: That's a great outcome. So, for someone listening who wants to start tackling test data quality, what's the first step tomorrow morning? Lucas: Start by auditing your test data. Look at how it's created, how it's stored, and how it's refreshed. Ask yourself: is it versioned? Is it masked? Is it fresh? And do you know what's in it? You might be surprised at what you find. And then pick one test data problem and fix it, even if it's just adding a seed data migration to your codebase. Luna: It sounds like a small step, but it can have an outsized impact on your team's sanity. Lucas: Yeah, and that's the takeaway. Test data quality isn't glamorous, but it's the foundation of a reliable test suite. If you ignore it, you'll keep paying the cost in flaky tests and lost trust. If you invest in it, your CI pipeline becomes a true safety net instead of a sieve. Luna: And that's a trade worth making, I think. Lucas: Agreed. And maybe the next time a test fails on a Tuesday, you'll know exactly where to look.