Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Data Quality Matters More Than Test Coverage
Transcript
- Lucas: So here's a number that gets thrown around a lot in QA circles: 'We need 80 percent code coverage.' And I get it—it's a nice target. But I think it misses the point. Luna: Right, because you can have 80 percent coverage and still ship a buggy release. I've seen it happen. Lucas: Exactly. I've been looking into a study that found roughly 70 percent of test failures in enterprise software are caused by bad test data, not bad code. That's a huge number. Luna: Seventy percent? That's wild. So we're chasing coverage metrics while ignoring the data that our tests run against. Lucas: Precisely. And a perfect example of this is the CrowdStrike outage from July 2024. You probably remember it—a single faulty content update brought down 8.5 million Windows systems worldwide. Luna: Oh, that was massive. Airlines, hospitals, banks—all stopped. But that was a bug in the update itself, not test data, right? Lucas: On the surface, yes. But when you dig into the root cause analysis CrowdStrike published, a key factor was that their test data didn't properly represent the range of real-world system configurations. The update passed testing because the test environment ran a narrow set of data. Luna: So the test data was too clean, too uniform. It didn't catch the edge case that triggered the failure in production. Lucas: Bingo. This is the classic case of 'garbage in, garbage out' for test data. You can have a thousand test cases, but if they all run on the same pristine dataset, you're not really testing anything useful. Luna: So what's the alternative? Do we need to mirror production data exactly? That seems like a compliance nightmare. Lucas: It is—especially with GDPR, HIPAA, and privacy laws. The trick is synthetic data generation. You create artificial datasets that have the same statistical properties as production data—the same distributions, the same anomalies—but no real personal information. Luna: So you get the realism without the risk. But is synthetic data actually good enough to catch those edge cases? Lucas: It can be, if you build it right. The key is to model the variability you see in production. For example, if your production data has 5 percent null values in a certain field, your synthetic data should too. If it has outliers—say, a customer with a million-dollar transaction next to one with a fifty-cent transaction—your test data needs that spread. Luna: That makes sense. But I imagine a lot of teams just take a production database dump and anonymise it. Is that enough? Lucas: It's a start, but it has pitfalls. One issue is data volume—you can't run your full test suite against a 10-terabyte production database every time. That's where data subsetting comes in: you take a representative slice, maybe 5 percent of the data, but you ensure it preserves the key relationships and distributions. Luna: So you shrink the dataset but keep the shape. That sounds like a good compromise. Lucas: Right. And you also need data versioning. Just like you version your code, you should version your test datasets. Because when a test fails, you need to know exactly what data it ran against. Was it last month's data? This week's? Did you change the distribution? Luna: That's a great point. I've definitely been on teams where a test fails, and we spend hours debugging only to find out the test data had changed underneath us. Lucas: Exactly. Data versioning gives you reproducibility. Without it, flaky tests often aren't really flaky—they're just reacting to shifting data. Luna: So what's the first step for a team that knows they have a test data problem? Where do you start? Lucas: I'd say start by auditing your test data quality. Look at the last 50 test failures. How many were caused by data issues versus logic bugs? If it's anywhere near that 70 percent figure, you have your answer. Luna: Then invest in synthetic data generation tools. There are open-source options like Datafaker, and commercial ones too. Lucas: Yeah, and pair that with data masking for any production data you do use. The combination gives you realistic, safe, versioned datasets that make your test suite actually meaningful. Luna: It's funny, we talk so much about test coverage percentages, but if the data is bad, the coverage is a mirage. Lucas: Exactly. And that's the takeaway: focus on data quality first, and coverage will follow naturally. Luna: Speaking of making testing meaningful, Lucas, we've been having these conversations every week, and I know a lot of listeners really get value from them. If you find this show useful, one way to keep it going is to support it—just like you'd version your test data, a little consistency goes a long way. Lucas: That's a good parallel. And it's easy—head over to buy me a coffee dot com slash fexingo. Whatever you feel is fair. It helps us keep the show ad-free and focused on practical advice. Luna: And we really appreciate it. Now, back to test data. One thing we haven't touched on is how often you should refresh your test datasets. Lucas, what's your take? Lucas: I'd say at least once per release cycle, ideally more often if your production data changes significantly. For SaaS products, that could be every two weeks. The goal is to keep your test data aligned with the real-world patterns your users are generating. Luna: So it's not a one-and-done thing. You have to treat test data as a living asset. Lucas: Exactly. And that's the mindset shift that separates teams who ship confidently from teams who are always fighting fires. Luna: Alright, I'm convinced. Next sprint, I'm putting test data hygiene on the backlog. Lucas: Do it. And you'll probably find that your flaky test rate drops faster than any coverage metric ever could.