Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Synthetic Test Data Beats Production Data
Transcript
- Lucas: So you're building a test suite, and you need realistic data. The obvious move is to grab a copy of production — a few million rows, real customer names, real transaction amounts, real addresses. And that is exactly what gets companies sued. Luna: And not just sued — fined. GDPR, CCPA, HIPAA... the list of regulations that treat customer data as toxic waste gets longer every year. Lucas: Right. So a growing number of teams are moving to synthetic test data — artificially generated data that looks and behaves like production data but contains zero real Personally Identifiable Information, or PII. And the results can be striking. I want to talk about one case in particular: a mid-sized fintech company I'll call PayFlow — not their real name — that processed about two billion dollars in transaction volume annually. Luna: And they had the classic problem: every sprint, QA needed fresh data that mirrored real customer behavior. So someone would just run a SQL script that copied a slice of the production database into the test environment. Lucas: Exactly. And that meant every single tester, every CI pipeline, every staging server was holding a copy of real customer data. Emails, account numbers, transaction histories. One security audit later, they realized they were in violation of their own internal data governance policy. So they decided to go synthetic. Luna: What did they use? A generative model, or something more rule-based? Lucas: They started with a rule-based approach — actually, a tool called Tonic, which lets you define statistical profiles for each column. You say: 'the age column should follow a normal distribution with mean 38 and standard deviation 12.' 'The email column should look like firstname.lastname at a domain from this list.' And it generates rows that match those distributions. But they quickly hit a wall. Luna: Let me guess — edge cases. Real production data has weird outliers. Someone born in 1901. A transaction of seven cents. A customer with a 200-character name. Lucas: Exactly that. Their synthetic data was too clean. It passed the basic unit tests, but when they ran integration tests that involved fraud detection models, the models started flagging synthetic transactions as anomalies — because real-world fraud patterns involve subtle correlations between, say, transaction amount, time of day, and device fingerprint. Those correlations weren't in the synthetic data. Luna: So they had to model the joint distributions, not just the marginal ones. Lucas: Precisely. They switched to a generative adversarial network — a GAN — trained on a masked version of their production data. The GAN learns the correlations between columns, so if in production, high transaction amounts tend to occur in the evening from mobile devices, the synthetic data reflects that. After retraining, their fraud model's false-positive rate on synthetic data dropped from 23 percent to about 4 percent. Close enough to production to trust the tests. Luna: But GANs are expensive to train and maintain. And they can still leak information — there's research showing that GANs can memorize rare rows and reproduce them almost exactly. Lucas: That's a real concern. Differential privacy can help — adding controlled noise during training so that even if the GAN spits out a row that looks like a real person, you can't prove it came from the training set. PayFlow added a differential privacy layer with epsilon set to 8, which is a moderate privacy guarantee. Their security team signed off on it. Luna: But this isn't just a fintech story. I was reading about how Adobe uses synthetic data for testing their Photoshop and Lightroom features. They have to test against millions of possible images — different resolutions, color spaces, metadata formats. They can't use real user photos for legal reasons. So they generate synthetic images with known ground truth. Lucas: And that's actually a simpler case than PayFlow, because the data — pixels — has well-understood statistical properties. Financial data has long tails, fat tails, regime changes. A GAN trained on last year's transactions might not generalize to this year's if the economic environment shifted. Luna: So what's the practical advice for a team that's considering synthetic data today? Where do you start? Lucas: I'd say: start small. Pick one critical table — say, 'transactions' — and generate a synthetic version. Compare the distributions of key fields: amount, date, merchant category. Use a statistical test like Kolmogorov-Smirnov to measure how close the synthetic distribution is to the real one. Then run your existing test suite against the synthetic data and see how many tests break. That tells you where your data fidelity gap is. Luna: And don't forget about referential integrity. If you generate synthetic users and synthetic transactions, the foreign keys have to match. That's another place where naive generation fails. Lucas: Right. PayFlow's first attempt generated users and transactions independently — so the transactions referenced user IDs that didn't exist. They had to build a dependency graph. They ended up using a tool called DataCove that lets you define the schema relationships and generates data in topological order: first customers, then accounts, then transactions. Luna: And that brings up the maintenance cost. Once you have a synthetic data pipeline, you have to maintain it. Every schema change — a new column, a new table — means updating the generation logic. Lucas: Exactly. And teams often underestimate that. They think they'll generate a one-time dump and be done. But the best approach is to treat synthetic data as a living product, not a project. PayFlow dedicated one engineer half-time to maintaining the generation pipeline. They also added a schema drift detector: a cron job that runs daily, compares the current production schema to the synthetic schema, and alerts the team if they're out of sync. Luna: That's smart. So the lesson is: synthetic data can be a huge win for privacy, test reliability, and even speed — but it requires investment. It's not a free lunch. Lucas: And that investment is worth it if your alternative is either risking a data breach or testing on stale, incomplete data. For PayFlow, the ROI was clear: they reduced PII exposure by 94 percent, and cut test data provisioning time from two weeks to four hours. That's a sprint-level improvement. Luna: That is a concrete number I can take back to my team. Two weeks to four hours is the kind of metric that gets a skeptical manager's attention. Lucas: And I should mention: they measured that before and after very carefully. The two weeks included the time to request a production snapshot, get security approval, run the masking script, validate the masked data, and then load it into the test environment. The four hours included generating the synthetic data, validating the distributions, and loading it. The whole pipeline. Luna: So the speed gain came from removing the human-in-the-loop — the security approval step alone took three days. Lucas: Exactly. And that's where synthetic data really shines. It's not just about privacy — it's about velocity. Teams can spin up fresh test datasets on demand, with full control over the characteristics. Want to test a scenario where 10 percent of transactions are above $10,000? Just set the distribution parameter. You can't easily do that with production data. Luna: I want to bring up one more risk: synthetic data can introduce false confidence if it doesn't capture the long tail. PayFlow's GAN missed a rare edge case — a transaction where the currency code was 'XBT' for Bitcoin. That was a legitimate value in production, but it appeared in only 0.001 percent of rows. The GAN essentially forgot it. Lucas: And that's a classic problem with generative models: they tend to smooth over rare events. The solution is to explicitly inject known edge cases. PayFlow created a 'rare values list' — a set of low-frequency but critical values that must appear in every synthetic dataset. They hardcoded those into the generation process, separate from the GAN. Luna: So a hybrid approach: GAN for the common patterns, plus rule-based injection for the rare but important cases. Lucas: Exactly. And that's what I'd recommend to any team today. Don't rely purely on a black-box generator. Maintain a list of critical edge cases — regulatory values, boundary conditions, legacy data formats — and make sure they're always present. Treat it like a test oracle for your synthetic data. Luna: Speaking of things that are worth maintaining: the show itself. We get to have these conversations because of listeners who see value in what we do, and we deliberately keep the podcast free of ads. If you appreciate that approach and want to support it, the link is buy me a coffee dot com slash fexingo. Lucas: Yeah, I'd echo that. It's a simple way to help us keep the episodes ad-free and focused on the technical details that matter to us. No pressure, but if it's something you've been thinking about, that's where to go. Luna: Alright, back to synthetic data. So we've covered the fintech and Adobe cases. I'm curious — what about open-source tools? Are there viable options for teams that can't afford commercial tools like Tonic or DataCove? Lucas: Absolutely. There's a Python library called Faker that's been around for years — it generates fake names, addresses, phone numbers, and can be extended with custom providers. For more sophisticated needs, there's SDV — the Synthetic Data Vault — which uses statistical models and deep learning to generate tabular data. It's not as polished as the commercial tools, but it's free and actively maintained. Luna: And for referential integrity, you'd need to script that yourself with SDV, right? Lucas: Yes, that's the trade-off. SDV handles single-table generation well, but multi-table with foreign keys requires manual orchestration. There's a newer tool called Gretel that has a free tier and handles relational data better, but it's not fully open-source. Luna: So the landscape is maturing. A few years ago, your options were either writing a bunch of custom SQL or buying an expensive enterprise tool. Now there's a spectrum. Lucas: And I think that's a good place to leave it. Synthetic test data is not a silver bullet, but it's a powerful tool in the QA arsenal. If you are still copying production databases into test environments, it might be time to explore the synthetic path. Luna: One final thought: as ai generated content becomes more common, the line between real and synthetic data will blur. But for testing purposes, we need to be deliberate about it. Know what you're generating, validate it, and treat it as a product. Lucas: Well said. That's all for this episode. We'll be back next time with another angle on building reliable software.