Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Synthetic Test Data Avoids Privacy Violations
Transcript
- Lucas: Let me start with a number that stopped me: according to a 2025 survey by the QA consultancy PractiTest, nearly 40 percent of software teams have had a test data related privacy incident in the past two years. That's not a theoretical risk anymore. Luna: Forty percent feels high. I mean, we all know production data in test environments is a bad idea, but that suggests a lot of teams are still doing it. Lucas: Or they're using de-identified data that gets re-identified, or they're masking fields but missing something. The common thread is that real data carries real liability. And that's where synthetic test data comes in — data generated artificially that mimics the statistical properties of production without containing any actual person's information. Luna: So instead of copying a customer database and scrambling the names, you generate entirely fake records that look real enough for testing. That's been around for a while — tools like Mockaroo and Faker have been doing basic generation for years. Lucas: Right, but the shift in the last eighteen months is the use of generative AI to create much more realistic synthetic data. Older tools relied on rules and random values — they could fake a name and an email, but they didn't capture the complex correlations you get in real datasets. For example, a patient's zip code, age, and diagnosis code have statistical relationships. A rule-based generator might produce an eighty-year-old with a pediatric condition. That breaks your test. Luna: And that's the thing — if your test data isn't realistic, your tests can pass in staging but fail in production because the real data has patterns you never exercised. So generative models are learning those patterns and producing synthetic data that preserves them. Lucas: Exactly. And before we go deeper, I want to mention something that actually connects here. Tests only protect you if the data feeding them is trustworthy. And we've been able to keep this show independent and ad-free because listeners who find the content useful choose to support it. If you've gotten value from these episodes, you can help at buy me a coffee dot com slash fexingo. Even a small contribution helps us keep the conversation going. Now back to synthetic data — a team I spoke with at a mid-sized healthcare software company recently made the switch. Luna: Good segue. What was their setup before synthetic data? Lucas: They were using de-identified production data. They'd strip patient names, social security numbers, and addresses, then load it into their staging environment. Problem is, de-identification isn't a one-and-done. Researchers at Imperial College London showed in 2024 that 99 percent of re-identification attempts succeeded on datasets that were considered "de-identified" using standard methods. So this team was sitting on a privacy time bomb. Luna: And they had a real incident? Or was it proactive? Lucas: Proactive, fortunately. The legal team flagged that their data pipeline had a gap — a single misconfigured field could expose real patient data to a test environment accessed by offshore contractors. So they decided to switch to fully synthetic data. They evaluated two approaches: a rule-based generator and a generative AI model trained on their production schema. Luna: Which one won? Lucas: The AI model, but with a significant caveat. The rule-based tool was faster to set up — they had it running in two days. But when they ran their regression suite against the rule-based data, four test cases failed that had never failed before. Turned out the rule-based generator couldn't reproduce the correlation between certain medication codes and lab result ranges. It was generating impossible combinations. The AI model needed about two weeks to train, and they had to be careful about data governance — they trained it on the de-identified production data, but the model never stored examples. It only learned distributions. Luna: So the model itself isn't a privacy risk because it doesn't memorize specific records. That's crucial for compliance under HIPAA or GDPR. Lucas: Exactly. Once they had the ai generated dataset, they validated it against three metrics: column-level statistics like means and standard deviations, cross-column correlation matrices, and edge-case coverage — did the synthetic data include the unusual but critical scenarios like a patient with zero claims or a diagnosis code that's rarely used? The synthetic data passed all three. And when they ran the regression suite, all tests passed. Luna: But did it catch any bugs that the old data would have missed? That's the real test. Lucas: It did, actually. The team found a race condition in their claims processing pipeline that only surfaced when two synthetic patients had identical timestamps for overlapping procedures. The old de-identified data had been scrubbed of timestamp precision — every timestamp was rounded to the nearest hour. That hid the race condition for months. The synthetic data preserved sub-second precision because the generative model learned the real distribution. Luna: That's a great example. The synthetic data wasn't just safer — it was more accurate for testing purposes. But I'm guessing there's a catch. What about data drift? If the production data changes over time, the synthetic data becomes stale. Lucas: That's the number-one operational risk. Teams that adopt synthetic data sometimes forget to retrain the generative model. I've seen cases where the synthetic dataset is six months old, but the production distribution has shifted — maybe the company started serving a new demographic, or a new product line changed data patterns. The synthetic data becomes increasingly unrealistic, and tests start passing when they shouldn't. Luna: So you need a validation step in the CI pipeline that compares synthetic data statistics against production periodically. If the drift exceeds a threshold, trigger a retraining job. Lucas: That's the best practice. Some teams run that comparison weekly. Others do it on every deployment to staging. The metric they use is something called the Population Stability Index — it's a statistical measure that quantifies how much the distribution of a variable has changed. If the PSI exceeds 0.2 for any critical field, the pipeline blocks the deploy until the synthetic data is refreshed. Luna: That's smart. It turns a manual process into an automated guardrail. I'm curious about the cost side. Training a generative model and running inference for millions of rows isn't free. Is this only feasible for well-funded teams? Lucas: The cost has come down significantly. A year ago, training a bespoke generative model might have cost several thousand dollars in compute. Now, with models like Gretel's or Mostly AI's offerings, you can generate a million synthetic records for under a hundred dollars. For most mid-market companies, that's easily justified against the legal risk of a privacy breach. A single HIPAA violation can cost fifty thousand dollars per record. Luna: So the ROI is clear for regulated industries. But what about a SaaS startup that doesn't handle health data or financial data? Is synthetic data worth the overhead? Lucas: I'd say yes, for one reason: test data consistency. Startups often have the messiest test data because they're moving fast. They copy production databases haphazardly, they have stale snapshots, they hardcode data in test fixtures. Synthetic data forces you to define what your data should look like. That process alone — documenting schemas, distributions, and edge cases — improves the team's understanding of their own system. Luna: It's a side effect that's almost as valuable as the data itself. So if a team wants to get started, what's the minimum viable approach? Lucas: Start with one critical service. Pick the database that has the most sensitive data or the most frequent test failures. Generate synthetic data for just that service, validate it against production statistics, and run your existing test suite. If the tests still pass and you see an improvement in bug detection, expand from there. Don't try to replace all production data at once — you'll hit integration issues with services that still consume real data. Luna: And what about tools? Is there an open-source option or are you locked into a vendor? Lucas: There are open-source options like SDV — the Synthetic Data Vault — which has been around for years. It supports tabular data, time series, and relational databases. The learning curve is steeper, but it's free. On the vendor side, you have Gretel, Mostly AI, and Tonic.ai, which offer managed services with better integrations. I'd recommend prototyping with SDV to understand the workflow, then decide if you need the managed convenience. Luna: That's a good path. One more thing I want to touch on — the risks of synthetic data itself. There's been research showing that synthetic data can amplify biases present in the training data. If the real data underrepresents certain groups, the synthetic data will too, and your tests might miss issues affecting those users. Lucas: Absolutely. That's a real concern. If your production data has, say, only five percent of users from a particular region, the synthetic data will mirror that. But because it's generated, you can deliberately oversample minority groups to create more representative test data. That's actually easier to do with synthetic data than with real data, because you're not constrained by actual counts. You can tell the generator to produce twenty percent of records from that underrepresented region. Luna: So you can consciously design for fairness in testing. That's a powerful capability. I wonder how many teams are aware of that. Lucas: Not many, based on conversations I've had. Most teams are still focused on the privacy compliance angle. The fairness testing use case is more advanced. But I think it will become standard within two years, especially as regulators start examining algorithmic bias more closely. Luna: Let's bring it back to that healthcare team. Did they measure the actual compliance improvement after switching to synthetic data? Lucas: They did. Their legal team estimated that each synthetic dataset reduced the review time from two weeks to two days — an eighty percent reduction. More importantly, they eliminated the need for a data-sharing agreement between the production and test environments. The test environment no longer touched any real patient data. That got them through a HIPAA audit with zero findings. Luna: That's a concrete win. And the race condition they found is a bonus that justifies the switch on testing quality alone. Lucas: Right. And I think that's the story to hold onto. Synthetic data isn't just a compliance checkbox. When done well, it produces better test coverage, catches bugs that real data masks, and gives teams control over the statistical properties of their test data. The healthcare team's experience is repeatable for any team that deals with sensitive data. Luna: Agreed. And with the cost coming down and open-source options available, there's really no excuse to keep using production data in test environments. Lucas: That's the takeaway. Thanks for listening. If you want to continue the conversation, you know where to find us.