Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Synthetic Test Data Outshines Production Data Scraping
Transcript
- Lucas: So Luna, we talk a lot about test automation and coverage, but there's a foundational problem that keeps coming up in conversations with QA teams: where does your test data actually come from? Luna: Yeah, and I bet most teams say they just grab a copy of production and call it a day. Lucas: Exactly. And that's the default for a reason — it's the easiest path. You run a database export, anonymize a few columns, and you've got realistic data. But that approach is getting riskier every year. Luna: Because of compliance, right? GDPR, CCPA, even internal security policies. Lucas: Right. Even if you mask personally identifiable information, there's always the chance of a leak. And then there's the data quality problem: production data is full of historical quirks that might not reflect the edge cases your new feature actually needs. Luna: So what's the alternative? You just make up data from scratch? Lucas: That's exactly what a growing number of teams are doing — synthetic test data. Instead of copying production, you generate artificial data that mimics the statistical properties and relationships of real data, but contains no actual customer information. Luna: I've seen tools like Tonic.ai and even open-source libraries like Faker. But does the generated data actually catch the same bugs? Lucas: That's the million-dollar question. I talked to a fintech startup called LendLayer — they process loan applications — and they switched to synthetic data two years ago. Their QA lead told me synthetic data actually caught a bug that production data never would have triggered. Luna: How? I'd think production data would have more variety. Lucas: Counterintuitive, right? But production data has a limited set of patterns — it reflects what users have actually done. If you're testing a new interest calculation for a loan product that doesn't exist yet, production data won't have that combination. Synthetic data lets you specify exactly the scenarios you want. Luna: So you can create edge cases that are statistically rare but logically important. Lucas: Exactly. LendLayer's team generated 10,000 synthetic loan applications with specific parameters — high loan amounts, low credit scores, unusual repayment periods — and found a rounding error in their new calculation engine that would have only affected about 0.1 percent of real applications, but each error was about $50. Luna: Over time that adds up. And production data wouldn't have had those combos because nobody had taken that loan yet. Lucas: Right. And it gets better from a compliance standpoint. Since synthetic data contains no real customer information, you can put it in any environment — developer laptops, CI pipelines, even offshore QA teams — without worrying about data privacy laws. Luna: That alone saves so much overhead. I've seen teams spend weeks just getting approval to use production data in staging. Lucas: LendLayer cut their test data preparation time by 70 percent. They used to have a dedicated engineer running monthly scrubbing scripts. Now they generate fresh data on demand through a pipeline that rebuilds the synthetic dataset every night. Luna: But isn't there a fidelity problem? If the data doesn't perfectly match production distributions, aren't you risking false positives or missed bugs? Lucas: That's the trade-off. You need to invest in modeling your production data distributions accurately. Tools like Tonic allow you to specify constraints and relationships — for example, 'if a customer is under 18, they can't have a credit card' — so the synthetic data respects business rules. Luna: So it's not just random generation. You're essentially building a statistical model of your production data. Lucas: Yes. And that model gets better over time. The first pass might miss some subtle correlations, but you iterate. And the beauty is, you can also inject anomalies that don't exist in production at all — to test your system's robustness. Luna: That's actually something chaos engineering advocates for, right? Injecting failure scenarios. Lucas: Exactly. Synthetic data allows you to test the boundaries of your system proactively. You can generate data that's one millisecond from midnight, or with null values in critical fields, or with Unicode characters in names. Luna: Those are the bugs that slip through because QA tested with production data that was all clean. Lucas: And speaking of clean data — this is exactly the kind of practical conversation that we love bringing to you without any ads. We deliberately keep Fexingo ad-free because we think the content should speak for itself. Luna: Yeah, it's a conscious choice. If you find value in these episodes and want to support that model, the link is buy me a coffee dot com slash fexingo. Lucas: Even a small contribution helps us keep the focus on the substance, not the sponsors. Okay, back to synthetic data — I want to talk about a specific pitfall I've seen teams hit. Luna: What's that? Lucas: Over-relying on synthetic data for performance testing. If your synthetic data generator creates data that's too uniform — say, all zip codes are five digits — your database indexes might behave differently than they would on production. Luna: So you get performance test results that don't translate to production. Lucas: Exactly. The solution is to model the cardinality and skew of your real data. If 80 percent of your customers are in three cities, your synthetic dataset should reflect that. Otherwise you're testing against an unrealistic distribution. Luna: So there's still a need for production data profiling. You just don't copy the actual rows. Lucas: Right. You profile the schema, the distributions, the referential integrity — but you generate new values. That's the key distinction. You're not masking production data; you're creating data that looks like production but isn't. Luna: And I imagine that gets easier with tools that integrate directly into your CI/CD pipeline. Lucas: Absolutely. Tonic, for example, has a CLI and API so you can regenerate test data on every build. Some teams even version their synthetic data configurations alongside their code. Luna: That way, if a test fails because of a data change, you can roll back the data configuration just like you'd roll back code. Lucas: Exactly. And that's a huge improvement over production snapshots, which are opaque — you don't know exactly what changed between two snapshots. Luna: So synthetic data gives you reproducibility and traceability. Lucas: Yeah. And it also solves the 'stale data' problem. Production snapshots age — customer records get updated, orders get shipped. After a few weeks, you're testing against a reality that no longer exists. Synthetic data can be generated fresh every time, reflecting your current schema and business rules. Luna: That's a strong argument. But I've heard concerns about the upfront cost — modeling your data takes time and expertise. Lucas: It does. But LendLayer told me the investment paid off in three months. Before, they had a full-time engineer managing data scrubbing. Now that engineer works on test automation instead. The time to generate dropped from two days to fifteen minutes. Luna: And the compliance risk reduction is hard to quantify but real. Lucas: Especially for companies in regulated industries. One healthcare startup I spoke with switched to synthetic data specifically because they couldn't legally put any patient data in their CI pipeline. They were running tests on a tiny hand-curated dataset, missing tons of bugs. Luna: So synthetic data unblocked their automation. Lucas: Completely. Their test coverage went from 20 percent to 85 percent within two months. And they could finally run tests in parallel without tripping over data dependencies. Luna: What about data that has complex referential integrity? Like a customer with multiple accounts, each with transactions, all linked by foreign keys. Lucas: Modern synthetic data tools handle that. You define the schema once, and the generator ensures referential integrity — you won't get a transaction referencing a nonexistent account. Some tools even allow 'conditional generation' where the value in one column depends on another. Luna: So you can say 'if account type is premium, then balance must be above $10,000'. Lucas: Exactly. That's where the modeling effort pays off — you encode your business logic into the data generator, and suddenly your tests are exercising exactly the paths you want. Luna: I wonder if synthetic data could also help with testing data migrations. Like if you're moving from one database to another, you could generate data that exercises every column type and constraint. Lucas: That's a great application. In fact, I know a team that used synthetic data to test their migration from MongoDB to PostgreSQL. They generated millions of documents with nested arrays, null values, and special characters — things that might break the migration script. Luna: And they caught issues before touching real production data. Lucas: Exactly. The key takeaway is that synthetic test data isn't just a compliance workaround. It's a testing superpower — it gives you control, reproducibility, and the ability to test scenarios you can't find in production. Luna: But you still need to validate that your synthetic data is representative. Otherwise you're just testing fiction. Lucas: Right. The mantra is 'garbage in, garbage out.' You need to invest in profiling and validation. Run statistical checks — compare distributions, check for missing combinations. Luna: So it's not a set-it-and-forget-it solution. Lucas: No, but neither is production data scraping. That requires constant maintenance too — schema changes, new regulations, data growth. Synthetic data just shifts the complexity from compliance to modeling. And for many teams, that's a trade worth making. Luna: Alright, I'm convinced. Next time someone asks where our test data comes from, I'll make sure we have a better answer than 'production'. Lucas: That's the goal. Thanks for the conversation, Luna.