Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / How Synthetic Data Bakes In Hidden Bias
Transcript
- Lucas: So here's a problem you don't hear about as often as biased hiring algorithms or facial recognition failing on darker skin — but it might be just as pervasive. It's the bias baked in by synthetic data. Luna: Synthetic data — meaning data generated by AI, not collected from the real world? Lucas: Exactly. Companies use it because real data is expensive, scarce, or privacy-sensitive. You take a real dataset, train a generative model on it, and then produce millions of new examples that look statistically similar but are artificial. It's a huge industry — Gartner predicted that by 2030, synthetic data will be used in sixty percent of all AI projects. Luna: That's a big number. But if the synthetic data is generated from a biased real dataset, doesn't it just inherit the same biases? Lucas: That's the surface problem. But it gets worse. Let me give you a concrete case. A few years ago, researchers at a major medical center built a model to detect skin lesions from photos. The real training data had about eighty percent lighter skin tones and twenty percent darker skin tones — not great, but typical. To balance it, they generated synthetic images of darker skin lesions using a GAN. That should help, right? Luna: It sounds reasonable. More data on the underrepresented group. Lucas: But the GAN had learned its understanding of 'dark skin' from that twenty percent slice. The synthetic images it created were less diverse in texture, lighting, and lesion morphology than real darker skin examples. The final model performed worse on actual darker-skinned patients than if they'd just used the original unbalanced dataset. Luna: So the synthetic data amplified the bias rather than correcting it? Lucas: Exactly. The synthetic examples were statistically similar to the real ones — but they collapsed the variation. And that's the hidden danger: you think you've solved the representation problem, but you've actually made the model brittle. Luna: And because the data is artificial, it's harder to audit. You can't just look at a thousand synthetic images and know what nuances they're missing. Lucas: Right. The issue is compounded by a feedback loop researchers call 'model collapse.' When an AI is trained on data that was itself produced by an AI, over multiple generations, the distribution narrows. Errors compound. Rare features get lost. A study from this March in Nature showed that after five cycles of training on synthetic data, a language model's diversity of output dropped by over forty percent. Luna: So it's not just medical imaging. It's any domain where synthetic data is reused across iterations — chatbots, recommendation systems, even self-driving car simulations. Lucas: Exactly. And here is where the regulation side gets interesting. The EU AI Act, which came into force last year, has a provision that requires high-risk AI systems to disclose the provenance of their training data. That includes whether any portion is synthetic. But the act doesn't specify how to validate the quality of that synthetic data. Luna: So companies can say 'we used synthetic data' and check a box, but there's no standard for checking whether the synthetic data is biased. Lucas: That's the gap. And it's a hard technical problem. There are emerging tools — like distribution comparison metrics, or using an independent real-world validation set — but they add cost and complexity. Most startups just want to ship. Luna: It reminds me of the early days of data augmentation in computer vision — rotating images, adding noise — and no one thought about how that augmentation might introduce artifacts. Synthetic data is the same thing at a much larger scale. Lucas: And with higher stakes. A mislabeled cat photo is one thing. A synthetic medical image that causes a misdiagnosis is another. I think we need to start thinking about synthetic data the way we think about clinical trial data — you need to know not just where it came from, but how it was generated and what assumptions that generation process baked in. Luna: Speaking of what keeps this show going — we're able to explore these questions because we don't have advertisers pulling us in a different direction. If today's episode helped you think about this issue more clearly, buy me a coffee dot com slash fexingo is how you can keep that going. Lucas: Yeah, that genuinely makes a difference. It's a small gesture that lets us stay independent and dig into stories like this one. Luna: So let's go deeper on that regulatory gap. The EU AI Act's requirement for data provenance — what would you like to see added? Lucas: I think the next logical step is a requirement for 'bias impact assessments' specifically on synthetic data components. So if your training set is thirty percent synthetic, you need to demonstrate that the synthetic portion doesn't introduce statistically significant differences in performance across demographic groups. Luna: That would force companies to do the kind of audits we're talking about. But it also requires regulators to have expertise in generative models, which most don't yet. Lucas: True. The European Commission is building an AI Office, and they're hiring technical staff. But it's going to take time. Meanwhile, there are third-party auditing firms emerging — companies like Credo AI and FairNow are starting to offer synthetic data validation services. Luna: So market pressure might move faster than regulation. If big buyers like hospitals or insurers start demanding certified bias-free synthetic data, that could shift the industry. Lucas: That's the optimistic scenario. The pessimistic one is that synthetic data becomes so cheap and ubiquitous that the quality checks are seen as a competitive disadvantage. 'Our model trained on a trillion synthetic tokens — we don't have time to check for bias, we need to get to market.' Luna: That sounds like the early 2010s with social media platforms — move fast and break things, except the things here are people's health, jobs, and freedoms. Lucas: Exactly. And the thing about synthetic data is that it's self-reinforcing. The more you use it, the more you depend on it. If you have a model that's been trained on three generations of synthetic data, you can't just go back to real data — you'd lose the scale. Luna: So you're locked in. That's a structural risk that I think investors and regulators are only beginning to grasp. Lucas: There is one promising technical approach that might help: what some researchers call 'data genealogy.' You keep a record of every transformation applied to every training example — including whether it was generated synthetically, and from which parent data. If a bias is later found, you can trace it back to its source. Luna: That's like a blockchain for data provenance. But it adds storage and computation costs. Lucas: It does. But in high-stakes domains — medical devices, credit scoring, hiring — the cost of not having it is higher. I think we'll see regulatory mandates for data genealogy within three to five years. Luna: And until then, what should a practitioner listening to this do? If you're building an AI product and considering synthetic data? Lucas: Three things. First, always keep a held-out real-world test set that represents your target population — not just the demographic mix, but the full variation. Second, run disaggregated evaluations: check model performance separately for each subgroup you care about. Third, if you use synthetic data, document exactly how it was generated and validate that the synthetic distribution matches the real distribution on clinically or operationally relevant features — not just pixel-level similarity. Luna: That last point is crucial. A lot of synthetic data quality metrics look at global statistical distance — but the model might fail on a rare but important subgroup that's underrepresented even in the synthetic data. Lucas: Right. And that's the lesson from the medical imaging case. The darker skin lesions were rare in the real data, and they remained rare in the synthetic data because the GAN learned the distribution, including the skew. Augmenting a rare group with naive generation doesn't fix the underlying data collection problem — it just masks it. Luna: So synthetic data is a tool, not a cure-all. It can help, but it can also hide problems if you use it without rigor. Lucas: Exactly. And I think the broader takeaway is that AI ethics isn't just about algorithms — it's about data supply chains. The decisions you make about where your data comes from, how you augment it, and how you validate it have ethical consequences that ripple through the entire system. Luna: That's a good note to end on. Next time, we're going to look at what happens when AI systems are deployed in criminal justice — and how predictive policing tools are being challenged in court right now. Lucas: Looking forward to it. For now, thanks for listening.