Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / How Image Generators Are Trained on Non-Consensual Photos
Transcript
- Lucas: If you've ever typed a prompt into an image generator like dall e or Midjourney and got back something eerily perfect, there's a good chance it was trained on a dataset called LAION-5B — roughly five billion image-text pairs scraped from the open web. Luna: Five billion. And we're learning that a non-trivial fraction of those images were uploaded without the subject's consent. Lucas: Exactly. A Stanford Internet Observatory report from late 2025 found more than 3,000 instances of non-consensual intimate images — think revenge porn, upskirt shots, even child sexual abuse material — inside the LAION-5B dataset. And that dataset has been used to train dozens of commercial and open-source models. Luna: How do images like that even end up in a training set? I mean, someone had to deliberately include them. Lucas: Right, so LAION-5B is a public index of image URLs and captions — it doesn't host the images themselves, just links to them. The problem is that the web is full of content uploaded without consent, and the scraping process didn't filter for that. It just grabbed everything from certain platforms, including forums and image hosts that are known for hosting abusive material. Luna: So the model never 'sees' the original image, but it learns patterns from the caption — like a description that includes a person's name and a degrading context. That's still training on harm. Lucas: That's the core issue. The model doesn't need the pixel data to encode associations. If a caption says 'Jane Doe in a compromising position,' the model learns to associate Jane Doe with that scenario. And then, if you prompt the model with Jane Doe's name, it might generate something that looks like her in a similar pose. Luna: And that's not just a privacy violation — it's potentially defamation, harassment, even a crime in some jurisdictions. Lucas: The Stanford report triggered a wave of calls for better dataset curation. But here's the thing: LAION-5B is open source. Anyone can download the index and use it. So the cat is already out of the bag. The question is what to do about models that have already been trained on that data. Luna: And whether future datasets can be cleaned more effectively. There's a project called 'Dolma' — an open dataset from the Allen Institute for AI — that claims to have built in filters for exactly this kind of content. Lucas: Yeah, Dolma uses a combination of blocklists, hash matching against known CSAM databases, and even a classifier trained to flag potentially non-consensual intimate content. But it's a cat and mouse game — abusers change URLs, rehost images, use coded language in captions. Luna: So what's the legal landscape look like? Are regulators catching up? Lucas: Slowly. The UK's Online Safety Act, which came into full force in early 2026, makes platforms legally responsible for user-generated content — but it doesn't directly address training data. In the US, the DEFIEND Act, proposed last year, would make it illegal to train AI on non-consensual intimate images, but it hasn't passed yet. Luna: And even if it passes, enforcement is a nightmare. How do you prove a model was trained on a specific image? It's not like you can inspect the weights. Lucas: Right — you'd need a technique like 'data provenance tracing,' which is still early-stage. Some researchers are working on watermarking images in a way that survives training, so if a model generates something that includes that watermark, you can trace it back to the source. But it's not foolproof. Luna: It feels like the industry is still in the 'move fast and apologize later' phase. Companies train on everything they can scrape, then deal with the fallout when someone finds the bodies in the dataset. Lucas: That's a fair summary. And it's not just intimate images — it's copyrighted material, medical records, private conversations. The whole 'fair use' argument for training data is being tested in courts right now. But the non-consensual image case has a moral clarity that copyright doesn't. No one can argue that revenge porn should be fair use. Luna: Speaking of fair use, I wanted to ask about something that connects to this. You know, a lot of listeners might wonder how we can support shows like this that dive into these complex issues without being beholden to advertisers. Lucas: Yeah, that's actually a good point. This podcast is listener-supported — a small group of people chip in monthly through buy me a coffee dot com slash fexingo, and that literally covers the hosting, transcription, and editing. It's what keeps us ad-free and able to spend time on stories that need more than a soundbite. Luna: And we really appreciate that. It allows us to follow threads like this one for as long as they need. So, back to the datasets — are there any bright spots? Lucas: There are. One is the rise of 'consent-based datasets' — collections where every image is explicitly contributed by the person in it, or with clear legal permission. For example, the 'OpenImages' dataset from Google has strict sourcing guidelines, though it's much smaller. Luna: But can you build a general-purpose image generator on just consent-based data? Or is it always going to be a compromise between scale and ethics? Lucas: That's the billion-dollar question. Some argue that you can't have a truly useful model without web-scale data, and that we need to accept some level of noise. Others say that's a false choice — that with better filtering and synthetic data augmentation, you can get close. Luna: Synthetic data — like generating training images from 3D scenes? That removes any privacy concern because the images are artificial. Lucas: Exactly. NVIDIA's 'StyleGAN' and 'GET3D' models can generate photorealistic human faces that are completely fake. Some companies are now training their downstream models entirely on synthetic data for specific tasks. The trade-off is that the model may not generalize as well to real-world scenarios. Luna: So for now, we're stuck with a messy middle: imperfect datasets, imperfect filters, and a patchwork of regulations. What should a responsible company do today? Lucas: At a minimum, run your entire training set through a hash-based filter for known CSAM — that's already standard practice at major labs. Beyond that, use classifiers to flag likely non-consensual content, and if you find it, delete those links and retrain your model. Some companies, like Stability AI, have done that after the Stanford report. Luna: But retraining is expensive. And if you don't have the resources of a Stability AI, you might be tempted to skip it. Lucas: And that's where regulation could help — making it a legal requirement, not just a best practice. Until then, it's on the community to hold itself accountable. But given the incentives, I'm not optimistic. Luna: So the bottom line: the images in your training set matter. And right now, too many of them are there without consent. Lucas: Yeah. And the people in those images — real people — are paying the price. The next time you generate a perfect image with an AI, it's worth thinking about what went into making that possible. Not all of it is pretty. Luna: Thanks for digging into this, Lucas. Lucas: Thanks for having me, Luna. And thanks to everyone who supports the show. It makes conversations like this possible.