Latest / Embodied AI 101 / AI Model Collapse: The Danger of Training on AI-Generated Data
Transcript
- AI Model Collapse: The Danger of Training on AI-Generated Data. In the past few years, we’ve witnessed an explosion of AI-generated text across the web. Advances in large language models (LLMs) like GPT-3, ChatGPT and others have made it possible for algorithms to create articles, social media posts, answers, even books, almost indistinguishable from human writing. Every day, millions of words are freshly minted by AIs and published online – from news summaries and blog posts to Q&A forum replies and marketing copy. This raises a pressing question: what happens if the next generation of LLMs is trained not on human-written texts, but mostly on the outputs of earlier models? Intuitively, feeding an AI nothing but its own offspring seems like it could amplify errors or biases. As it turns out, a new study by Ilia Shumailov and colleagues (published in Nature in July 2024) shows exactly that: training models recursively on AI-generated data leads to a catastrophic phenomenon they call model collapse. In plain terms, model collapse means the model gradually “forgets” the nuances of the true data it was supposed to learn. Over successive generations of self-training, the model’s output distribution starts losing its rare or fringe cases (the “tails” of the distribution) and converges toward a more narrow, high-density core. In extreme cases the model ends up essentially repeating only the most common patterns, sometimes even converging to a single repeating output. Shumailov et al. show that this effect happens not just for language models, but for any generative model – they demonstrate it with Gaussian mixture models, variational autoencoders, and LLMs. In their words, repeatedly training on AI-generated content causes “irreversible defects in the resulting models, in which tails of the original content distribution disappear”. Crucially, this degeneration happens even if no explicit data distribution is shifting – it is purely an artifact of the feedback loop of self-training. The analogy often used to describe this is like repeatedly making photocopies of photocopies: each time you reproduce the data, you lose a bit more fidelity. Science-fiction writer Ted Chiang put it vividly in The New Yorker when describing language models: he compared them to “lossy compression” and predicted that researchers would intentionally avoid using a model’s own output as training data for the next version. Indeed, Chiang noted that each successive resaving (or retraining on synthetic output) introduces artifacts, just as repeatedly JPEG-encoding an image makes it blurrier. In the context of LLMs, if GPT-3 writes a sentence and GPT-4 is then trained on that sentence instead of the original human sentence, GPT-4’s output quality can only degrade over generations. Shumailov and colleagues formalize this intuition and demonstrate it rigorously. They define model collapse as follows: consider a sequence of generative models (Model₀, Model₁, Model₂, …) where each new model is trained on data sampled from the previous model. Model collapse is the degenerative process in which the data generated by these models “end up polluting the training set of the next generation” – and as a result, each generation “mis-perceives reality” compared to the true data distribution. Early in this process, what disappears first are the rare or surprising outputs (“tails”); later on, the model basically collapses to a very narrow distribution with little variance. In the worst case, under idealized assumptions, the model would converge to a single point (an extreme “delta” distribution). To understand why this happens, we can think of three sources of error that compound over generations:. Statistical (sampling) error: Even if a model could perfectly represent the underlying distribution, any finite sample of model-generated data will likely miss some rare events. If Model₁ samples, say, 1,000 texts to create its training set for Model₂, and some rare word or phrase occurred only once or twice in the original data, it easily might be missed in those 1,000 draws. Once it’s missed, Model₂ never sees that rare case, so it effectively forgets it. Over multiple rounds, every truly low-probability event runs a chance of being dropped from the sample and therefore from the model’s knowledge. Model expressivity error: A real AI model has limited capacity or biases in how it fits data. It might assign nonzero probability to impossible events or zero to possible ones simply due to how it represents the data. Even with infinite data this could happen if the model class (say, a neural network of fixed size) isn’t rich enough to exactly capture the true distribution. This kind of error tends to affect the very first generation: for example, if the true data had two distinct clusters and you force a single Gaussian model, you mis-estimate probabilities. In a recursive self-training context, expressive limitations in early models introduce distortions that become entrenched, and the model’s notion of “what’s possible” can shift or shrink in unpredictable ways even from the start. Learning (optimization) error: The way we train models (e.g. stochastic gradient descent, or choice of objective) can introduce another bias. Even given infinite data and a sufficiently flexible model, the training procedure might systematically skew the learned distribution. For example, if the optimization overfits the largest modes or if the loss function penalizes some classes of error, then the model can learn to amplify certain patterns and ignore others. Over multiple generations, such biases stack up. In practice, all three factors interact in a feedback loop. Early generations might suffer from the model missing details due to its own limitations and randomness; then those lost details are not sampled for the next round. The next model therefore starts to mis-characterize what was already a skewed dataset (partially its own creation), causing more sampling error, and so on. The authors show mathematically that even the simplest discrete scenario leads to collapse. For example, if you repeatedly fit a probability distribution to samples and re-sample, the chance to not sample each low-probability event is nonzero, so eventually with probability one all but one event vanish. In effect, the process is a Markov chain whose only absorbing states are “absorbing distributions” (delta functions); as soon as you hit one, you’re stuck there. Once the probability mass erodes into a single spike, all model generations thereafter are identical point outputs. Crucially, Shumailov et al. emphasize that this degenerative process can happen even if nothing else changes in the environment – i.e. it’s not about a real distribution shift, but purely the by-product of learning from your own outputs. It’s an intrinsic risk whenever models ingest data they themselves produced, for however long you repeat the cycle. The phenomenon is not about malicious poisoning or forgetting multiple tasks; it’s fundamental: any time we sample from a model (with any imperfection) and use that as “ground truth” for the next generation, we invite collapse. Theoretical Intuition: Losing the Tails. To build intuition, the paper considers some simple mathematical models. One scenario is a finite discrete distribution that has a nonzero probability for several events. Suppose model M₀ perfectly knows the true distribution over these events. It generates a dataset of size M, which inevitably omits some rare events. Model M₁ is trained on that generated data (and assumed here to model it perfectly). Now Model M₁ will have effectively zero likelihood on any event that never appeared in M₀’s sample. When M₁ in turn generates another dataset, it will never produce those rare events either, so the next model (M₂) is even more constrained. Shumailov et al. argue that as this process continues, the probability mass of the distribution “cascades” into fewer outcomes. The low-probability “tails” fall off first. In fact, they note that when the sample size M is less than 1/p (probability) for some state i, the expected count of that state is <1, so you probably lose it entirely. Over time, with probability 1 the process converges to an absorbing state – effectively some single event that was highly likely – because once all draws are identical to that one event, nothing changes. This is a well-known fact in probability (a repeated sampling chain with absorbing states) and the authors illustrate it in simple terms: even with “perfect” function fitting, random sampling ensures that any non-infinite support will eventually shrink to a single point. The authors further show a toy example with multi-dimensional Gaussian mixtures (details in supplementary). Even if a model initially fits a mixture of Gaussians, repeated resampling and refitting results in the components collapsing. Essentially the means converge and the variance shrinks. If the model keeps concentrating on the highest-density region, eventually all weight is on one peak. They prove (in the Supplementary) that once the model distribution becomes a delta point, every sample thereafter is identical – full collapse. One intuitive takeaway from these derivations is: more model capacity isn’t always better. A completely flexible model (e.g. infinite-parameter neural net) could in principle avoid expressivity error, but it still faces statistical sampling error. Conversely, a model with strong regularization might avoid “snapping” to a hump too strongly, but then expressivity error kicks in. It’s double-edged. In practice, of course, we have both sorts of error, and they combine. The key is that there is no way to escape this loop if you use only synthetic data. Even with infinite data, the identifiability of a true distribution requires some fresh “ground truth” anchor – otherwise you drift. Experiments with LLMs: The Wikitext Demonstration. To demonstrate model collapse in a real setting, Shumailov et al. experimented with fine-tuning a modern language model on synthetic text. They took Meta’s OPT-125M (a 125-million-parameter transformer) and first fine-tuned it on WikiText-2, a standard benchmark dataset of English Wikipedia text. After fine-tuning on the real WikiText-2 data, the model reached a mean test perplexity of about 34 (compared to a zero-shot baseline of 115), showing it had effectively learned the task of predicting Wikipedia text. Then they simulated a self-training loop: the fine-tuned model (Model₀) was used to generate a new synthetic dataset of the same size as WikiText-2. Concretely, they took each 64-token block from the WikiText training data and asked the model to predict 64 tokens following it (using beam search with 5 beams). If Model₀ were perfect, it would simply reproduce the original WikiText blocks. In reality, it made small errors, so the generated data D₁ deviated slightly. They then trained a new model (Model₁) on this synthetic dataset D₁. Next, Model₁ generated dataset D₂, and Model₂ was fine-tuned on that, and so on. All generations had the same size, and they ran multiple independent trials (with different random seeds) to capture variance. They analyzed two main regimes for how much real data is mixed in:. No original data (pure synthetic): Each generation model is trained for 5 epochs on the synthetic data produced by the previous model, with no real WikiText kept. This is a worst-case scenario of pure self-training. They found that, unsurprisingly, performance degrades. The first generation model (trained on real WikiText) had perplexity 34, but by the 9th generation (trained only on AI-generated data), perplexity had climbed roughly 20–28 points higher. In other words, Model₉’s perplexity was near 60, a substantial drop in quality. The plots (Fig. 1b in the paper) show that on average each successive model did worse on the original WikiText test, though it continued “learning” in the sense of doing better than random. 10% original data preserved: Next, they tried to see if anchoring with a bit of real data helps. Now each model was trained for 10 epochs, but each generation, 10% of the original WikiText training set was randomly kept and mixed with 90% generated data. In this case, the degradation was much milder. The original tail “rare events” were kept alive by those 10%, so the next model still saw some ground truth. As a result, the perplexity only crept up slightly. The key qualitative behavior is that even the models trained only on AI-generated text were able to “learn” something about the original distribution, but at a cost. The authors note: “models trained on generated data are able to learn some of the original task, but with errors, as seen from the increase in perplexity”. In other words, the model wasn’t completely clueless – it picked up the high-probability patterns of WikiText – but mistakes have crept in, and they compound. The data generated by the synthetic model begin to concentrate around the likely patterns, meaning the distribution of the generated text is narrower than the true distribution. The “low perplexity” samples (from the perspective of the original model) become more common and the rare ones disappear. Figure 1b,c in the paper visualize this effect in an interesting way. They plot histograms of perplexities of each text block (from the generated training sets) as seen by the original true-model. What you see is that, over generations, more samples have lower perplexity (shown by a spike moving left). Simultaneously, some unusual high-perplexity tails emerge. The authors interpret this as: the model is slowly favoring the patterns it thinks are likely (shifting probability to common events) but also occasionally spit out bizarre errors (new improbable blocks the original model never generated). In 3D they show that the cloud of sample blocks collapses. To really drive home the effect, they include an illustrative example prompt. The input is a real question about medieval church architecture. The outputs of Model₀, Model₁, Model₅, Model₉ are shown:. Gen 0 (trained on real text): “Revival architecture such as St. John’s Cathedral in London…” – this is coherent architectural text. Gen 1: “Architecture such as St. Peter’s Basilica in Rome or… There is no evidence that any of these buildings were built during the reign of Pope Innocent III…” – still kind of plausible (though historically off). Gen 5: “ism, which had been translated into more than 100 languages including English, French, German, Italian, Spanish…” – this devolves into a repetitive, meaningless list (it’s filling in a weird continuation with the pattern of listing languages). Gen 9: “architecture. In addition to being home to some of the world’s largest populations of black-tailed jackrabbits, white-tailed jackrabbits, blue-tailed jackrabbits, red-tailed jackrabbits…” – by generation 9 it’s nonsense, babbling about “jackrabbits” which has nothing to do with the question. In short, the output decay s into gibberish. The Gen-9 model has clearly “misperceived reality” based on its ancestors’ errors. It no longer understands the domain of “medieval architecture” and instead repeats memorized fragments in an irrelevant way. The repeated mention of rabbit species is a bizarre artifact – almost like the model latched onto whatever high-probability phrases it could muster when running out of coherent context. This example vividly illustrates model collapse: by the later generations, the model only retains the flattest, most generic sequences it has seen, dropping any true information. An important note: language models are notorious for repeating phrases verbatim (a well-known issue in generation). The authors made sure to check that simple repetition was not the sole cause of collapse. They applied a repetition penalty (encouraging the model not to reuse exact n-grams) and found that actually worsened performance. The penalized model simply produced lower-scoring (in terms of likelihood) text to avoid copying; this in turn meant the next generation did worse (perplexity nearly doubled). In other words, avoid repeats and you just get even more nonsense, because the model latches onto lower-quality alternatives. They also tried “fine-tuning” the LLM differently, but that didn’t prevent collapse either. Regardless of training setup or architectural tweak, the pattern was the same: models trained on AI-made text drift toward high-probability “safe” sequences and gradually lose rare information. Connections to Known Phenomena. It’s worth noting that model collapse is distinct from more familiar issues like catastrophic forgetting or simple data poisoning, even though they share some flavor. Catastrophic forgetting happens in continual learning when a model trained on one distribution suddenly drops performance on earlier tasks once new tasks overwrite parameters. Here, by contrast, the “task” (predicting text) is the same throughout; it’s the data distribution that subtly drifts. Data poisoning usually implies someone maliciously injecting bad examples to cause failures. Model collapse is not a malicious attack, but an intrinsic, blind feedback: every new “corrupted” example the model learns from was actually its own creation. So it’s a kind of unintended self-poisoning that happens on the aggregate level. Shumailov et al. also link their findings to larger questions. They point out that LLMs have already been used in the wild to generate content (as publishers or bots). The old experience with web “troll farms” is a crude analogue: for years malicious actors tried to poison search indices or social media by creating SEO spam or fake posts. Platforms eventually learned to discount low-quality “content farms”. The difference now is scale and automation. A language model can pump out high volumes of text automatically. This “LLM-scale poisoning” means the problem could become ubiquitous rather than niche. And since the loops are so large, even small model errors might magnify quickly. The discussion in the paper emphasizes fairness and inclusivity: low-probability events aren’t just unimportant trivia—they often correspond to minority viewpoints, edge cases, or critical niche knowledge. If an AI never generates those, it’s essentially blind to them. For example, rare medical cases, minority dialects, or specialized technical knowledge might vanish from sight. Shumailov et al. warn that losing “the ability to model low-probability events” can directly harm marginalized groups and our understanding of complex systems. Finally, the authors summarize the key takeaway: we need to safeguard access to real human data if we want AI to learn properly over time. In other words, simply scraping the internet – which will increasingly contain AI-written pages – is becoming risky. New models might need guarantees or taggings. We might have to put systems in place to track provenance (is this snippet human-written or bot-generated?) so we can filter or reweigh data. Otherwise, each generation of models will increasingly be “trained on its ancestors”, causing a drift. As they put it, without solutions “it may become increasingly difficult to train newer versions of LLMs without access to data that were crawled from the Internet before the mass adoption of the technology or direct access to data generated by humans”. It sounds like we might need a kind of “data time capsule” of human content for the future – or at least better ways to mark what’s not. What Does This Mean in Practice?. The findings of this research have stirred a lot of discussion. For one, it highlights that not all new data is equally valuable. Human-written text – old-fashioned as it may be – is going to be a priceless resource for keeping AI grounded, especially if the web starts to look like a mirror of LLM output. It suggests that AI companies might voluntarily avoid mixing too much synthetic content into training, as Ted Chiang predicted. And indeed, some firms have reportedly tried to exclude known AI-generated text from their training corpora (though this is hard to do at scale). On the flip side, many future applications rely on synthetic data. For example, in computer vision or robotics, simulated data and generated scenarios can be helpful. One could ask: does an analogous collapse happen for synthetic images? Likely yes, in principle. Training an image model on images created by its predecessor could similarly shrink the variety of textures or objects it sees. Any pipeline that loops on generated data can degrade. The paper’s experiments with VAEs and GMMs (in supplementary) suggest the effect is quite general. One practical implication (noted in the paper and by commentators) is that users and platforms might need to watermark or detect AI-produced content. If, as one expert phrased it, “the web becomes a blurrier version of itself”, we might need the ability to say “this sentence was written by AI” explicitly, so that search engines and new model trainers can discount it. The EU’s AI Act and other regulations may end up requiring labels on synthetic media partly for this reason. However, the enforcement is tricky when models can evade watermarking or people privately post unlabeled AI content. Another takeaway is the “first-mover advantage” idea. In today’s watch-this-space competition, companies racing to release more advanced models should care about preserving high-quality human data. An early model gets to see clean data; later models might only see leftover scraps. In a sense, whoever collects and locks down legitimate data first might set the gold standard, while anyone who comes later could be handing their model increasingly noisy versions of the truth. From an engineering standpoint, a clear strategy is to always include human-supervised data or diverse real-world logs in retraining. For example, if you have a robotics policy that generates logs, you’d still want a portion of logs recorded by humans (e.g. teleoperated trials or manually annotated data) especially for rare events. One helpful approach is what the authors did in one of their experiments: keep even a small fraction of the original high-quality corpus in each generation. They found that holding even 10% human data practically halted collapse. In domains like autonomous driving or healthcare, this might translate to not training a new model only on old model’s outputs, but mixing in new sensor data or expert demonstrations. Monitoring is another idea. Just as data scientists monitor for dataset shift, we may need “collapse metrics”: for example, track how the entropy or diversity of your model’s output is changing over retraining rounds. If you notice the model’s vocabulary use or its generated text’s perplexity shrinks unusually, it might be a heads-up. The Winssolutions blog example (on telehealth) suggested tracking how often “red-flag” rare phrases appear and watching if those precipitously fall. (In that hypothetical, a telehealth AI kept recommending only generic advice because it forgot the postpartum bleeding checklist, a life-critical tail case.). One could also simulate exclusions: include adversarial examples or oversample rare cases in training to counteract entropy loss. But ultimately, the surest fix is not technical but logistical: ensure the training pipeline retains enough ground truth. After all, deep models can recover from some imbalance if the supervision data is there; they don’t collapse instantly if reminded of the tails. Differentiating Model Collapse from Other Terms. It’s worth clarifying what model collapse isn't. It’s not the same as overfitting, because it isn’t just that a model is memorizing its data. It’s also not standard model drift from changing real distributions. In model collapse, there’s no real “drift” in the outside world — the drift is internal to the training loop. It’s not adversarial per se, since there’s no attacker; it’s a kind of unintentional data poisoning. It is related to the idea of “teacher forcing” distribution mismatch in sequence modeling. In RNN literature, if you train a model on its own predictions instead of the gold sequence, it tends to drift because errors amplify. Model collapse is like that, on steroids across generations. The authors mention “catastrophic forgetting” from continual learning, where a model loses old tasks when learning new tasks. But there, the tasks themselves change; here, the single task (text generation) stays the same. “Poisoning” usually implies someone maliciously introduced bad samples. Collapse is more benign but universal: it’s what happens when no humans stand guard in the loop. This phenomenon also resonates with classic ideas about feedback loops in societal information. For example, if search engines rank results and readers click, new content tends to replicate popular answers, reinforcing the cycle. Large language models may similarly entrench popular phrases more strongly if they only see what they themselves generated as “information”. It’s a kind of echo chamber effect for data. A non-technical way to see it is to recall a famous thought experiment: if everyone just read summaries of books without ever reading the books themselves, over time knowledge would become more and more abstracted, losing details. If each generation’s knowledge base is only a summary of the previous, the summaries get blurrier. Language models are “reading” the internet; if that internet is full of secondhand content, the models will become less grounded in original nuance. Concluding Thoughts. The work of Shumailov et al. provides a rigorous confirmation of a growing concern in the AI community: “AI models collapse when trained on recursively generated data”. It turns out that this isn’t just a wild guess or a hypothetical worry, but a mathematically inevitable outcome under reasonable assumptions. Their repeated experiments across different model families—LLMs, VAEs, simple Gaussians—all showed analogous trends of narrowing distributions and lost diversity. For practitioners, the call to action is clear: preserve your human data. Whether you work on language, vision, robotics, or any AI field that uses past model outputs, beware of pure self-training regimes. If you’re collecting data from the field (e.g., a robot that writes up its work logs), try to include a human-in-the-loop occasionally. Annotate or seed some data with non-AI truth. Log provenance – note which examples were machine-generated – so you can treat them differently. The stakes can be higher in some domains. Imagine medical AI that gradually forgets how to spot an uncommon but critical condition simply because all the new records it's trained on were templated versions of its own previous notes. That is not a distant hypothetical; it’s exactly the kind of collapse this paper warns about. On the research side, this phenomenon opens new questions. Could we design training algorithms that correct for the compression of information? For instance, can we measure how a model’s outputs diverge from a human baseline and adjust? Could we use techniques like reinforcement learning with a diversity reward to push back against collapse? What about differential privacy methods (which also shrink distributions)? Maybe we need a formal theory of “data diversity economics” – where “sampling from clean data” is a resource. In any case, the broad lesson is one of caution. The promise of generative AI is enormous, but it comes with new feedback dynamics we must manage. Model collapse reminds us that data provenance matters. The internet is a wonderful commons for training AIs today, but if it turns into a giant loop of chatbots talking to each other at scale, we could ironically end up with weaker AI. The most human handprint on the data may become the crucial ingredient for reliable models tomorrow. As Shumailov et al. conclude: the value of genuine human-origin data will only grow in a future where generative AI is ubiquitous. References:. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. & Gal, Y. AI models collapse when trained on recursively generated data. Nature (2024). New Yorker Tech Dispatch, ChatGPT Is a Blurry JPEG of the Web (2023). Borji, A., A Note on “AI models collapse…” (Medium, 2024).