Latest / Embodied AI 101 / AI Model Collapse: What Happens When AI Trains on Its Own Outputs
Transcript
- AI Model Collapse: What Happens When AI Trains on Its Own Outputs. Imagine you take a photocopy of a photograph, then make a photocopy of that photocopy, and so on. Each generation is slightly noisier: eventually the image degrades into static. A striking new study by Shumailov et al. (2024) shows that something eerily similar can happen with machine learning models. When a generative AI model is trained primarily on data produced by earlier versions of itself, the model’s generative capabilities degrade over time. In their Nature paper, the authors define this phenomenon as “model collapse”, a degenerative feedback loop in which successive generations of AI forget important parts of the original data distribution. This result has profound implications for how we collect and curate training data for future models. If unchecked, it suggests that large AI systems might unintentionally “pollute” their own training data, leading to narrower, less informative models. Shumailov et al. found that “indiscriminate use of model-generated content in training causes irreversible defects in the resulting models”, so that the “tails of the original content distribution disappear”. More vividly, Dr. Ilia Shumailov describes model collapse as “AI spiraling into the abyss, feeding on its own mistakes and becoming increasingly clueless and repetitive”. In practical terms, this means that as models learn from their own synthetic outputs, they gradually lose the ability to produce the full diversity of the original data. Rare or unusual events (the heavy tail of the distribution) disappear first, and eventually the model’s output may converge to a near-constant or nonsensical blurb. This is no minor quirk – it is a fundamental limit on how well generative models can learn from data that has itself been generated by AI. In what follows, we’ll unpack the model collapse phenomenon: what it means, why it happens theoretically, and how the authors demonstrated it in practice. We’ll see examples from simple Gaussian and mixture models up through real language models. Crucially, we’ll explain why repeatedly training on synthetic data forces a model to “mis-perceive” reality, forgetting improbable but important content in favor of high-probability modes. Finally, we’ll discuss the urgent implications: if ever-larger models are to be trained on the future (increasingly synthetic) Internet, how do we prevent this insidious collapse?. What is Model Collapse?. In colloquial terms, model collapse is the idea that a generative AI system will gradually degenerate if it is trained on data that comes from earlier AI systems – especially including itself. Put simply: a model’s outputs eventually “poison” the dataset of the next model. In the words of Shumailov et al.: model collapse is “a degenerative process whereby, over time, models forget the true underlying data distribution”, even though the data-generating process hasn’t changed externally. Think of it like a self-reinforcing echo chamber: small imperfections accumulate until the model’s world-view drifts far from reality. The authors formalize this with a definition (see Fig. 1a of their paper). Imagine we train Model 0 on some true dataset (say Wikipedia) to learn distribution $p_0$. We then generate a new synthetic dataset from Model 0 and use that to train Model 1. Next, we train Model 2 on data generated by Model 1, and so on. Each generation’s training data is increasingly “polluted” with synthetic content. Model collapse refers to the deterioration that emerges if one continues this cycle indefinitely. Shumailov et al. distinguish two stages of collapse:. Early model collapse: The model begins to lose information about the tails of the distribution. In practice, this means rare or low-probability events present in the original data get dropped out. Late model collapse: Eventually the model’s learned distribution converges to something very narrow, often a single mode (point estimate). The variance of the model’s output shrinks dramatically relative to reality. In both cases, the model “mis-perceives reality” because it never sees the original distribution firsthand – only its own echo. Three key sources of error combine to drive collapse in generative training:. Finite sampling (statistical) error: No matter what, each generation only has finitely many samples. Some rare events simply won’t survive the random draw. Over many generations, even minute sampling noise compounds. Model expressivity limits: A neural net (or other model) is not truly infinite in capacity. It may introduce tiny distortions: giving non-zero probability to things outside the support of the original data, or zeroing out marginal parts of it. For example, modeling a two-Gaussian mixture with a single Gaussian will inevitably smear out or collapse one mode. Even perfect knowledge can be undone by expressivity mismatch. Learning (functional) approximation error: The learning algorithm (optimizers like SGD) and chosen objectives also introduce bias. Training dynamics can systematically mis-estimate probabilities. For instance, some analyses show SGD has a structural bias even in overparameterized regimes. In a generational setting, these errors stack up. Even if each model is slightly overparameterized and trained carefully, the combination of a small sampling glitch, a tiny expressive mismatch, and an SGD bias can lead the next model to gradually wander off. Over many iterations, the drift is pulled in the same direction (shrinking variance, cutting tails) and cannot reverse. Intuitively, each step of “learning from the previous model” is like resampling the same population with tiny flaws – and randomness ensures that eventually only a few outcomes survive. Mathematically, Shumailov et al. show that this is essentially guaranteed over the infinite horizon. In a simple discrete example, one can view the process as a Markov chain: if at any stage a category’s probability is small enough to draw zero samples, that information is lost forever, and the chain moves closer to an absorbing state (a delta function at one category). More generally, they prove a powerful Gaussian-case result. Suppose at each step you estimate the mean and covariance from samples of the previous model’s Gaussian. Then almost surely, the covariance collapses to zero variance (a single point) as generation number grows, and the model’s distribution drifts arbitrarily far from the true distribution. In plain terms: even if you perfectly propagate means and variances at every stage, the generative distribution still narrows until it has no width, diverging in Wasserstein distance from reality. Put simply: even an idealized sequence of generative models will collapse unless you bring in fresh real data. Shumailov et al. highlight that this is inevitable if one just keeps feeding model outputs to models. The only way out is injecting ground-truth content. Intuitions: How Does Collapse Occur?. Beyond formal proofs, it helps to build an intuitive picture. Imagine a probability distribution over many text sequences (or images, or whatever). A training model samples a finite dataset from it. The very rare sequences might not appear in that sample. Now the model memorizes or learns this dataset and becomes the new “generative process.” When you resample from the model, those missing rare sequences have zero chance of reappearing. This is the first bite out of the tail. Next generation: you approximate this now-slightly-shrunk distribution (imperfectly, since the model might fudge some details). Resample again. More rare events drop out. The support of the distribution keeps shrinking. In neural terms, this happens at two levels. First there’s the data sampling glitch: by chance, the very low-probability training points get lost. Second, the neural net might even make confident errors: it might unknowingly assign zero probability where there should be some, or incorrectly boost some probabilities. Each generation compounds the gaps and bumps introduced previously, pushing the distribution to concentrate on what’s easiest (highest-probability) to reproduce. A helpful analogy: think of trying to describe the world by repeatedly summarizing your own previous summary. Each retelling loses a bit of detail. Eventually your story lops off entire branches of detail that were rare to begin with. The authors compare it to making successive photocopies: eventually you just get a blob. The upshot is that over generations, the learned model becomes increasingly “brittle.” It starts to favor the high-probability core of the data and loses the long tail. Odd corner cases and minority examples vanish. In late stages, the model might only output a very narrow set of familiar phrases or patterns, sometimes garbled or repetitive, because it has forgotten all else. Demonstrations on Simple Models. Shumailov et al. first illustrate collapse on toy generative systems, to build intuition. In the supplement they discuss two stylized settings. Discrete example (no model error): Imagine an originally uniform distribution over a finite set of states. We take a large but finite sample, train a model that exactly reproduces that empirical distribution, and resample again. One can show this is a Markov chain with absorbing states (all probability on a single state). Statistically, eventually only one category will remain with non-zero mass, chosen according to the original distribution. This proves the “late collapse” to a pure mode. Gaussian example: More realistically, think of continuous data modeled as Gaussian. If each generation fits a Gaussian by sample mean/variance and resamples, the authors show analytically that the sample variance shrinks to zero almost surely and the distance to the original grows without bound. In other words, even if the original data have spread, after many self-train rounds the model’s “learned covariance” crashes to a point. They also note the early stage: at the start, only the very lowest probability events disappear. This is because if a data point has probability $q \le 1/M$ (with $M$ samples), expected count is < 1, so likely zero draws. So before converging to a point, the model first carves off the tails of the distribution where $q$ is tiny. These toy cases drive home the inevitability: unless you keep seeing new true data, any finite-sample model will slowly narrow down to an impoverished version of reality. The interesting point is that this holds even if each model is trained perfectly (no mislearning) – it's the act of resampling with finite data that creates the bleed. Collapse in Neural LMs: The Wikitext Experiments. The real shocker is that this abstract issue actually shows up in practical neural models. Shumailov et al. ran extensive experiments on a small language model to demonstrate collapse on text. Here’s what they did:. They took the OPT-125M causal language model (a 125-million parameter transformer released by Meta) as the base model, fine-tuned on the Wikitext-2 dataset. This model had an initial validation perplexity of about 34 (down from 115 zero-shot). Data generation: To create training data for the next generation, they fed the model each Wikitext-2 context (block of text) and asked it to generate 64 tokens. They used a 5-beam search to get deterministic (or near-deterministic) “best guess” continuations. Going through the entire original dataset, they compiled an artificial “synthetic” dataset of the same size as Wikitext-2. (If the model had zero error, this synthetic data would replicate the original content exactly. In practice, it contains mistakes and strange loops.). Generation loop: Starting with the model fine-tuned on real Wikitext-2 (call this gen 0), they then fine-tuned a fresh copy of OPT-125M using only the synthetic data from gen 0, to get gen 1. Then they generated new data from gen 1, trained gen 2, and so on. They repeated this for up to 9 or 10 generations, each time discarding the model from 2 or more generations ago. Training regimes: They tried two settings. In one, after the initial round, they used only synthetic data in each new generation (no real data retained). In the other, they kept 10% of the original real data in the training pool for each generation (so the model never entirely forgot its tuning on real Wikipedia). In all cases, each generation started from the previous model’s weights and fine-tuned further. Each experiment was run five times with different random seeds to ensure consistency. The results were clear and concerning. In the pure-synthetic regime, the model’s performance on the original task steadily degraded. Over 5 generations, the average perplexity (measured on the original Wikitext-2 test set) nearly doubled. In the 10%-real regime, the degradation was much less – meaning that even a small anchor of genuine data kept the model roughly on track. (Specifically, they reported that without real data, the model “lost some performance, from 20 to 28 perplexity points”, whereas with 10% real data, there was “only minor degradation”.). More than just raw metrics, the distribution of outputs changed qualitatively. The paper shows histograms of perplexities of individual generated sequences (Figure 1b–c). As generations progressed, more and more sampled sequences were easier (lower perplexity) for the original model. In other words, each new model became biased toward producing the highly likely text that generation 0 (and the original training data) liked. At the same time, a “long tail” of bizarre sequences appeared – outputs that would almost never come from the original model. Fig. 1 of the paper illustrates this: by generation 5 or 9, the model is churning out oddities. For example, given a prompt about Gothic architecture, Gen 0 responds sensibly about cathedrals, but Gen 5 spirals into a strange listing of languages, and Gen 9 devolves to nonsense about “black-tailed jackrabbits”. These gibberish outputs reflect the accumulating errors: they have very low probability under the real data distribution, meaning the model is mis-perceiving reality based on its inherited flaws. Importantly, the authors checked that this collapse wasn’t driven merely by a well-known quirk of neural text generation: repetition. Many LMs tend to get stuck repeating phrases (the problem text generators). Could it be that each model was just issuing more and more repeats, so they blamed "collapse" incorrectly? To rule that out, they ran a variant where the model was penalized for repetitions during generation. This forced it to avoid loops, but the result was paradoxical: the model then patched in even more low-quality, improbable text to diversify, and the collapse effect was stronger. In short, forcing the model to avoid repeats did not fix the core issue; in fact, it made performance worse. The synthetic-data-trained models had increasingly low diversity of high-quality phrases, and even when stripped of loops, they still trended toward high-likelihood yet low-variance content. Another crucial finding: fine-tuning doesn’t cure collapse. Even though each generation started from a strong pretrained model, further fine-tuning on synthetic data still degraded the model. Over generations, fine-tuning on its own outputs made the model gradually “forget” the true pattern of English. This implies that continuous training on synthetic web content – something many practices involve – can erode an LLM’s grounding in real language. What the Experiments Tell Us. The LLM experiments confirm the theoretical picture. The models were learning: even after multiple synthetic training rounds, the model could still write somewhat coherent English, and training with only synthetic data still achieved some adaptation (perplexities were not astronomical, just worse). But the quality and diversity of the output degraded. Concretely:. Shift to high-probability phrases: Over generations, the model increasingly parroted phrases it “knew” very well from the original corpus. These high-probability chunks accumulate in the output. It’s like the model learning to play only the very common notes and ignoring the rarities. The perplexity histograms show a big spike of sequences that the original model already gave high probability to. Emergence of improbable errors: Simultaneously, the model began hallucinating strange content – sequences that the original model (and humans) would never produce. These represent the model’s own invented errors accumulating. In probability terms, the model is assigning non-negligible probability mass to things outside the original support. The authors note this as a hallmark of late collapse: the model has invented its own improbable sequences, essentially “misperceiving reality”. First-mover advantage: A revealing insight is that early generations fare better simply by virtue of being closer to the original data. Shumailov et al. describe a “first mover advantage” – the very first model trained on human data has a critical edge. Each successor suffers from inherited bias. So if we think of GPT-3 vs GPT-4 vs beyond, GPT-3 (trained on fresh data) might remain superior in some sense to a hypothetical GPT-5 trained mostly on GPT-3’s own generations. Essentially, whoever trains first “locks in” a more faithful version of the data. Subsequent models that train on tainted data will drift away. Data retention matters: Their two training regimes drive home an important mitigation: keep some real data around. Even 10% of genuine Wikitext injected each generation nearly arrested the collapse. In practice, this suggests that future training pipelines must ensure that model training is regularly anchored by human-created content, rather than relying exclusively on scraped (and increasingly synthetic) web text. Irreversibility: Perhaps most starkly, the authors warn that the damage is irreversible if allowed to propagate unchecked. Once the model gets trapped in this feedback loop, simply adding a bit of real data later may not fully recover the lost tail. Stopping collapse is easier by not letting it start. The authors explicitly caution that as AI-generated content floods the Internet, genuine human-interaction data (like social media with human users) will become more precious for training new models. Implications and Mitigations. The discovery of model collapse is a real wake-up call for large-scale AI development. Virtually all contemporary LLMs and image models rely on giant unlabeled web corpora. But in the last few years those corpora have been getting “contaminated” with AI-generated text and images. If this continues, any new model trained the usual way will be training on data that includes (or is dominated by) its own ancestors’ outputs. According to Shumailov et al., that will slowly erode model quality in subtle but severe ways. Fairness and tail events. Losing the tail of the distribution can have real-world consequences. Rare phrases or underrepresented viewpoints might disappear from the model’s vocabulary. The authors note that these low-probability events often correspond to marginalized perspectives or critical but complicated technical details. If a language model progressively forgets them, its answers become biased towards the “mainstream” narrative. In robotics or vision, this could mean failing to recognize unusual situations the system might actually encounter. Security / data poisoning analogy. The authors mention a loose analogy to data poisoning attacks, where an adversary intentionally injects bad examples to skew a model. Here, the cause is not malicious, but the effect is similar: the distribution shifts away. Unlike classic poisoning—which is often targeted at specific tasks—model collapse is a global side-effect of training procedure. It’s a self-inflicted poisoning, in a sense. Comparison to catastrophic forgetting. It’s tempting to liken collapse to catastrophic forgetting (where a model sequentially trained on new tasks forgets old ones). But those are different. Catastrophic forgetting usually happens when tasks change and the model “overwrites” old knowledge. Model collapse is peculiar: even though nothing explicitly changed distribution (the original world hasn’t shifted), the model still forgets parts of it. It’s not forgetting a label – it’s losing probabilities. It actually learns less than it should because its training data was missing those examples. Scaling and future models. The authors stress that collapse could become more pronounced as we push to larger models trained on ever-larger datasets scraped from the web. Present-day models like GPT-4 were largely trained on data up to around 2021 – mostly human text. But what about the corpus for GPT-6 or 7 in 2030? If it contains 50% or more AI-generated text, the collapse effect could make the model steadily worse in certain respects. The fear is that future LLMs might become algorithmically conservative, reproducing popular tropes but failing at nuance, because they trained on an Internet-literature shrunk by their predecessors. Tracking and curation. An obvious mitigation is to tag or filter synthetic data. If we can reliably tell which documents are AI-generated, model trainers could exclude them or treat them differently. But that’s a huge challenge: as authors point out, “the need to distinguish data generated by LLMs from other data raises questions about the provenance of content crawled from the Internet”. Currently, LLM provenance is mostly invisible. Possible solutions might involve industry-wide standards or metadata indicating the source. But such coordination is far from trivial. Human in the loop. Another safeguard is to continually inject verified human-generated data into the training mix. This could come from curated data sources, licensed content, or even ongoing annotation projects. The key is to ensure that each new model sees enough fresh real-world diversity. In effect, model collapse “promises to become ever more challenging” as human vs. AI content blur. First-mover advantage reinterpreted. There is an interesting silver lining: if you have a powerful model trained on clean data, that model will remain more trustworthy (its descendants degrade). This means early models have a built-in advantage. In practical terms, it could make open data sets like Wikipedia extraordinarily valuable; any domain dominated by human curation (scientific literature, vetted forums, etc.) becomes gold. Related Work and Concepts. This phenomenon brings together ideas from several strands of machine learning research:. Data Distillation / Bootstrapping. In ML, it’s common to use teacher networks to label new data (semi-supervised learning, knowledge distillation). The surprising twist here is that even unsupervised learning (next-token prediction) suffers a similar fate if bootstrapped repeatedly. Generative Feedback Loops. In robotics and control, training a model-based controller on simulated rollouts can drift if simulation is imperfect. The scenario here is analogous: if your simulator (the AI model) is flawed, retraining on its outputs eventually diverges from real physics. Catastrophic Forgetting – but worse. As noted, task-forgetting in continual learning is somewhat related. However, model collapse is more fundamental: it doesn’t require a task switch or forgetting; it happens under a stationary task with stationary distribution, strictly due to recursive data generation. Catastrophic forgetting usually requires model parameters shifting, whereas collapse can happen purely from data distribution shift, even if the parameters move minimally each round. Data Poisoning. Carlini et al. and others have studied how adversarial data (maliciously crafted) can alter model behavior. Model collapse is not adversarial per se, but the net effect is like a slow poisoning where the “poison” is the model’s own innocuous mistakes. “AI safety” echoes. Some safety researchers have warned about AI-trained-on-AI corpus. For example, Yudkowsky and others have mentioned that if the internet becomes mostly “AI crap”, new models will degrade. This work provides the first rigorous study of that intuition. Conclusion. Shumailov et al.’s work is a timely alarm bell. As we automate content generation, we risk creating a feedback loop that hollows out the richness of our data. The paper makes clear: a model trained on its own progeny gradually contracts its world. Early and late stages of collapse mean losing oddities and converging on narrow consensus. For robotics and ML practitioners, the takeaway is stark. If your new vision or language model is going to be trained on scraped web data, you must ensure that fresh, human-origin data remains in the mix. Ignoring this could make future models surprisingly brittle on corner cases. In some scenarios (e.g., safety-critical perception), that brittleness could be unacceptable. Mitigation might require new dataset curation techniques: for instance, labeling samples by whether they are AI-generated, or privately sharing large vetted corpora among research groups. At minimum, researchers should be aware of this risk. In narrative terms, model collapse tells us that an AI system cannot safely “eat its own tail” forever. The Internet of tomorrow will contain more AI-written text and images; if our training pipelines do not adapt, we may inadvertently train models that “think” they live in a poorer, predictable world. Preserving access to genuine human content (the rare events, the quirky facts, the counter-narratives) will be a key challenge for the next generation of AI. References: The phenomenon of model collapse and its implications are presented in Shumailov et al., Nature 631, 755–759 (2024), with supporting theory and experiments. A commentary by Shumailov (and colleagues) summarises the dangers in accessible terms. Further details are available in the paper’s supplementary materials.