Latest / Embodied AI 101 / When AI Trains on Its Own Output: The Model Collapse Problem
Transcript
- When AI Trains on Its Own Output: The Model Collapse Problem. In 2024, researchers Ilia Shumailov and colleagues published a provocative warning: if we let large language models (LLMs) recursively train on text generated by earlier models, they can gradually collapse. Over successive generations of training on model-made content, the resulting models lose diversity and accuracy – “irreversible defects,” in the authors’ words. In this model collapse scenario, rare or unusual content (“tails” of the distribution) disappears, and models end up “mis-perceiving reality”. In short, AI may be erasing the very nuance that makes our language rich. This stands in stark contrast to the usual AI narrative that more data – even synthetic – is always better. Instead, the study shows that inevitably training on too much AI-written data can create a negative feedback loop. This episode unpacks the model collapse phenomenon reported by Shumailov et al. (Nature 2024) – one of the most influential AI papers of the year – and explores why it happens, how it was demonstrated, and what it means for the future of LLM training. We’ll see that preserving human-written data becomes crucial in an era when the internet could be flooded with AI-generated content. Throughout, we’ll reference the authors’ findings and related analysis to make clear what they did, why it matters, and how developers might adapt to avoid collapse. What Is Model Collapse?. First, what do the authors mean by model collapse? In everyday terms, imagine a scenario where an AI model generates a large body of text, and that text is then used as training data to train a successor model. If we continue this process, generation after generation, something alarming can happen: the models start to believe their own hallucinations. In the authors’ words, model collapse “is a degenerative process” in which each model’s own output ends up “polluting the training set of the next generation”. In effect, models train on a reality contaminated by their ancestors, and begin to systematically forget parts of the true distribution of language. The authors identify two phases of collapse. In early model collapse, the model gradually forgets the tails of the distribution – the rare, unusual content. Over time, the generated text contains fewer and fewer uncommon details. In late model collapse, the degeneration is extreme: the model converges to something carrying “little resemblance to the original” distribution, with much lower variance and possibly collapsing onto a near-deterministic output. One can imagine that after many rounds of recursive training, the model only knows a narrow, repetitive version of language, having erased the wilder corners of human creativity. To illustrate, Shumailov et al. refer to their Figure 1: a cartoon where “model 0” is trained on clean human data, then generates synthetic data for “model 1”, which is trained on a mix of real+synthetic, and so on. Ideally, if everything were perfect, each model’s outputs would just match the real data distribution. But in reality, the authors note, “the data they generate… end up polluting the training set of the next generation”. Each generation’s model is biased by the previous one’s hallucinations. As they state, “being trained on polluted data, they then mis-perceive reality”. In more intuitive terms: every time you train on model-generated text, you risk reinforcing whatever errors or simplifications the model already has. Rare facts or edge cases that the model fails to generate simply vanish from the training pool. Over multiple iterations, even a small bias compounds into a large drift. The authors underscore this process as nearly unavoidable – eventually the model becomes a caricature of itself. Why Does Collapse Happen?. Understanding why model collapse occurs involves a mix of statistical insight and learning theory. Shumailov et al. break down the root causes into three interlocking sources of error that compound over generations. These are worth summarizing carefully:. Statistical approximation error. Every time we sample a new dataset from the old model, we only draw a finite number of examples. Even if the original model perfectly captured the true distribution, sampling at each stage can randomly drop low-probability events. Essentially, if an event is rare, there’s a non-zero chance it never appears in a finite sample. Over many iterations, the missing of rare events is persistent. The authors describe this as a primary source of collapse: because “number of samples being finite… information can get lost at every step of resampling”. As one puts it, even if you had an ideal teacher, an insufficient number of student answers means some knowledge will be missing. Functional expressivity error. This stems from model capacity limits. A model architecture has only so much flexibility, so even with infinite data it might not perfectly capture the true distribution. As the paper notes, neural networks are universal approximators only in the infinite limit of size. In reality, a model might inadvertently assign non-zero probability to impossible outputs or force probabilities to zero for actually possible ones. A simple example: if your true data is a mixture of two Gaussians but you fit only one Gaussian at each step, you inevitably distort the distribution – that single Gaussian might put mass where none should be, and lose one of the modes. Such expressivity mismatches introduce a modeling error even in the first generation and carry through. In our context, an LLM with limited capacity might oversimplify language patterns and thus skew the generated distribution each time. Functional approximation error. Even if the model class is rich enough, practical training adds its own bias. This error comes from the learning algorithm (e.g. stochastic gradient descent) and chosen objectives. For instance, SGDs can have structural bias and objective functions may not perfectly align with the end goal. The paper calls any systematic bias from the training procedure “functional approximation error”. Intuitively, even a large model trained on ample data won’t be perfect; these imperfections also carry forward generation after generation. The authors compare this to a limit where data is infinite and models infinitely expressive – even then, “limitations of learning procedures” can introduce further drift. Each of these errors is small on a single round, but importantly they compound. More data and better models can help to reduce them, but adding more synthetic data can also exacerbate some types of error. For example, a very high-capacity network might “overfit” the synthetic training set and assign weird high probabilities to nonsensical outputs, which then get sampled exaggeratedly by the next model. Shumailov et al. note that in practice we see a feedback loop: each slight imperfection from one generation seeds more errors in the next. This cascading effect tends to cut off the distribution tails (rare things) and gradually shrink the overall support. Put mathematically, the authors even prove a stark theorem in a simple setting: if you repeatedly sample from a Gaussian and refit a new Gaussian, eventually almost surely the variance collapses to zero. More precisely, Theorem 3.1 shows that in this process the fitted covariance goes to 0 with probability 1, and the fitted distribution drifts arbitrarily far from the original Wasserstein-2 distance. In plain language, the model “forgets” its spread entirely and becomes a point mass over time. This toy proof illustrates the inevitability: with enough iterations, the iterative refitting process drives the model to a delta function, no matter how it started. This provides a clean illustration that even a seemingly ideal case (Gaussian data and fitting) leads to total collapse. More generally, Shumailov et al. note that this collapse is inevitable in the limit of infinite generations, even under “almost ideal conditions” (no estimation error, perfect fitting, etc.). In other words, there’s no magic CNN trick to avoid it if you just train one model on the outputs of the last, ad infinitum. The statistics of sampling guarantee the fuzzier corners disappear first; later, the entire model converges to a single point or mode. The authors specifically remark that “even in the best learning conditions, model collapse was inevitable”. Thus, unlike catastrophic forgetting (where models lose old tasks when learning new ones) or adversarial poisoning (where attackers inject bad data), model collapse is fundamentally a built-in risk of recursion in learning itself. Experimental Demonstrations. Having defined the problem, the authors back it up with both theory and experiments. They first demonstrate collapse in simpler models to build intuition, and then in a realistic language-model fine-tuning scenario. The results are qualitatively striking. Simple models: GMMs and VAEs. To show that model collapse is not just an artifact of large neural nets, the team tested much simpler generative systems where behavior is easy to visualize:. Gaussian Mixture Model (GMM): Imagine a toy 2D dataset drawn from two distinct clusters (two Gaussians). They repeatedly fit a GMM (with the correct number of components) to data generated by the previous GMM. Over successive generations, they observe the two clusters merging and eventually vanishing. In fact, after enough iterations, the model’s variance collapsed to essentially zero. The TechTarget summary of the work describes this: “Within 50 re-generations, the data distribution completely changed. By generation 2,000, there was no longer any variance in the data.”. Intuitively, the GMM first misestimates one tail slightly so the next sample has even less variation, then it keeps shrinking. Variational Autoencoder (VAE): They trained a simple VAE on images of handwritten digits (MNIST). The model generates new digit images, which were fed to train the next-generation VAE. Over generations, the digits became blurrier and more indistinct. By a few rounds in, the images were reduced to crude smudges – essentially meaningless blobs. As summed up in a secondary report, “images got progressively blurrier until each digit resembled a roughly uniform smudge.”. Again, rare strokes and details got lost first; eventually even the model’s memory of a particular digit class vanished. These toy experiments confirm the principle: recursive generation causes models to lose fine details and diversity, no matter how trivial the domain. If collapse can turn digits into smears, how bad might it get on real language?. Language-model experiments. The core of the paper is a proof-of-concept with a real language model. The authors fine-tuned a moderate-sized transformer on a text task, then repeatedly trained new versions on synthetic text. Their setting was realistic but computationally practical: rather than retraining a multibillion-parameter model from scratch, they started from a pre-trained OPT-125M model (from Meta’s release) and fine-tuned it on an English text dataset (WikiText-2). This choice was pragmatic – as the paper mentions, training a single moderately large model can produce “twice the American lifetime’s worth of CO₂” – so they settled for a smaller model as proof-of-concept. They note even these experiments required weeks of training. Here is the procedure they followed:. Base fine-tuning (Gen 0): The OPT-125M model is fine-tuned on the real WikiText2 dataset. This yields a well-trained model with a perplexity of about 34 on the test set (compared to 115 for zero-shot). In other words, it learned the distribution of text decently well. Generating synthetic data: They then created a synthetic training set by prompting the fine-tuned model (Gen 0). They took each 64-token block from the original data and had the model autoregressively predict the next 64 tokens (using five-way beam search). By covering the entire dataset this way, they built an artificially generated corpus of the same size as WikiText2. Training Gen 1: Using only that synthetic corpus (no real text), they fine-tuned a fresh copy of the model (actually restarting from the best Gen 0 model). This becomes “generation 1”. Iterate: They repeated the cycle: use gen 1 to generate the next synthetic corpus, and train gen 2 on it (and so on), for several generations. They ran two flavors of this: one where no real data was kept after generation 0 (pure synthetic every round), and another where 10% of the original human data was reintegrated at each step (to see if small “anchors” of real text help). The outcomes are clear: when the model is trained only on AI-written data, it degrades steadily. As each new generation is trained on “polluted” data, the model’s performance on the original task worsens. Quantitatively, the authors report that the initial perplexity of ~20 (for gen0) rises to about ~28 by later generations. (Perplexity is like error: lower is better.) In the mixed setting with 10% real data retained, the degradation is much smaller – only a minor bump up in perplexity. These results match the collapse narrative. Without fresh human text, the model still “learns something” (it doesn’t fail completely) – flatly, it partially retains the task – but it makes more mistakes each time. The reported phrase is telling: models trained on generated data could “adapt to the underlying task, losing some performance, from 20 to 28 perplexity points”. In other words, you can train on synthetic text and not get total gibberish, but you do incur a noticeable loss. Crucially, the histogram in Fig.1b (described in the paper) shows that over generations the distribution of sequences produced by the model shifts. In early gens, the model still mostly produces sequences that the real-data-trained model would also produce. But at the same time, a long tail of weird outputs grows: later generations “start producing samples that would never be produced by the original model”. These are the hallucinations that accumulate. The average and variance of perplexity across samples also steadily increases with each generation. To give a concrete flavor, the authors include an example (Example 1). They show a real prompt about medieval church architecture and the corresponding outputs from Gen 0 and Gen 1 (and onward). In generation 0, the model answers plausibly about Perpendicular-style cathedrals. By generation 1, however, it has already veered off – it starts talking about St. Peter’s Basilica in Rome, which is unrelated to late-Gothic English parish churches. By a few more iterations (shown in the supplement and press reports), the answers become bizarrely unrelated. A commentator reports that by generation 4, the model’s response to the architecture question was “completely unrelated text about jackrabbits”. In other words, the model had completely lost track of the original context. Even if that extreme example is anecdotal, it illustrates how absurd the drift can be when no ground-truth is ever reintroduced. Finally, the experiments confirm that keeping human data around stops collapse. In the second regime with 10% real data each round, perplexity remained nearly stable. The paper notes simply: “preservation of the original data allows for better model fine-tuning and leads to only minor degradation of performance”. In practice, even a small “seed” of genuine text anchored the model’s knowledge. This suggests a practical lesson for training: always keep a human-authored core dataset if you want to keep the model grounded. Implications and Outlook. Why does all this matter? As LLMs become everyday contributors to online content, the next generation of models will inevitably draw on that content. If unchecked, model collapse warns of a grim future: AI systems speaking a very narrowed dialect, blind to subtlety. Shumailov et al. explicitly paint this scenario in their discussion. They liken the flood of AI content to a massive automated poisoning of the web, an echo of past interventions. As they note, search engines have had to fight “click farms” of human-generated spam by downgrading farmed articles. But LLMs turn that concept on its head: the “farms” themselves become automated and ubiquitous. With no easy way to filter the flood, the internet’s training data could become uniformly synthetic – and according to the authors, “no human-generated training data [makes] AI systems malfunction”. A key takeaway is a first-mover advantage. The researchers emphasize that models trained before the AI-content wave will have higher quality, because their training data still contains original human text. Future models scraping the web after it’s populated with synthetic text will be at a disadvantage: their data is contaminated from the start. Concretely, they write: “To sustain learning over a long period of time, we need to make sure that access to the original data source is preserved and that further data not generated by LLMs remain available.”. In essence, early models built on genuine human interaction data enjoy a monopoly on authenticity. The authors also raise important equity and fairness points. The disappearing “tails” of the distribution are often precisely the content relevant to minorities or rare phenomena. They note: “Preserving the ability of LLMs to model low-probability events is essential to the fairness of their predictions: such events are often relevant to marginalized groups.”. If models lose those low-probability dark corners of the web, they risk systematic bias – they simply never learn to talk about the unusual or the minority perspective. This heightens concerns that a synthetic-data-only regime might inadvertently erase voices that are already underrepresented online. Another crucial implication is data provenance. Once AI-generated text is mixed into the data pool, how will future model trainers know which is synthetic and which is human? The paper flags this explicitly: “The need to distinguish data generated by LLMs from other data raises questions about the provenance of content that is crawled from the Internet: it is unclear how content generated by LLMs can be tracked at scale.”. Without some kind of watermarking or tagging of AI text, model builders won’t easily know which portions of the web to trust. The authors even suggest a form of industry coordination: platforms or developers might have to share metadata or databases indicating what text is genuine. Otherwise, “it may become increasingly difficult to train newer versions of LLMs without access to data crawled… before the mass adoption of the technology or direct access to data generated by humans at scale.”. In short, if the web becomes saturated by AI content, we may lose the raw materials needed to train future AI properly. What should practitioners do with these findings? The simplest strategy is already hinted at by the experiments: keep human data in the mix. Even a small fraction of authentic data prevents outright collapse. In practice, this could mean curators of training corpora must deliberately preserve old human-written text or continuously collect fresh human inputs. One could earmark a portion of every training run exclusively for human-sourced material. Other mitigations have been proposed in broader discussions (beyond this paper). For example, watermarking LLM outputs (as OpenAI and others have considered) could allow filters to remove synthetic text from training pools. Platforms might automatically label AI-generated posts to make them identifiable. There may also be algorithmic tricks: injecting noise or diversity, or sampling outputs in ways that preserve rare patterns (the Ali Borji follow-up work suggests the effects depend on how you measure distribution distance, for instance). A provocative perspective in the community is that accumulating synthetic data on top of original data (instead of replacing it) can avoid collapse – essentially the same conclusion as keeping a human-data reservoir. In any case, the core insight is a caution against complacency with “synthetic data” as a free resource. The attractive idea of generating an endless supply of training text has a catch: without ground truth, the models start learning in a hall of mirrors. As one tech commentator put it, data pollution will not only render new models homogeneous but could make “data from human interactions… harder to find and more valuable.”. The advice from Shumailov et al. is clear: treasure your human-written data. It will be the lifeboat for future LLMs in a sea of AI-copy. Looking forward, we can imagine several research paths. One is to explore the boundary of this effect: How much synthetic data is too much? The paper showed even a few generations matter. But what if we artificially constrained models to output more diverse or uncertain sequences (to simulate tails)? The paper tried a “no-repeat” penalty but still saw collapse (repetition alone wasn’t the culprit). Another path is to develop training algorithms that explicitly counteract the bias (for example, regularizing towards prior distributions, or ensembling multiple models’ outputs). Finally, more work can quantify collapse in larger-scale or open settings – say, by analyzing real-world model-view corpora over time – to see if subtle signs have already appeared in live systems. Ultimately, this research reminds us of a fundamental machine-learning principle: Garbage in, garbage out. If the training set becomes full of its own predictions, the system can only reinforce its “beliefs,” good and bad. In contrast, human data provides the grounding truth. In the authors’ words, generative AI’s expansion must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. As we continue building and retraining LLMs, ensuring a steady supply of genuine human language will be crucial to avoid degenerating into our own echo chamber.