The Copy of a Copy: Model Collapse and What Really Happens When AI Learns from AI

The Hook: Nine Generations to the Jackrabbits

The text the experiment starts from is remarkably dull. It comes from a Wikipedia article on English church towers and begins: "some started before 1360 — was typically accomplished by a master mason and a small team of itinerant masons, supplemented by local parish labourers, according to Poyntz Wright. […]"

A language model is asked to continue. In generation zero, still fine-tuned on genuine human text, it does so dutifully: "Revival architecture such as St. John's Cathedral in London." Not quite factually right — but recognisably a sentence about architecture.

Then the recursion begins. The model generates text, that text becomes the training material for the next generation, which generates text in turn, and so on. In the first generation the focus shifts to Rome and Buenos Aires: "architecture such as St. Peter's Basilica in Rome or St. Peter's Basilica in Buenos Aires." By the fifth the topic is gone; the text starts mid-word and talks about translations: "ism, which had been translated into more than 100 languages including English". And in the ninth generation one reads this:

"architecture. In addition to being home to some of the world's largest populations of black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits, red @-@ tailed jackrabbits, yellow @-"

A question about Gothic church towers has turned into jackrabbits. Not even real ones — a list of colour variants trailing off into nothing until the text breaks off mid-token. The odd "@-@" sequences, incidentally, are not a failure of the model but the way the wikitext2 corpus encodes hyphens; the model learned them along with everything else and now reproduces the artifact. The paper itself does not explain them.

This example comes from a paper published in Nature in July 2024: Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, "AI models collapse when trained on recursively generated data" (Nature 631, issue 8022, pp. 755–759, online 24 July 2024). Public reception turned it into a prophecy: the internet is filling up with AI text, future models will train on that text, and so AI will choke on its own output. That this tone was not confined to the tabloids is something critics later note explicitly: "Industry leaders, premier research journals and popular science publications alike have prophesied catastrophic societal consequences stemming from model collapse."

This article takes the matter apart, and the result is less comfortable than either camp would like. The phenomenon is real and cleanly measured. The catastrophic prognosis rests on an assumption that is unrealistic. But — and this is the part the reassurance misses — no reassurance follows from that. Dropping the assumption does not make the opposite thesis true, the best-evidenced part of the phenomenon survives the reassurance, and the one paper that examines the regime closest to reality finds collapse at a synthetic fraction of one percent.

Part 1: What "Model Collapse" Actually Denotes

The definition in the paper (Definition 2.1) is more cautious than its reception suggests. Model collapse is "a degenerative process affecting generations of learned generative models" in which the generated data "end up polluting the training set of the next generation".

More important than the definition is the distinction that follows immediately:

"In early model collapse, the model begins losing information about the tails of the distribution; in late model collapse, the model converges to a distribution that carries little resemblance to the original one, often with substantially reduced variance."

Early collapse means: the edges of the distribution disappear. The rare, the unusual, the dialect, the special case, the minority view — everything with little mass in the training material is more likely not to be drawn at all when resampling, and so ceases to exist for the next generation. Late collapse means: the model ends up at a distribution with substantially reduced variance that bears little resemblance to the original. The jackrabbit list is an image of that state.

This distinction is not a detail, and it is worth remembering — it will turn out to be the decisive one by the end of this article. The two stages are neither equally well evidenced nor equally avoidable.

Why the Tails Die First

The mechanism is less mysterious than the name. The paper names three error sources that compound across generations.

First, statistical approximation error: you draw finitely many samples from a distribution. An event with probability 1 in 10,000 is missing entirely from 10,000 draws with replacement with a probability of about 37 percent — that is the limit 1/e, 36.79 percent when computed. What is missing cannot be learned by the next generation, and what it does not learn it cannot generate. The loss is one-directional: tails can disappear, but they cannot return from nothing.

Second, functional expressivity error: no finite network represents an arbitrary distribution exactly. Third, functional approximation error: learning procedures and optimisers are themselves biased.

Notably, the paper shows collapse even when the latter two sources are idealised away — "even for cases with almost ideal conditions for long-term learning, that is, no function estimation error". Finite sampling alone suffices. No flaw in the model is required; it is enough that randomness has a direction.

Mathematically this is a random walk with no restoring force — as long as the only source of information is the previous generation. The whole debate hangs on that subordinate clause.

Part 2: The Experiment, Read Closely

The setup: the model fine-tuned is OPT-125m, a causal language model from Meta, obtained via Hugging Face. The corpus is wikitext2. Generation uses a five-way beam search, and each experiment is run five times with different random seeds. For reference: the original model fine-tuned on real data reaches a mean perplexity of 34, against 115 at the zero-shot baseline.

Crucially, the paper compares two regimes. Verbatim:

"Five epochs, no original training data. Here the model is trained for five epochs starting on the original dataset but with no original data retained for subsequent runs."

"Ten epochs, 10% of original training data preserved. Here the model is trained for ten epochs on the original dataset and with every new generation of training, a random 10% of the original data points is sampled."

Of the first regime — no original data retained — the paper writes that the model loses "some performance, from 20 to 28 perplexity points". Of the second — ten percent of the original data resampled each generation — it says that preserving the original data allows for better fine-tuning and leads "to only minor degradation of performance". No figures are given at that point.

Caution is in order here against the obvious reading, namely that ten percent of real data suffices to avert the decline. That inference is not supported, for three reasons.

First, the two regimes differ in two variables, not one: ten epochs instead of five and ten percent resampled original data instead of none. From two simultaneously altered quantities the effect of neither can be read off. Second, the paper gives no figure for the second regime, and an unquantified observation cannot carry a principal finding. Third — and this is the sentence missing from the reception — it summarises both regimes together: "Both training regimes lead to degraded performance in our models." Ten percent makes the decline smaller, not a non-decline.

The contrast remains an indication that the proportion of real data matters — but an indication is all it is here.

An Ambiguity in the Paper Itself

There is one place where the paper is not unambiguous, and it concerns exactly the point that later proves decisive. The caption to Figure 1a describes a scheme in which data are accumulated: "at step n, data are added to the overall data from step n − 1 and this combination is used to train model n."

The theoretical section, by contrast, introduces a mixing formula, p_i+1 = α_i p_θ(i+1) + β_i p_i + γ_i p₀, in which the three terms weight the new model output, the previous generation and the original distribution — and then sets two of the weights to zero: "we consider β_i = γ_i = 0, that is, data only from a single step are used."

So the schematic describes accumulation while the theory computes with replacement. I am of the opinion that this tension in the text is a substantial reason why the paper was so widely read as a statement about the real internet, even though its mathematical core claim rests on the narrower assumption.

As a footnote on the history of science: that very sentence was mis-set in the original. An Author Correction dated 21 March 2025 (Nature 640, issue 8058, p. E6) records that the text read "α_i = γ_i = 0" where it should have read "β_i = γ_i = 0". A single symbol — but one that decides which term drops out. It is an Author Correction, not a retraction; it says nothing about the conclusions.

What the Authors Themselves Qualify

The paper is considerably more explicit in its caveats than its reception:

"in all experiments, generational learning is only performed on a finite (usually small) number of generations, whereas claims of the section 'Theoretical intuition' are mostly presented in the limit of generations going to infinity."

"complete collapse may or may not occur, even after a small number of steps."

And on scale, the authors concede that they did not test larger models outside the fine-tuning setting — with a remarkable justification: because training a single moderately large model produces twice an American lifetime's worth of CO₂, "we opted to not run such an experiment".

The paper's conclusion is accordingly about data access, not doom. The authors speak of a "first mover advantage" and write: "To sustain learning over a long period of time, we need to make sure that access to the original data source is preserved and that further data not generated by LLMs remain available over time." At the same time they note: "it is unclear how content generated by LLMs can be tracked at scale."

Part 3: The Lever — Replacing or Accumulating

In April 2024 the paper appeared that isolated the decisive difference: "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" by Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey and eleven further authors (arXiv:2404.01413, first version 1 April 2024).

The authors contrast two protocols. Under replacing, each generation's training set is swapped for the output of the last generation. Under accumulating, the new synthetic data are added to everything so far, including the original real data.

For linear regression with isotropic features they prove the result exactly (Theorem 2, for T ≥ d+2 samples per iteration). Under replacing, the expected test error grows linearly in the number of generations n:

E = σ²d/(T−d−1) · n

Under accumulating, by contrast:

E = σ²d/(T−d−1) · Σ_i=1..n 1/i² ≤ σ²d/(T−d−1) · π²/6

The punchline lies in a series every first-year student knows. The sum of reciprocal squares converges — to π²/6, about 1.645. The error is therefore bounded above, independently of the number of generations. Instead of a random walk with no restoring force you have a convergent series. The reason is simple: the share of synthetic data in the total holdings falls with each generation, because the holdings grow.

Two clarifications rarely found in the reception of this result. First, the error still grows monotonically, just towards a bound; the series reaches 1.5498 at ten generations and 1.6350 at a hundred. From the tenth to the hundredth generation, then, only about 5.5 percent is added. Second, and this is the more important number: under this definition E is the estimation error above the noise floor, and at one generation the sum equals 1. The bound π²/6 therefore means up to 64.5 percent more estimation error than for a model that has seen only real data. "Bounded" here means: no crash. It does not mean: free. The authors themselves interpret it soberly: "The summability of 1/i² prevents the test error from growing indefinitely."

Empirically it was tested across three model families: causal transformers on TinyStories (a GPT-2 with 9 million parameters as well as Llama-2 variants with 12, 42 and 125 million parameters), a GeoDiff diffusion model on GEOM-Drugs, and variational autoencoders on CelebA with 200,000 images.

And here the limitations begin — they do not devalue the result, but they bound its reach:

• Not a complete safeguard in every architecture. For the VAEs the test error still rises under accumulation — "albeit much more slowly than with replacing data". The dichotomy "unbounded growth or a constant" is therefore incomplete; there is at least a third possibility, documented in the same paper.
• Contrary evidence. The authors point to Martínez et al., who found "slightly contradictory evidence" for a different architecture on a much smaller dataset: rapid performance deterioration even under accumulation.
• Accumulating means an unboundedly growing dataset. That is a condition, not an aside: later iterations train on larger data volumes than under replacing. Under a fixed compute budget — the production case — the guarantee does not hold in this form.
• The scale. The largest language model is a 125-million-parameter model on TinyStories, a corpus of machine-generated kindergarten English. Extrapolating from there to frontier models would be exactly the move one faults the press for.

The third limitation was measured by Joshua Kazdan, Rylan Schaeffer and co-authors ("Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World", arXiv:2410.16713, NeurIPS 2024 Workshops). They introduce a third protocol, Accumulate-Subsample, in which data accumulate but each model trains on a fixed quantity. The finding: the test error "deteriorates more quickly than Accumulate's loss but more slowly than Replace's loss, and frequently converges, albeit to a higher plateau than Accumulate." So: slow and gradual rather than explosive, but also not bounded as under pure accumulation. The same paper finds something else that complicates any blanket recommendation: "when real data are scarce, an appropriate amount of synthetic data reduces the test loss on real data, whereas when real data are ample, synthetic data increases the test loss on real data." The value of synthetic data depends on how much real data you already have — and it can be negative.

Part 4: The Image Side — When Models Go MAD

In parallel and independently came the paper with the best name in the field: "Self-Consuming Generative Models Go MAD" by Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi and five further authors around Richard G. Baraniuk (arXiv:2307.01850, 4 July 2023; published at ICLR 2024). MAD stands for Model Autophagy Disorder, and the analogy is explicitly mad cow disease: an illness that arises when an organism consumes its own kind.

The formal definition is spare: a MAD process is a sequence of distributions following a random walk such that the expected distance from the true distribution increases over time.

The real contribution is the distinction between three loops:

Measurement uses two quantities readable as quality and diversity: precision and recall. Without sampling bias both fall — "autophagy can lead to a rapid decline of both quality and diversity". With sampling bias, that is, when generation is trimmed towards likely outputs: "with sampling bias, quality can be maintained but diversity degrades even more rapidly."

That is an uncomfortable finding for any practice that filters synthetic data by quality: you buy precision with diversity. The abstract frames it accordingly as an either-or — without enough fresh real data, "quality (precision) or diversity (recall)" decreases.

For the middle row of the table, the paper's own section heading is the most compact summary: a fixed set of real data "may delay but not prevent MADness".

Two limitations belong with this. First, this is an imaging paper: the models tested were DDPM, StyleGAN-2, WGAN and normalizing flows on datasets including MNIST and FFHQ. The authors write that the concepts carry over to text and language models, but offer no experiments of their own for that. Second, they classify the purely synthetic loop as unrealistic themselves: it "does not accurately reflect the reality of generative modeling practice" and is "in a sense the worst case".

Part 5: The Counter-Current — and It Concerns Exactly the Real Case

Anyone ending the story here as "collapse refuted" would have skipped the most important paper. For all the findings so far compare recursive protocols. Real training is none of them: to the best of the literature's knowledge, no frontier lab trains iteratively on its predecessor's outputs. The real case is a different one — a single training run on a corpus with an unknown, growing share of synthetic material. In other words: mixing.

And for mixing, the state of the evidence is not reassuring.

Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian and Julia Kempe, "Strong Model Collapse" (arXiv:2410.04840, first version 7 October 2024), examine precisely this regime within the scaling-laws paradigm. Their first result is explicitly a negative one: "we establish a robust negative result which shows that model collapse generally persists even when mixing real and synthetic data, as long as the fraction of training data which is synthetic does not vanish."

And the magnitude of that fraction is the real shock. From the abstract: "even the smallest fraction of synthetic data (e.g., as little as 1% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance."

One percent. Not half, not a third. And the damage is not a crash but the end of scaling: larger training sets stop helping.

On whether larger models help, the same paper gives an answer with structure rather than direction. On the one hand: "larger models can amplify model collapse." On the other, the theory predicts a reversal beyond the interpolation threshold — "larger models may mitigate the collapse, although they do not entirely prevent it" — so that the authors overall predict "the existence of a double-descent curve regarding the model collapse phenomenon". Anyone hoping that bigger models simply grow out of the problem finds no simple assurance here, but a curve with two branches. The scope must be qualified: the theory is developed for high-dimensional linear regression and random projections on Gaussian data, and for fully trained networks it does not apply "directly" by the authors' own statement.

Pointing the same way with different tools are Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef and Merouane Debbah ("How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse", arXiv:2404.05090, 7 April 2024). From the abstract: "we demonstrate that model collapse cannot be avoided when training solely on synthetic data." For mixing they give an upper bound whose shape is sobering: "the amount of synthetic data should be exponentially smaller compared to real data in order to ensure that p^(m) remains close to p⁽¹⁾."

That sentence has to be measured against its object of study. The experiments run on tiny Shakespeare with a character-level tokenizer and a vocabulary of size 65, and the authors note that their results describe "only the statistical approximation error" and that the purely synthetic scenario is "unlikely to happen in real-world applications but serves as the worst-case scenario". The direction of the finding is robust; transferring the word "exponentially" to today's training runs would not be.

Finally, a paper that frames the damage differently — and may be the most consequential for practitioners. Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton and Julia Kempe, "A Tale of Tails: Model Collapse as a Change of Scaling Laws" (arXiv:2402.07043, first version 10 February 2024; ICML 2024, PMLR 235), describe collapse not as catastrophe but as a change in the scaling laws. Their Theorem 2.1 gives, for a distribution tail with exponent β, the form

E_test ≍ T^{−(β−1)/β} + k^−(β−1)

The second term does not depend on the data volume T but only on k, the truncation point of the tail in the synthetic material. As soon as T reaches roughly k^β, "the AI-generated sample size T ceases to be a 'scalable' resource" — the error runs into a plateau.

That is a different and possibly more unpleasant claim than "the model degrades". It does not degrade. It stops getting better. For mixing they prove a result with a grokking character (Theorem 3.2): the length of the plateau scales roughly as k^β/π, where π is the proportion of real data. At π = 0 the plateau lasts forever; at any π > 0 it eventually ends. Decisive for interpretation: the authors speak of mitigation ("significantly mitigates"), not avoidance. And their grokking result does not hold in pure form for probabilistic models, where it appears instead as an S-shaped curve.

Part 6: Eight Definitions and a Misunderstanding

In March 2025 the paper appeared that names the confusion: Rylan Schaeffer, Joshua Kazdan, Alvan Caleb Arulandu and Sanmi Koyejo released the preprint "Position: Model Collapse Does Not Mean What You Think" (arXiv:2503.03150); the work has since appeared under the changed title "Position: Multiple Definitions & Unrealistic Assumptions of Model Collapse Distract from Real World Threats" as an ICML 2026 paper (PMLR 306, pp. 171990–172003). It is a position paper; it trains no models of its own, but does contain its own numerical simulations.

The central observation: the literature uses "at least eight different definitions" of model collapse. The authors list them — catastrophic increase of population risk; any increase of population risk; asymptotically diverging population risk; collapsing variance; change in scaling law; disappearance of or entanglement of real data modes; disappearance of real tail data; appearance of hallucinated data.

This is not pedantry; it has a concrete consequence the authors demonstrate with an example. Alemohammad et al. declare collapse under Definition 2 (any increase), while Gerstgrasser et al. work with Definition 3 (asymptotic divergence). Hence: "under the same results, model collapse can simultaneously 'occur' and 'not occur' in different researchers' opinions." Two groups can look at the same curve and contradict each other truthfully.

On how realistic the replacement assumption is, the authors are blunt. It is "highly unrealistic", and the justification is disarmingly simple: "After a model finishes training, the entire internet is not deleted, nor is the next model necessarily trained solely on its predecessor's outputs."

They further record that, to the best of their knowledge, iterative retraining has not been used for any frontier model, naming the GPT series, Claude 1 through 3, PaLM 1 and 2, Gemini and DeepSeek V3. That statement must be read precisely, however: it stands under "to the best of our knowledge", concerns closed training procedures, and is about iterative retraining, not about synthetic data as such. That frontier labs use synthetic data is uncontroversial — just not in the recursive generational scheme. A statement of non-knowledge about proprietary pipelines refutes the idealised replacement picture; it does not evidence accumulation.

And the paper draws its own limits. For accumulation with subsampling — the production case — it holds that "no analytic theory has been proven in this case to date." On data economics: "The jury is still out on whether the proportion of real data relative to available data will approach zero." Above all, the authors who appear here as the chief witnesses for the reassurance record explicitly what remains even under accumulation: "Subtle degradations in data distributions might still insidiously occur, such as loss of real tail data."

That brings us back to the distinction from Part 1 — and it is the point of this whole article.

Part 7: What Survives, and a Confusion Worth Avoiding

Three statements withstand the state of the evidence.

First: access to non-synthetic data is the only thing all the papers on the topic converge on. Shumailov et al. put it as a "first mover advantage", Alemohammad et al. as a need for fresh real data, Seddik et al. and Dohmatob et al. as a share that must not go to zero. Anyone training or fine-tuning models should track the proportion of real, demonstrably pre-synthetic data as a metric — and keep it away from zero. The open flank is demonstrability itself: Shumailov et al. name it explicitly as unsolved how AI content could be tracked at scale.

Second: adding is better than replacing — which is not a clean bill of health. For linear regression the difference is proven; empirically it is shown up to 125 million parameters on a toy corpus; for VAEs only in attenuated form; under a fixed compute budget not in this form; and when real data are ample, synthetic material can by Kazdan et al. even increase the test error. The recommendation is therefore: accumulating is the better mode if you need synthetic data at all. The prior question remains whether you need it.

What I expressly do not present as a finding, though it suggests itself: versioned data holdings and avoiding in-place overwriting are good data hygiene, but they follow from none of these papers. Theorem 2 is a statement about the composition of the training set per iteration, not about storage discipline — one can version completely and still train each generation only on the latest state, which is exactly the replacement protocol. I am of the opinion that versioning is nonetheless sensible here, but that is my recommendation, not a result.

Third: quality filters are not free. The MAD finding on sampling bias says that trimming towards likely outputs can preserve quality while diversity collapses faster. Filtering synthetic data by quality shifts the problem from one axis to the other. Where diversity is part of the requirement — and for minority cases, rare failure modes and dialects it is — it has to be measured in its own right.

The Most Common Confusion

Repeating real data is not recursive training on synthetic data. Both are discussed under "data scarcity" and are therefore regularly conflated.

Niklas Muennighoff, Alexander M. Rush, Boaz Barak and co-authors trained, in "Scaling Data-Constrained Language Models" (arXiv:2305.16264, NeurIPS 2023), more than 400 models from 10 million to 9 billion parameters — the largest has 8.7 billion — with up to 900 billion training tokens. Their finding on repeating real data: up to four epochs produce "negligible changes to loss compared to having unique data". Their example: a model with 8.7 billion parameters, four epochs on 44 billion unique tokens, finishes with only 0.5 percent higher validation loss than the single-epoch model on 178 billion unique tokens. Meaningful gains extend to around 16 epochs, "beyond which returns diminish extremely fast".

That is a statement about reusing human text — not about learning from machine text. Conflating the two draws false comfort from one finding for the other problem, or conversely false alarm. The same paper contains further data-scarcity findings (mixing in code doubles the effectively usable tokens; perplexity filtering works, deduplication does not), and these too belong to the scarcity debate, not the collapse debate. In particular, the perplexity filtering that works there on real data does not contradict the price that point three above assigns to filtering synthetic data in a recursive loop: these are different regimes.

Fact Check

146 claims of the draft were checked against the primary sources — arXiv abstract and full-text pages, PubMed Central for the Nature version (CC BY 4.0), Crossref for the bibliographic data, and PMLR and the ICLR proceedings for publication venues. Checking was carried out in two separate passes by reviewers who had not written the text: one for hard facts, quotations and source attributions (66 + 46 + 15 claims), one tasked with attacking the synthesis claims and actively searching for counter-evidence (19 claims).

Result: 8 FALSE, 20 TO BE WEAKENED, 3 NOT VERIFIABLE — 31 corrections in total, plus 6 internal contradictions resolved.

The three most important corrections:

1. The hook was assembled wrongly. The draft quoted a "seventh generation" ("architecture in England") as part of the Nature paper. No generation 7 exists there: the Nature version shows generations 0, 1, 5 and 9, while the arXiv preprint shows 0, 1, 7 and 9. The draft had mixed the two versions and attributed both to Nature. Replaced by generation 5 of the Nature version.
2. The draft's central claim was a fallacy from an uncontrolled comparison. The draft concluded that ten percent of preserved original data had turned "decline into minor degradation". But the two regimes differ in two variables (ten epochs instead of five and the data share), the paper gives no figure for the second, and it summarises both with the sentence "Both training regimes lead to degraded performance in our models." The inference was reduced to a hypothesis and the missing sentence added.
3. One correction made in the first checking pass was itself wrong. That pass had deleted the venue "ICLR 2024" for Alemohammad et al. because the arXiv comments field says "pre-print" — a fallacy from "arXiv names no conference" to "there is none". The paper is published as an ICLR 2024 contribution; the attribution was restored.

In addition, a missing source was added that shifts the state of the evidence: "Strong Model Collapse" (arXiv:2410.04840) examines the mixing regime closest to real training and finds collapse at a synthetic share of one percent. A draft that faults the press for selective reading while omitting this paper would have been vulnerable itself. The resulting rework changed the article's core claim; the introduction, Part 5 and the conclusion were recast accordingly.

Not verifiable and therefore removed or flagged: the characterisation of the Nature paper as "one of the most successful science stories of the year" (no metric given), and the publication venue of "Strong Model Collapse", which is not stated on the accessible pages and is therefore not asserted here.

Note: Automated AI check against primary sources; no human expert review.

The Central Takeaway

Model collapse is a real, reproducible phenomenon — and at the same time a lesson in how one correct finding becomes two false prognoses, one in each direction.

The first false prognosis is the familiar one: AI chokes on its own output. It rests on the assumption that the old data holdings disappear — and that assumption is unrealistic, because the internet is not deleted after a training run. But all that follows is that the pessimistic prognosis is unsupported, not that it is false, and certainly not that the optimistic one is proven. The non-fulfilment of a sufficient condition for collapse is no proof of non-collapse. That is formally the same error, merely with the sign reversed.

The second false prognosis is the reassurance. It overlooks three things. The bound under accumulation is proven for linear regression and still permits up to 64.5 percent more estimation error; it presupposes an unboundedly growing dataset that a production budget does not provide; and it concerns the protocol nobody runs. For the regime closest to real training — one run on a corpus of mixed provenance — the only paper that examines it directly finds collapse at a synthetic share of one percent, and there the damage is not a crash but the end of scaling. For the scale of today's frontier models, this literature contains no evidence at all — in either direction.

What remains, then, is less spectacular and more precise than either headline: the best-evidenced part of the phenomenon is the least dramatic one — and it survives the reassurance. Early collapse, the loss of the distribution tails, sets in quickly and has been reproduced several times. And of all papers, the one that criticises the collapse alarm most sharply upholds it explicitly: "Subtle degradations in data distributions might still insidiously occur, such as loss of real tail data." What disappears is not the model's capability but the rare material within it — the dialect, the special case, the atypical failure mode. That is not a breakdown. It is a quiet narrowing, and it does not show up in average perplexity, only where you look for it deliberately.

The practical prompt follows directly: measure not only quality in your models but diversity — separately, at the tails, and over time. A metric that describes the mean cannot indicate a narrowing of the tails. Anyone measuring quality alone will fail to see precisely the part of the phenomenon that is best evidenced.

And the methodological takeaway reaches beyond AI. In Schaeffer et al., two research groups can look at the same curve and contradict each other truthfully because they define the same word differently — eight definitions for one term are not pedantry but a source of error. So anyone reading a strong claim about a phenomenon should ask two questions before believing or dismissing it: under which definition? And under which assumption about what happens to the old data?

Reflection Question

The model-collapse debate drew its force from an assumption — that the old data holdings are replaced — travelling unspoken from the mathematics into a prognosis about the world. And its reassurance drew force from dropping that same assumption and treating the result as proven.

Which assumption of that kind sits inside a system you are responsible for? That is: where do you have a metric, a model quality or a forecast that demonstrably holds — under a condition you never wrote down explicitly and therefore do not monitor? And would your measurement even show it if not the average but only the tail were getting worse?

Cross-References in the Vault

• How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI – what the laws claim whose change Dohmatob et al. describe as collapse.
• More Parameters, Fewer Errors: Double Descent, Grokking, and the Riddle of Generalization – background for the grokking result and the predicted double-descent curve of collapse.
• From Noise to Image: Diffusion Models and the Physics of Generative AI – the models MAD and the diffusion tests run on.
• Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI – the causal transformers of the accumulation experiments.
• Praise and Blame for the Machine: RLHF and the Art of Teaching AI What We Want – a productive use of model outputs in training that precisely does not lead to collapse.
• The Ghost in the Machine: How to Read a Neural Network From the Inside – what disappears inside the model when the distribution tails die.

Sources and Further Reading

1. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., Gal, Y.: AI models collapse when trained on recursively generated data. Nature 631 (8022), pp. 755–759, online 24 July 2024. DOI 10.1038/s41586-024-07566-y. Full text (CC BY 4.0): https://pmc.ncbi.nlm.nih.gov/articles/PMC11269175/
2. Author Correction thereto: Author Correction: AI models collapse when trained on recursively generated data. Nature 640 (8058), p. E6, 21 March 2025. DOI 10.1038/s41586-025-08905-3
3. Preprint version: Shumailov, I. et al.: The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493 (v1 27 May 2023, v3 14 April 2024). https://arxiv.org/abs/2305.17493
4. Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L., Koyejo, S.: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. arXiv:2404.01413 (1 April 2024). https://arxiv.org/abs/2404.01413
5. Dohmatob, E., Feng, Y., Subramonian, A., Kempe, J.: Strong Model Collapse. arXiv:2410.04840 (7 October 2024). https://arxiv.org/abs/2410.04840
6. Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. L., Koyejo, S.: Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World. NeurIPS 2024 Workshops (M3L, ATTRIB). arXiv:2410.16713. https://arxiv.org/abs/2410.16713
7. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., Baraniuk, R. G.: Self-Consuming Generative Models Go MAD. ICLR 2024. arXiv:2307.01850 (4 July 2023). https://arxiv.org/abs/2307.01850
8. Dohmatob, E., Feng, Y., Yang, P., Charton, F., Kempe, J.: A Tale of Tails: Model Collapse as a Change of Scaling Laws. ICML 2024, PMLR 235. arXiv:2402.07043. https://arxiv.org/abs/2402.07043
9. Seddik, M. E. A., Chen, S.-W., Hayou, S., Youssef, P., Debbah, M.: How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse. arXiv:2404.05090 (7 April 2024). https://arxiv.org/abs/2404.05090
10. Schaeffer, R., Kazdan, J., Arulandu, A. C., Koyejo, S.: Position: Multiple Definitions & Unrealistic Assumptions of Model Collapse Distract from Real World Threats. ICML 2026, PMLR 306, pp. 171990–172003. Preprint version as Position: Model Collapse Does Not Mean What You Think, arXiv:2503.03150 (5 March 2025). https://arxiv.org/abs/2503.03150
11. Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., Raffel, C.: Scaling Data-Constrained Language Models. NeurIPS 2023. arXiv:2305.16264. https://arxiv.org/abs/2305.16264

← All articles