Training Models on AI-Generated Text Has a Failure Mode, and Labs Are Now Actively Managing Around It

The concern known as model collapse describes what happens when a model is trained, generation after generation, on data that increasingly includes output from earlier models rather than solely human-generated content: rare events, unusual phrasing, and minority viewpoints get progressively underrepresented, because each generation of synthetic data reflects what the previous model considered typical, and atypical-but-real patterns that were thin in the training set can disappear entirely rather than merely shrink. Researchers demonstrated the effect clearly in controlled settings, training models recursively on their own outputs until quality and diversity measurably degraded. The practical alarm this triggered was proportionate to a real problem but the mitigation turned out to be more tractable than the original framing suggested. Labs are not simply avoiding synthetic data — a large and increasing share of training data for frontier models is synthetic by design, used deliberately for tasks like generating reasoning traces, code, and instruction-following examples where human-authored data is scarce or expensive to produce. The distinction that matters is not human versus synthetic but curated versus uncurated: synthetic data generated and filtered specifically for a training objective, checked against quality and diversity criteria, behaves very differently from synthetic data scraped indiscriminately from a web that increasingly contains unlabeled AI output mixed in with human writing. The unsolved part of the problem is provenance at web scale. A curated synthetic dataset built in-house for a specific purpose is fully within a lab's control, but the open web that still supplies a large share of general-purpose pretraining data has no reliable labeling of what fraction of any given page was AI-generated, and that fraction is rising as AI writing tools spread across content platforms, marketing copy, and low-effort publishing. Detecting AI-generated text reliably enough to filter it out at scale remains an unsolved and arguably unsolvable problem in the adversarial sense — generation and detection are in the same arms race that has defeated attempts to watermark or fingerprint text robustly. What labs report actually doing about this is less about filtering out synthetic content and more about maintaining strong anchors to verified human-generated and curated sources — deliberately weighting training mixtures to preserve the true underlying distribution of language, facts, and rare events rather than trusting scale to average out any distortion introduced by synthetic content in the mix. That approach treats data curation, not synthetic-versus-human as a category, as the actual lever, which reframes model collapse from an existential risk to the field into an ordinary, if consequential, data-engineering discipline that well-resourced labs can manage and less careful ones cannot.