Running Out of Data: The Looming Data Shortage

The assumption behind "bigger and better"

Every time a new generation of language models arrives, it's natural to expect the next one to be even larger and more capable than the last. That expectation is built on an assumption: that there will always be more data to train on.

But that assumption is running into a hard limit — and it's one of the more surprising challenges facing the field.

Where would the new data even come from?

GPT-4 was trained on a vast slice of the publicly available text on the internet. That raises an uncomfortable question: if GPT-4 has already read a significant portion of what's out there, where does the data for GPT-5 — or whatever comes after it — come from?

This isn't a hypothetical concern. AI developers face a real risk of running out of fresh, high-quality data, which could slow down how quickly model capabilities continue to improve.

A second-order problem: AI training on AI

The data shortage gets more complicated because of a side effect of ChatGPT's own success. Now that generative AI tools are widely used, a growing share of the new articles, posts, and content appearing online is itself written by AI.

That creates a feedback loop with a real downside. If future models are trained partly on text that earlier models generated, they're learning from repetitive, derivative content rather than genuinely new human knowledge and perspective. Rather than learning fresh, nuanced information, a model risks simply echoing what previous models have already written — amplifying existing hallucinations, biases, and inaccuracies instead of correcting them.

In short: the more AI-generated content fills the internet, the harder it becomes for future models to learn anything truly new from it.

A third pressure: the data is being locked down

On top of a shrinking supply of fresh material, access to existing data is also becoming more restricted. As the hype around generative AI grew, a number of large organizations took legal action to prevent their content from being scraped and used to train AI models without permission.

Some of the earliest lawsuits came from The New York Times, Shutterstock, and best-selling author John Grisham. Around the same time, platforms like Reddit and Quora changed their policies to make it illegal for AI developers — including OpenAI — to scrape their platforms' data.

To work around this growing legal and access barrier, OpenAI launched a content licensing program, striking deals with organizations that own some of the world's largest proprietary datasets. Reportedly, OpenAI has offered publishers between $1 million and $5 million per year for access to their archives, signing agreements with companies like Shutterstock, Axel Springer, The Associated Press, Le Monde, and Prisa Media.

As competition between large tech companies developing LLMs intensifies, expect the price of access to large, high-quality proprietary datasets to keep climbing.

Why this is a genuine constraint on the field

Put these three pressures side by side and the shape of the problem comes into focus:

A finite supply of fresh, human-generated text — much of the easily-accessible internet has already been used.
A growing share of AI-generated content polluting that supply — tomorrow's training data increasingly includes today's AI output.
Increasingly expensive, legally fraught access to what remains — lawsuits and licensing fees are turning "free to scrape" into "expensive to license."

Together, they add up to a real structural constraint on how AI can keep improving. Unlike compute, which can (in principle) simply be built, high-quality, genuinely novel data can't be manufactured on demand.

Diagram — Three Pressures Squeezing the Data Supply

Key takeaway

The assumption that each new generation of language models will simply have "more data" to learn from is running into real limits: the supply of fresh, human-written text is finite, AI-generated content risks creating a feedback loop of repetitive and lower-quality material, and access to existing proprietary data is becoming more restricted and expensive due to lawsuits, platform policy changes, and licensing deals (OpenAI alone has reportedly paid $1–5 million per year to publishers like Shutterstock, Axel Springer, and the Associated Press). Data — not just compute — is becoming one of the central bottlenecks in AI development.

What's next?

You've now seen four real challenges that shape how generative AI gets built and used: unreliable outputs, steep costs, latency, and a looming data shortage. With that grounded view of the landscape, the final lesson brings it all together with a strategic question every organization has to answer: should you build your own AI model, or buy access to one that already exists?