← Course Home

Working with Generative AI

Foundation Models

4 min read

The old approach: one model, one task

For most of AI's history, models were narrow. You trained a model to classify images. A different model to transcribe speech. Another to detect fraud. Another to translate text.

Each was an expert in its single domain. Each required its own training data, its own architecture decisions, its own evaluation pipeline. And crucially, each was useless outside the task it was designed for.

This wasn't a failure — it was the natural result of the compute and data constraints of the time. Narrow models were tractable. General models weren't.

LLMs changed the calculus

When large language models arrived, something unexpected happened: scale produced generality.

GPT-3, trained purely to predict text, could write code it was never explicitly trained to write. It could answer factual questions, translate languages, summarize documents, and solve math problems. Nobody programmed these capabilities in — they emerged from training on enough diverse data at sufficient scale.

This was a qualitative shift. A single model, trained once on general data, could do many things — often without any task-specific training at all.

LLMs also quickly expanded beyond text. The same architectural ideas that worked for language transferred to images, audio, video, and code. Models became multimodal — able to process and generate across content types.

What is a foundation model?

A foundation model is a large, general-purpose model trained on broad data that serves as a base for a wide range of downstream applications.

The term was coined at Stanford in 2021 to describe models like GPT-3, BERT, and DALL-E — models that had grown large and general enough to serve as a foundation rather than a finished product.

The key characteristics:

  • Enormous scale — billions to trillions of parameters, trained on internet-scale data
  • General capability — competent across many tasks without task-specific training
  • Adaptable — can be customized via prompt engineering, RAG, or fine-tuning for specific applications
  • Reusable — one model underpins many different products and use cases

Visual suggestion: A pyramid diagram: foundation model at the base, with arrows pointing upward to multiple downstream applications — customer service bot, coding assistant, document summarizer, medical tool — all built on the same base.

Why this matters strategically

Before foundation models, building an AI application meant building a model. You gathered data, designed an architecture, trained the model, evaluated it, and deployed it — for each application.

Foundation models flip this. You start from a capable base and adapt it. The expensive, data-hungry, compute-intensive pretraining is already done. You're working with a model that already understands language, reasoning, and world knowledge — and you're shaping it toward your specific use case.

This dramatically lowers the cost and expertise required to build AI applications — while raising the performance ceiling.

The concentration of power

There is a trade-off. Building and maintaining a frontier foundation model requires staggering resources — billions of dollars in compute, world-class research teams, and proprietary training datasets.

This limits who can play at the frontier level. OpenAI, Anthropic, Google DeepMind, Meta AI, and a handful of others are the primary developers of frontier foundation models. Most organizations — even large ones — are consumers of these models, not producers.

As Sam Altman of OpenAI has noted, the development of the most capable foundation models will likely continue to be dominated by Big Tech and well-funded institutions. The competitive advantage for everyone else lies in how effectively they apply foundation models — not whether they build their own.

Key takeaway

Traditional ML models were narrow specialists. LLMs introduced general capability that emerged from scale. Foundation models formalize this: large, general-purpose, adaptable models that serve as the base for building many different AI applications. Their development is resource-intensive and concentrated — but their use is becoming broadly accessible.

What's next?

Foundation models are reshaping what's possible — but they come with real-world friction. The next set of lessons looks at the practical challenges organizations run into when building with generative AI: unreliable outputs, steep costs, slow response times, and a looming shortage of training data.