From Language Models to LLMs
How LLMs Are Trained
8 min read
The challenge: labeled data doesn't scale
In the Introduction to AI course, we saw that supervised learning requires labeled data — examples where a human has already provided the correct answer. Think of the flashcards you might use to study for a vocabulary quiz: one side shows the word, the other side shows its meaning. Someone has to write that meaning down before the flashcard is useful.
For many tasks, this works well. A model that detects spam email can learn from a million messages a human has already marked "spam" or "not spam." A model that prices houses can learn from past sales where the final sale price is already known.
For LLMs, this approach hits a wall almost immediately — not because labeling is hard, but because of scale.
Consider what an LLM needs to learn from. GPT-3 trained on roughly 500 billion words — pulled from books, websites, Wikipedia, code repositories, forums, scientific papers, and more. Picture a stack of printed books tall enough to need an elevator, and then imagine a teacher reading every single page and writing "the correct answer is ___" in the margin. To label all of that with human-provided answers would require:
- Deciding what "the correct answer" even is for a random page of raw text
- An enormous number of human annotators working around the clock
- Costs in the hundreds of millions of dollars — at minimum
- Years, if not decades, of time
It's not just impractical. It's conceptually incoherent. If you handed someone a paragraph from a novel and asked them to "label the correct answer," they'd reasonably ask: the correct answer to what question? Raw text doesn't come with a built-in question and answer the way a labeled photo does ("Is this a dog? Yes.").
Unsupervised learning doesn't help either
The other obvious alternative is unsupervised learning, which removes the need for labels entirely. A model is handed a pile of raw, unlabeled data and asked to find structure in it on its own — for example, grouping similar items together (a technique called clustering).
This solves the labeling problem. But it introduces a different one: it lacks a clear objective.
Imagine handing someone a box of mixed puzzle pieces with no picture on the box and saying "organize these however makes sense to you." They might sort by color, by shape, by size — there's no single "correct" grouping to aim for. That's exactly the situation an unsupervised model is in. It can cluster sentences by topic or writing style, but it has no concrete task — like "predict the next word" — to practice and improve at. Without that kind of target, it can't develop the deep, structured understanding of grammar, meaning, and context that generating fluent text requires.
So researchers needed something that combined the best of both worlds: the scale of unsupervised learning (no labeling required) with the clear objective of supervised learning (a specific right answer to aim for).
A quick comparison
Before we get to the solution, here's how the three approaches stack up side by side:
| Supervised learning | Unsupervised learning | Self-supervised learning | |
|---|---|---|---|
| Purpose | Learn to predict a specific, known answer (e.g., "is this spam?") | Discover hidden structure or groupings in data | Learn general patterns by predicting parts of the data from other parts |
| Where labels come from | A human (or paid labeling team) writes them by hand | No labels are used at all | The data generates its own labels automatically |
| Data requirements | Smaller datasets, but every example must be labeled — expensive and slow to produce | Large amounts of raw, unlabeled data | Massive amounts of raw, unlabeled data — the more, the better |
| Typical use cases | Spam filters, medical image diagnosis, house price prediction | Customer segmentation, anomaly detection, topic discovery | Pretraining large language models like GPT, building general-purpose language understanding |
The key column to notice is "where labels come from." Self-supervised learning is the only approach where the data itself hands over the answer key — which is exactly what makes it possible to train on the scale of the entire internet.
The solution: self-supervised learning
Self-supervised learning offers an elegant middle path. The model generates its own labels from the raw text — no human annotation required, but with a clear, concrete learning objective.
It works a bit like a fill-in-the-blank quiz that grades itself. Here's the process for language models, step by step:
- Take a sentence from the training corpus: "The transformer architecture changed the field of NLP."
- Hide the last word, turning it into a quiz question: "The transformer architecture changed the field of ___."
- Ask the model to guess the missing word.
- Compare the model's guess to the word that was actually there ("NLP").
- Adjust the model slightly so it's more likely to get a similar question right next time.
Notice what just happened: the "label" — the correct answer, "NLP" — came directly from the original sentence. Nobody had to write it down, look it up, or decide it was correct. The text already contained the quiz question and the answer key, bundled together. The model effectively plays both roles at once: it's the student attempting the quiz, and the teacher holding the answer key.
This trick — hide a piece of the text, ask the model to predict it, then reveal what was actually there — can be repeated for every sentence on the internet, billions and billions of times, with zero human effort beyond collecting the raw text in the first place.
Why this was the key insight
Self-supervised learning unlocked training at internet scale. Because labels are generated automatically from the text itself:
- No annotation cost — you can train on terabytes of text without paying a single annotator. The "answer key" is baked into the data.
- No human bias in labeling — the model learns from language as it's actually written and used, rather than through the lens of what a labeling team decided was "correct."
- Massive diversity — training data can span every topic, writing style, language, and subject domain that exists in writing. No human labeling project could ever cover that much ground.
Think of it like the difference between learning a language only from a textbook with a hundred pre-written exercises, versus learning it by reading every book, news article, recipe, comment thread, and instruction manual ever written — and quizzing yourself on every single sentence along the way. The second approach produces a far broader, more flexible understanding.
The result is a model with a broad, general sense of how language works — not finely tuned for any one task, but reasonably capable across a huge range of them.
After pretraining: supervised fine-tuning
The model that comes out of this process — often called the base model — is powerful, but raw. It's extremely good at one narrow thing: predicting what text statistically tends to come next. That's not the same as being helpful. A base model asked "How do I bake bread?" might continue with more questions in the same style ("How do I bake a cake? How do I roast a chicken?") rather than actually answering, because that's a pattern it has seen often in lists of FAQs.
To turn this raw prediction engine into a helpful assistant, models like ChatGPT go through a second stage: supervised fine-tuning. Here, the labels return — but in a much smaller, more carefully curated form. Human reviewers write examples of high-quality responses, and rate or rank the model's own answers by how helpful, safe, and honest they are. The model is then retrained to produce the kinds of responses humans preferred.
This second stage is what turns a model that merely continues text into an assistant that answers questions, follows instructions, and holds a conversation — the kind of behavior you experience when you chat with tools like ChatGPT.
Key takeaway
Supervised learning is too expensive and conceptually limited to train on internet-scale text — there's no practical way to hand-label hundreds of billions of words. Unsupervised learning solves the scale problem but lacks a clear objective to train toward. Self-supervised learning solves both: the model hides pieces of real text and quizzes itself on them, generating its own labels for free and learning from practically the entire written internet. After this pretraining phase produces a broadly capable but raw base model, a second phase — supervised fine-tuning with curated, human-rated examples — shapes that base model into a genuinely useful, conversational assistant.
What's next?
Now you know how LLMs are trained. The next lesson walks through the full pipeline — every phase from architectural design through evaluation — to show how a language model gets built from scratch.