From Language Models to LLMs
Language Models Explained
4 min read
A probabilistic view of language
Imagine playing a word association game. Someone says "I'd like a cup of..." and your brain immediately supplies "coffee" — not because you were told to, but because you've encountered that phrase hundreds of times. You've internalized the statistical patterns of the language.
Language models work the same way. They learn the probability of each possible next word, given all the words that came before, by training on enormous amounts of text.
This is not a database lookup. There is no table of "correct continuations." The model has internalized patterns from its training data and uses them to generate new text, word by word.
Two types of language models
Masked Language Models
A masked language model predicts a missing word that can appear anywhere in a sentence — the model sees both the words before and after the gap.
Example: "The cat sat on the ___." The model sees all surrounding context and predicts the blank.
These models are excellent for understanding tasks — classifying text, extracting information, answering questions — but they're not naturally suited for generation, since generating text means producing words in sequence, not filling in blanks.
BERT (from Google) is the canonical masked language model.
Autoregressive Language Models
An autoregressive language model predicts the next word using only the preceding words as context — it generates text left to right, one token at a time.
Example: Given "The cat sat on the," predict "mat." Then given "The cat sat on the mat," predict the next word. Repeat until done.
This is the approach used by GPT and its successors. It maps naturally to text generation: you provide a prompt, and the model continues it, token by token, until it decides to stop.
What "training" means for a language model
A language model is trained on a massive corpus of text — books, websites, articles, code, forums. The training objective is simple: given this sequence of words, predict the next one.
This task is repeated across hundreds of billions of examples. At each step, the model makes a prediction, compares it to the actual next word, measures how wrong it was, and adjusts slightly to do better. Repeated enough times across enough data, the model develops a rich statistical model of language — not just word frequencies, but grammar, facts, reasoning patterns, and style.
What makes a language model "large"?
The "L" in LLM stands for Large — referring specifically to the number of parameters, the adjustable numeric weights inside the model.
More parameters means more capacity to learn and store patterns from training data.
- Early neural language models: millions of parameters
- GPT-2 (2019): 1.5 billion parameters
- GPT-3 (2020): 175 billion parameters
- Modern frontier models: estimated to be in the trillions
Scale has proven to be one of the most reliable levers for improving language model capability. Larger models, trained on more data, consistently produce more coherent, factual, and nuanced text — up to a point that researchers are still actively studying.
Key takeaway
Language models predict the most likely next word given the preceding context. Masked models see surrounding context and fill in gaps; autoregressive models generate text left to right, token by token. Training involves learning from hundreds of billions of examples to internalize statistical patterns of language. "Large" refers to parameter count — and scale has proven to be a powerful driver of capability.
What's next?
Training a language model on the entire internet is expensive and complex. The next lesson looks at the different training strategies — supervised, unsupervised, and self-supervised learning — and why self-supervision is the key that made LLMs possible.