From N-Grams to Transformers: The Evolution of Language Modeling

The core task: predict the next word

Before tracing the history, let's anchor on what a language model actually does.

A language model predicts what word is most likely to come next, given the words that came before. That's the core task. The entire sophistication of modern generative AI — ChatGPT, Gemini, Claude — grows from this deceptively simple idea.

Understanding how that task evolved explains why transformers are so revolutionary.

The first approach: N-grams

The simplest language models are called N-grams. They predict the next word by looking at only the N-1 preceding words.

A unigram (n=1) predicts the next word with no context at all — just the most frequent word in the training data. Useless for coherent text.
A bigram (n=2) looks at one preceding word. Given "coffee," what word most often followed it in training data? Better, but still very limited.
A trigram (n=3) looks at two preceding words — more context, better predictions, but still severely limited.

The fundamental problem: N-grams have no memory beyond their tiny window. A trigram has no idea what was written three sentences ago. They capture local statistical patterns, not meaning.

A step forward: Recurrent Neural Networks

Recurrent Neural Networks (RNNs) offered a solution. Unlike N-grams, RNNs maintain a hidden state — a form of memory that updates as each word is processed.

An RNN reads a sequence one word at a time. After each word, it updates its internal memory to reflect what it has read so far. When predicting the next word, it draws on that accumulated memory — not just the last N words, but (in theory) the full history.

This worked much better. But RNNs had a stubborn weakness.

The vanishing gradient problem

When training on long sequences, the influence of early words tends to disappear. The mathematical signals used during training — called gradients — diminish as they travel backward through many steps. By the time the signal reaches the beginning of a long document, it's too weak to be useful.

In practice, RNNs effectively "forget" context beyond the last few dozen words.

LSTMs: a memory upgrade

Long Short-Term Memory networks (LSTMs) addressed this with a more sophisticated memory design. LSTMs introduced gates — mechanisms that learned which information to keep and which to discard. Context from early in a document could survive across many steps.

LSTMs produced real improvements in translation and speech recognition. But they had one major drawback: they're slow. Each word must be processed sequentially — you can't parallelize the computation. For training on internet-scale text, LSTMs were simply too slow to be practical.

Transformers: parallel attention

The 2017 paper "Attention Is All You Need" introduced a fundamentally different approach.

Transformers process an entire sequence simultaneously — all words in parallel. Instead of sequential memory, they use an attention mechanism: every word directly attends to every other word, calculating how relevant each word is to understanding the current one.

For each word, the model computes an attention score for every other word in the sequence. These scores determine how much each word influences the representation of the current word. Long-range dependencies — relationships between words many sentences apart — are captured without the vanishing gradient problem.

Because all this computation happens in parallel, transformers train dramatically faster than RNNs or LSTMs on modern GPU hardware.

Architecture comparison: N-gram uses a fixed context window, RNN passes memory sequentially left-to-right, and Transformer attends to all word pairs simultaneously

Why this matters

Transformers aren't just faster — they learn better representations of meaning from text. And because they scale efficiently, they can be trained on corpora that would have been unthinkable for RNNs.

Every major language model today — GPT-4, Gemini, Claude — runs on a transformer. The paper's title turned out to be quite literally true.

Key takeaway

N-grams were the earliest language models — fast but limited to tiny context windows. RNNs introduced memory but suffered from vanishing gradients. LSTMs improved memory but were slow to train. Transformers solved both problems with parallel attention, making it possible to train efficiently on massive datasets. Every modern large language model is built on a transformer.

What's next?

You now understand how language models predict text and how the transformer architecture made scaling possible. The next section looks at what happens when you actually scale these models — how LLMs work, how they're trained, and how they're built.