← Course Home

Practical Challenges in Generative AI

Latency: Why AI Sometimes Feels Slow

4 min read

Speed isn't a nice-to-have — it's the product

Today's users have very little patience for slow-loading anything. That's true for websites and apps in general — but it becomes especially critical when AI is involved.

When a business builds a customer-facing product around AI, or when a company wants to use AI to boost its own team's productivity, the stakes go up. A delay isn't just an annoyance anymore — it directly frustrates users, slows down operations, and can affect revenue. An AI feature that's accurate but slow can still feel like a broken feature.

So why is speed — latency — such a persistent challenge for these systems?

The root cause: language models generate one word at a time

The biggest latency challenge with today's large language models comes from how they're built: they use an autoregressive architecture, which means each word (technically, each token) the model generates depends on the words that came before it.

This creates a strictly sequential process. The model can't generate the fifth word until it has generated the fourth — because the fourth word is part of the context it needs to decide what comes next. Each step has to wait for the one before it to finish.

Here's a concrete way to see how this adds up. Imagine an LLM generating the sentence "My favorite sport is basketball" one word at a time, at roughly 0.2 seconds per word. Five words at 0.2 seconds each comes out to about one full second just to produce that single short sentence. Scale that up to a multi-paragraph response, and the wait becomes very noticeable — even though each individual step is fast.

This is fundamentally different from, say, loading a webpage, where many resources can be fetched and rendered in parallel — the browser doesn't wait for the HTML to finish before it starts pulling images and stylesheets. Sequential generation can't take that shortcut: there's an inherent floor on how fast a response can appear, no matter how powerful the underlying hardware is.

Diagram — Sequential Generation vs. Parallel Loading

Autoregressive generation — one token waits for the last“My”step 1 · ~0.2s“favorite”step 2 · ~0.2s“sport”step 3 · ~0.2s“is”step 4 · ~0.2s“basketball”step 5 · ~0.2s5 words × ~0.2s each ≈ 1 full second for one short sentenceLoading a webpage — many pieces fetched at onceHTMLImagesStylesScriptsall start together — total wait ≈ the slowest one, not the sumToken generation can’tdo this — each wordneeds the one before it.

What can be done about it

The sequential nature of autoregressive generation is baked into how these models work, so solving latency means designing around it rather than eliminating it. AI developers and researchers are pursuing three main angles:

  • Exploring different architectures. Researchers are investigating model designs that reduce the strict step-by-step dependency, opening the door to generating output faster.
  • Parallel computing. Where possible, distributing parts of the workload so that more happens at once, rather than strictly one step after another.
  • Optimizing model size. This is often the most immediately practical lever: smaller models are typically faster to run than their larger, more complex counterparts. A team building a latency-sensitive product may deliberately choose a smaller model — accepting some trade-off in raw capability — in exchange for the snappier experience their users need.

None of these are silver bullets on their own. In practice, teams often combine several of these strategies — picking an appropriately-sized model and optimizing how it's deployed — to bring response times down to something users won't notice.

Key takeaway

Latency matters because users have low tolerance for delay, and that tolerance shrinks further in business and customer-facing contexts. The core challenge is architectural: today's LLMs generate text one token at a time, with each step depending on the one before it, which creates an inherent floor on response speed. Developers address this by exploring new architectures, leveraging parallel computing, and — most practically — choosing smaller, faster models when speed matters more than raw capability.

What's next?

Cost and speed are challenges of running AI models. The next lesson zooms out to a challenge that affects the entire field's future: what happens when AI developers start running out of fresh data to train on?