← Course Home

Chunking Strategies

Semantic Chunking

3 min read

Chunking by topic, not by count

Fixed-size chunking treats every document the same way regardless of content — cut every N units, no matter what's actually being said. Semantic chunking takes a different approach: use the embeddings themselves to detect where the topic shifts, and put the chunk boundary there instead.

Picture a mixed-topic newsletter that jumps between politics, sports, and healthcare within the same document. Splitting it by a fixed character count is as likely to land a boundary in the middle of the politics section as at the edge of it. A chunker that's aware of meaning, on the other hand, can notice the shift from "politics" to "sports" and cut precisely there — producing chunks that actually correspond to one coherent idea each, which is exactly what makes them easier to retrieve correctly later.

The algorithm, step by step

Diagram — How a Semantic Chunker Finds Breakpoints

1Break intosentences2Embed eachsentence3Measure similarity between consecutive sentencesbreakpointdistance between pairs →4Merge sentences up to the breakpoint intoone chunk, then start the next chunk thereResult: chunks that hold together by meaning, not by a fixed word count —a topic shift becomes the natural chunk boundary.
  1. Break the text into small pieces — typically individual sentences.
  2. Generate embeddings for each small piece using an embedding model.
  3. Measure the similarity between each pair of consecutive pieces — how close is sentence 2 to sentence 1, sentence 3 to sentence 2, and so on.
  4. Merge and define breakpoints — as long as consecutive pieces stay similar, keep merging them into the same growing chunk. The moment similarity drops sharply — a spike in the distance between neighbors — that's treated as a breakpoint: end the current chunk there and start a new one.

That threshold for "sharp enough to count as a breakpoint" is itself a tunable setting — common approaches set it using a percentile, a standard deviation, or a gradient over the similarity scores, rather than a single fixed number.

The tradeoff

Semantic chunking produces more topically coherent chunks than a fixed-size cutoff, which tends to improve retrieval quality. The cost is computational: every sentence needs its own embedding before you even know where the chunk boundaries will be, which is more expensive than simply counting characters or tokens.

Key takeaway

Semantic chunking embeds individual sentences, measures how similar each one is to its neighbor, and draws chunk boundaries wherever that similarity drops sharply — producing chunks that track actual topic shifts instead of an arbitrary character count.

What's next?

Semantic chunking still relies purely on embedding similarity to find boundaries. The next lesson covers an even more recent approach that hands the reasoning about what belongs together to a language model itself.