Why Chunk Size Is a Design Decision

The problem with one vector per document

The simplest possible design is to embed each document as a single vector. It's easy to build — and it works badly the moment a document covers more than one idea.

Imagine embedding an entire Wikipedia article as one vector. That article might discuss a film's plot, its production history, its box office performance, and its awards, all in one page. A single embedding has to compress all of that into one point in vector space — an average, in effect, of every topic on the page. A query about the film's awards and a query about its box office would both get pointed at the exact same, blurry vector, because the document-level embedding can't distinguish which part of the document is relevant to which question.

Chunking fixes this by splitting a document into smaller pieces before embedding, so each piece gets its own vector — and the search index becomes an index of chunk embeddings, not whole-document embeddings. A query about awards can now match specifically the award-related chunk, not a smeared average of the entire page.

There's no single right way to chunk

Once you accept that documents need to be split, a new question appears: split them how? A few broad strategies show up again and again:

• Sentence splitting — cut on sentence boundaries, so every chunk holds whole sentences and doesn't fracture meaning mid-thought.
• Fixed-length splitting — cut every N characters, regardless of what's in them. Simple, but can slice a sentence — or a word — in half.
• Token-based splitting — cut every N tokens instead of characters. More fine-grained than sentence splitting, but can still cut off mid-sentence.
• Semantic chunking — use the embeddings themselves to detect where the topic shifts, and chunk along those natural boundaries.
• Hierarchical chunking — split along a document's existing structure: chapters, sections, subsections.

Each strategy trades off implementation simplicity against how well it respects the actual meaning of the text. The next three lessons walk through the most common approaches in more depth — fixed-size and overlapping chunks, semantic chunking, and a newer approach that hands the decision to an LLM.

Key takeaway

Embedding a whole document as one vector blurs together every topic it covers. Chunking splits a document into smaller, more focused pieces first, so each piece's embedding actually represents what that piece is about — but how you split matters just as much as whether you split.

What's next?

Start with the most common and straightforward approach: splitting by a fixed size, with and without overlap between chunks.