The RAG Pipeline, Step by Step

Two halves of one system

A RAG system has two distinct phases that run at completely different times, sharing one piece of infrastructure: the vector store.

Indexing — done offline, ahead of time

Before anyone asks a question, your documents need to be prepared:

1. Load the documents — pull in the raw source material, whatever form it's in.
2. Split into chunks — break long documents into smaller, more focused pieces (the subject of the next few lessons).
3. Embed each chunk — run every chunk through an embedding model to get its vector.
4. Store in a vector store — write those vectors, alongside the original text, into a database built for fast similarity search.

This step is expensive relative to a single question, but it only has to happen once per document — and again whenever your documents change.

Query — done every time, in real time

When a user actually asks something, a different, much faster flow kicks in:

5. Embed the query — using the same embedding model used for indexing, so the two vector spaces line up.
6. Search the vector store — find the chunks whose vectors are closest to the query vector.
7. Augment the prompt — insert the retrieved chunks into a prompt template alongside the original question, often with an instruction like "answer using only the context below."
8. Generate — the language model produces an answer grounded in the retrieved material rather than relying solely on what it memorized during training.

Seeing all eight steps in one example

Suppose a company builds an HR chatbot on top of its policy documents, and an employee eventually asks: "How many vacation days do new employees get?"

Indexing happens first, before that question ever exists. The system loads Employee_Handbook.pdf, PTO_Policy.docx, and a few wiki pages (step 1). It splits them into paragraph-sized chunks — one of which reads "New employees accrue 10 vacation days in their first calendar year. After two full years of employment, accrual increases to 15 days annually" (step 2). That chunk, along with every other chunk, gets embedded into a vector like [0.12, -0.44, 0.87, ...] (step 3), and the vector plus its original text is written into the vector store, giving it a fixed address in vector space (step 4). None of this knows or cares what anyone will eventually ask — it just makes the whole document set searchable.

Querying happens later, the moment the employee actually asks. Their question is embedded with that same embedding model (step 5), landing on the same map as the documents. The system searches the vector store and finds the vacation-days chunk sitting closest to the query — even though "How many vacation days do new employees get?" and "New employees accrue 10 vacation days..." barely share any words (step 6). That chunk gets inserted into a prompt template: "Using only the context below, answer the question. Context: '...' Question: How many vacation days do new employees get?" (step 7). The model reads that and generates: "New employees get 10 vacation days in their first year, increasing to 15 after two years of employment" — grounded in the retrieved policy text rather than anything memorized during training (step 8).

Why split it this way

Separating indexing from querying is what makes RAG practical at scale. Re-embedding your entire document set on every single question would be far too slow and expensive. Instead, the expensive work happens once, up front, and each question only pays the (much smaller) cost of embedding one query and searching an already-built index.

Key takeaway

RAG isn't one model — it's a small pipeline with an offline indexing half (load, split, embed, store) and an online query half (embed, retrieve, augment, generate) that share a single vector store.

What's next?

Step 1 of indexing — "load the documents" — sounds simple, but real documents are messy. The next lesson covers how documents actually get parsed and prepared before they're ever embedded.