Reranking: A Second Pass for Relevance

Retrieval is fast because it has to be

The first-stage search covered so far — dense retrieval, scanning a vector store — has to sift through a huge archive, possibly millions of documents, in a fraction of a second. To make that possible, it uses a relatively cheap comparison: embed everything once, then measure vector distance. That speed comes at a cost — it's not always the most accurate way to judge relevance.

A more careful second look

A reranker takes a much smaller job: given a query and a short list of already-retrieved candidates (not the whole archive), score how relevant each one actually is, and reorder the list accordingly. Because it only has to reason about a handful of candidates instead of millions of documents, it can afford to use a slower, more thorough model than the first-stage search could.

One popular way to build a reranker is a cross-encoder: instead of embedding the query and each document separately and comparing vectors afterward, a cross-encoder feeds the query and a candidate document into the model together, at the same time. That lets the model directly compare the two texts side by side before producing a single relevance score — a fundamentally more informed judgment than comparing two vectors that were each computed in isolation. All the candidates are still processed as one batch for efficiency, but each is scored against the query independently.

Seeing it reorder results

Picture three documents that first-stage retrieval already pulled back, in this order: Document #40, Document #68, Document #2. A reranker scores each one against the query and might return relevance scores of 20%, 15%, and 80% respectively — which flips the final order entirely, putting Document #2 first even though it was ranked last coming out of retrieval.

Why not just rerank everything from the start?

If a reranker is more accurate, why not skip retrieval and rerank the entire archive directly? Because cross-encoders are dramatically more expensive per comparison — scoring millions of documents against a query this way would be far too slow for an interactive search. The two-stage design exists precisely to get the best of both: retrieval's speed narrows the field, and reranking's accuracy sorts what's left.

Key takeaway

A reranker doesn't replace retrieval — it refines it. By scoring only the small set of candidates retrieval already found, often with a cross-encoder that reads the query and each candidate together, a reranker can afford far more careful judgment than a first-stage search ever could.

What's next?

So far, "better" or "worse" search results have been judged by eye. The final lesson covers how to actually measure search quality with numbers — precision, average precision, and mean average precision.