← Course Home

Ranking, Storing, and Measuring Results

Evaluating Search Quality with MAP

4 min read

You can't improve what you can't measure

Every technique in this course — chunking strategy, distance metric, reranking — is a lever that changes search results. But "changes" isn't the same as "improves." Without a way to score search quality, there's no way to know whether a given change actually made things better, or just different.

Semantic search borrows its evaluation approach from the broader field of information retrieval (IR). Scoring a system needs three ingredients: a text archive to search, a set of queries to run against it, and relevance judgments — a ground-truth record of which documents in the archive actually are relevant to each query.

Precision at k

Given a query, a system returns a ranked list of results. Precision at position k asks: of the top k results, what fraction are actually relevant? If the very first result is relevant, precision at 1 is 1.0 (1 relevant result out of the 1 you looked at). If the second result isn't relevant, precision at 2 drops to 0.5 (1 relevant result out of the 2 you looked at now).

Why simple precision isn't enough

Consider two search systems answering the same query, where the test suite says there's exactly one relevant document for it:

  • System 1 returns: relevant, not relevant, not relevant.
  • System 2 returns: not relevant, not relevant, relevant.

Both systems found the one relevant result somewhere in their top 3 — but System 1 put it first, and System 2 buried it in third place. Intuitively, System 1 did the better job. A metric that only checked "did a relevant result appear in the top 3 at all?" would score them identically and miss that difference entirely.

Average precision

Average precision (AP) fixes this by rewarding a system for ranking relevant results higher. For a single query, it's computed by looking at precision at each position where a relevant result actually appears, then averaging those precision values.

Diagram — Computing Average Precision for One Query

System 12 relevant docs exist for this query1relevantprecision@1 = 1.002not relevantprecision@2 = 0.503relevantprecision@3 = 0.67Average precision = (1.0 + 0.67) / 2 = 0.83System 22 relevant docs exist for this query1not relevantprecision@1 = 0.002not relevantprecision@2 = 0.003relevantprecision@3 = 0.33Average precision = (0 + 0.33) / 2 = 0.17Same number of relevant results in the top 3 for both systems — but System 1 ranked itsrelevant result higher, so it earns a higher score. Averaging AP across every query in atest suite gives Mean Average Precision (MAP) — one number to compare search systems by.

Walking through a slightly richer example, where a query has two relevant documents in the test suite:

  • A system's results are: relevant, not relevant, relevant.
  • Precision at position 1 (where the first relevant result lands): 1 relevant found / 1 result looked at = 1.0.
  • Precision at position 3 (where the second relevant result lands): 2 relevant found / 3 results looked at = 0.67.
  • Average precision = (1.0 + 0.67) / 2 = 0.83.

A system that instead buried both relevant results near the bottom of the same three results would score much lower — even with the identical count of relevant results retrieved — because AP is sensitive to where they land, not just whether they're present.

Mean average precision (MAP)

A single query's AP score only tells you about that one query. Mean average precision (MAP) averages the AP score across every query in a test suite, producing one overall number that summarizes how well a search system performs across the board — the standard metric for comparing one search system against another.

Key takeaway

Precision at k measures accuracy among the top k results; average precision rewards ranking relevant results higher rather than merely including them; mean average precision averages AP across an entire test suite into the one number teams actually use to compare search systems.

Where this leaves you

You've now walked the full path this course set out to cover: why keyword search breaks down, how embeddings and dense retrieval solve it, the complete RAG pipeline from raw documents to a generated answer, the tradeoffs between chunking strategies, the math behind "similar," the landscape of vector databases, how reranking sharpens results, and finally how to measure whether any of it actually worked. From here, the natural next step is applying it — building a small RAG system over a real, familiar set of documents, and watching precision and MAP move as you tune chunk size, distance metric, and reranking.