Measure retrieval before judging the answer

Learning objective

Evaluate retrieval separately from generation using precision@k, recall@k, MRR, and end-to-end accuracy.

Core idea

An incorrect answer can come from missing source material or from misusing source material that was supplied. Evaluate those stages separately before choosing a remedy. A more capable generator cannot make a missing exception reliably available, and a perfect first-ranked passage does not prove that the answer includes every required condition.

Use a labeled retrieval set to measure what the pipeline selects and a separate answer rubric to judge what Claude produces. Anthropic’s evaluation guidance calls for specific, measurable success criteria and representative tests. For retrieval, define the unit of relevance, cutoff, source version, and eligible corpus before reporting a number.

How it works

Establish relevance independently of generated answers

Anthropic’s RAG cookbook separates retrieval from end-to-end evaluation using records containing a question, relevant chunks, and a correct answer; some questions require synthesis across chunks. Add authenticated access scope and an effective date where needed. Reviewers should label those passages without treating the system’s generated response as ground truth. Otherwise, a plausible answer can persuade the reviewer to accept incomplete retrieval.

In information retrieval, query relevance judgments are often called qrels. Define whether your judgments apply to documents, chunks, or required supporting facts. Chunk IDs can change when a splitter changes; preserve source locators or rebuild judgments against the replacement units. Ten overlapping chunks containing the same paragraph are not ten independent pieces of answer support.

Include exact identifiers, paraphrases, multi-passage questions, historical versions, permission changes, and genuinely unanswerable questions. Keep the labels scoped to what the caller may access. If an answer requires a forbidden document, mark the intended outcome as unavailable or limited rather than rewarding a retrieval system for leaking it.

Compute precision@k and recall@k

Let R be the set of relevant eligible units for a query, and S_k the top k returned units. Use unique retrieval units rather than repeated copies of one hit. Under a fixed-cutoff convention, the measures are:

precision@k = number of relevant units in S_k / k
recall@k = number of relevant units in S_k / number of units in R

Precision asks how much of the ranked context is useful; recall asks how much relevant material was found. Anthropic’s cookbook divides precision by the number actually retrieved, whereas fixed-cutoff P@k divides by k. With three returned hits, two relevant, and k=5, cookbook precision is 2/3 while fixed-cutoff precision@5 is 2/5. Specify the convention: missing slots lower fixed-cutoff precision but do not enter the cookbook’s denominator.

Suppose four passages are relevant and the top five contain two of them. Precision@5 is 2/5, or 0.40; recall@5 is 2/4, or 0.50. Increasing the cutoff to ten and finding all four raises recall@10 to 1.00 while precision@10 remains 0.40. The generator receives more complete support but also more distracting text and a larger input.

For questions with no relevant eligible units, recall has a zero denominator. Evaluate correct no-answer behavior and irrelevant returns separately; do not invent a recall value or remove those questions from end-to-end accuracy.

Do not call “at least one relevant hit” recall when your rubric needs the fraction of all relevant units. The ir-measures docs distinguish that alternative as Success@k. Anthropic’s contextual retrieval article reports 1 − recall@20: the fraction of relevant documents missed within the top 20 chunks, using fraction-based recall. Compare its result with yours only when relevance units, corpus, and cutoff match.

Measure where the first useful result appears

Reciprocal rank is 1/r, where r is the rank of the first relevant result. Assign zero when no relevant result appears within the evaluated cutoff. Mean reciprocal rank (MRR) averages that value across queries. Anthropic’s cookbook defines MRR as the mean of reciprocal first-relevant ranks.

For three queries whose first relevant results appear at ranks 1, 4, and nowhere within the cutoff, MRR is (1 + 0.25 + 0) / 3, or approximately 0.417. This rewards finding a useful passage early. It is particularly informative when one source is sufficient or the interface exposes only the first few results.

MRR does not count how many other necessary passages are missing. A pipeline can achieve reciprocal rank 1 on every question while consistently omitting the exception required for the final answer. Use recall alongside MRR for questions that combine multiple clauses. State the cutoff so “not found” has an explicit evaluation boundary.

Judge generation and the complete system

Freeze a retrieved context and compare generators or prompts against an answer rubric. Then supply human-selected complete supporting context to test whether retrieval is the bottleneck. These diagnostic runs hold one stage steady; they are not replacements for production-shaped end-to-end tests.

Anthropic’s cookbook scores end-to-end accuracy as correct answers divided by total questions, using an LLM judge against the question and ground-truth answer. Its evaluation guidance documents LLM-based grading with explicit rubrics and reliability checks. Define correctness through the task’s rubric rather than accepting a judge’s unreviewed opinion. For a policy assistant, that can require the correct recommendation, applicable version, all material exceptions, and traceable support. Correct refusal to answer an unsupported question can count as success if the rubric defines it that way. Do not silently remove no-result cases from the denominator.

Microsoft’s RAG evaluator guidance distinguishes process evaluation of retrieval from system evaluation of responses. Groundedness and correctness are also distinct: an answer can faithfully repeat an outdated retrieved rule while failing the user’s effective-date requirement. Conversely, an answer based on general model knowledge can be factually right while failing a requirement to use approved supplied sources.

Citations support attribution, but do not certify answer completeness or correctness; see 2.2.4 for the search_result interface. Review the claim and supporting passage, not merely the citation marker.

Key distinctions and trade-offs

The following measures answer different diagnostic questions:

Keep test queries, relevance labels, and corpus releases versioned. Compare systems at the same cutoff and also at a controlled delivered-token budget. A higher recall obtained by sending much more text has a different latency and cost consequence from a better ranking at the same budget.

Report slices as well as averages. A corpus dominated by easy narrative questions can hide failures on identifiers or restricted-access queries. Count unauthorized retrieval as a separate failure condition rather than balancing it against improved relevance. The evaluation owner signs off on labels; the retrieval and generation owners receive stage-specific defects.

Architecture / decision scenario

A technology company evaluates a support assistant on 200 reviewed questions, including 60 that require both a troubleshooting instruction and a compatibility exception. Configuration A has MRR 0.92 but recall@5 of 0.58; configuration B has MRR 0.85 and recall@5 of 0.81. Both fit the same delivered-token budget. End-to-end accuracy is 72% for A and 84% for B.

The tempting choice is A because users see a useful first result sooner. The architect rejects that as the sole selection criterion: missing exceptions cause incorrect advice, and the release requirement is at least 82% end-to-end accuracy with no unauthorized passages. The evaluation owner confirms that relevance judgments include both required facts, rather than labeling only the main instruction.

The retrieval team selects B for a limited rollout and investigates identifier and exception slices. The generation team separately runs the same questions with complete human-selected context; failures that persist there require answer-stage fixes. Increasing k remains an experiment with measured token and latency costs, not the default remedy. Neither team can claim success from its own metric alone.

Common exam traps

• Calling one relevant hit complete recall.
• Using MRR to certify multi-passage coverage.
• Treating a grounded answer as necessarily correct for the user’s date and scope.
• Averaging away failures on identifiers, permissions, or unsupported questions.
• Changing the generator when the required passage never entered its context.

What to remember

Measure selection with precision@k, recall@k, and MRR, then judge the complete answer against a business rubric. Fix the stage demonstrated to fail. Keep relevance units, cutoffs, corpus versions, eligibility, and token budgets explicit so a better number represents a better system.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Retrieval Augmented Generation
• Anthropic, Introducing Contextual Retrieval
• Anthropic, Search results
• ir-measures, Measures
• Microsoft, Retrieval-Augmented Generation (RAG) evaluators