Separate retrieval scores from answer quality

Learning objective

Evaluate retrieval relevance, groundedness, and ranking metrics (such as MRR), separating retrieval quality from generated-answer quality.

Core idea

A retrieval-augmented generation (RAG) system makes two decisions: which evidence reaches the model and what the model says using that evidence. Evaluate them separately. A relevant document at rank one cannot prevent fabrication, and a perfectly grounded answer can faithfully repeat an irrelevant passage.

For architects comparing retrieval configurations, preserve the ranked results, the context actually supplied to the model, and the final response for each query. Those are three distinct artifacts. If truncation or context assembly removes the best passage, evaluating the original ranked list alone misses the failure that generation encounters.

Microsoft Foundry distinguishes process evaluators for retrieval from system evaluators for the generated answer. Use this separation to choose the right repair: search configuration, context assembly, or generation behavior. See 3.1.2 for cloud-run configuration and 3.1.3 for response regression comparisons.

How it works

Evaluate relevance before generation

When you have human relevance judgments, use builtin.document_retrieval. It maps retrieval_ground_truth and retrieved_documents, without requiring a judge deployment_name. The ground-truth records carry document_id and query_relevance_label; retrieved records carry document_id and relevance_score. Keep identifiers aligned with the unit being judged, especially when the retriever returns chunks rather than complete documents.

Microsoft’s example uses ground_truth_label_min=1 and ground_truth_label_max=5, while documenting SDK defaults of 0 and 4. Set the range to match your labels instead of treating those example values as a universal grading scale. A search engine score and a human relevance label serve different purposes: one ranks candidates, while the other defines the evaluation target.

Without ground truth, builtin.retrieval judges the relevance of context to query using a model deployment identified by deployment_name. This supplies a model-based relevance signal, with a 1–5 score converted to pass/fail by a threshold. It does not reconstruct missing document-level relevance judgments or establish an ideal ordering.

The context field is a plain string. Microsoft documents concatenating multiple chunks with a separator such as two newline characters. Save the exact assembled context as well as the ranked source records so you can detect omitted passages and changes in ordering. Do not substitute a reconstructed context that the tested model never received.

Choose ranking metrics by the question

Microsoft’s Architecture Center defines mean reciprocal rank (MRR) as the average reciprocal rank of the first relevant result. For positive queries with known relevant items, higher MRR is better. State your relevance criterion and cutoff; assign zero when a positive query retrieves no relevant result within that cutoff.

Keep negative queries whose answers are absent from the corpus. Microsoft recommends testing both groups, aiming for metrics near 1 on positive cases and near 0 on negative cases, and reporting their averages separately. Negative cases test whether retrieval surfaces unsupported evidence; they must not disappear because a library cannot compute their MRR.

For an illustrative three-positive-query test, the first relevant results appear at ranks 1 and 4, and the third query has no relevant result within the cutoff. The reciprocal ranks are 1, 0.25, and 0, so MRR is approximately 0.417. Moving the rank-four result to rank two raises the average to 0.5. These are example calculations, not Foundry acceptance thresholds.

MRR rewards finding one useful result early. Use Recall at K to measure coverage: relevant items in the top K divided by all relevant items. Foundry’s Fidelity provides the evaluator-side analogue. MRR does not establish that coverage. If one question needs both eligibility rules and an exception, placing the eligibility passage first can yield an excellent reciprocal rank while missing the exception entirely.

Normalized discounted cumulative gain (NDCG) evaluates ordering relative to an ideal ranking, supporting graded relevance judgments. Foundry’s document-retrieval evaluator also reports Fidelity, XDCG, Max Relevance N, and Holes. Fidelity measures good documents returned relative to known good documents; XDCG evaluates top-k result quality; Max Relevance N captures the maximum relevance among top-k chunks. Holes counts documents without ground-truth relevance judgments. A hole is a labeling gap, not proof that the document is irrelevant.

The documented output includes metric identifiers such as ndcg@3 and fidelity. Foundry’s documented metric list does not include MRR. Compute MRR separately when your acceptance contract requires it. One implementation option is Microsoft’s rankerEval, whose documentation shows ReciprocalRank(3), .score(), and .mean(). With no positive ground-truth labels, it returns NaN; .mean(nan_handling=...) accepts propagate (default), drop, or zerofill. Declare that handling policy and keep the negative-query report separate; replacing NaN with zero does not demonstrate correct negative-query behavior. Do not relabel NDCG as MRR because both concern ranking.

Evaluate the answer against its own evidence

Use builtin.groundedness to assess whether generated claims align with the supplied context. For a controlled RAG comparison, map the actual response and context, with deployment_name for the model judge. Microsoft also documents agent response modes, but that broader interface is not a reason to omit known context from a retrieval experiment.

Groundedness addresses the precision of response claims: the answer should not add unsupported content. It does not prove that the answer addresses the user’s question or includes all required facts. Use builtin.relevance with query, response, and deployment_name for answer relevance. A short response can be grounded and still fail the task by omitting a critical exception.

When you need completeness against an expected answer, the Response Completeness evaluator is preview and compares response with ground_truth. Groundedness Pro is also preview; it uses Azure AI Content Safety, takes query, response, and context, and returns a binary result without requiring a model deployment. Keep these status and interface distinctions explicit if you choose either option.

Compare one stage at a time

Run retrieval candidates against the same queries and relevance judgments before changing the generator. Microsoft documents a parameter sweep across search algorithms, top-k values, and chunk sizes using document-retrieval metrics. Save the resulting context for each candidate so answer evaluation can explain whether a retrieval improvement survives context assembly.

Then hold context fixed while comparing generation candidates. If groundedness changes with identical evidence, inspect the generator and its instructions. If answers fail only when the evidence changes, inspect retrieval and assembly first. This is an experimental design choice, not an automatic causal guarantee from a score.

Each score isolates one stage because it compares a different pair of artifacts.

Key distinctions and trade-offs

Retrieval relevance versus answer relevance: builtin.retrieval compares query and context. builtin.relevance compares query and response. Their names do not make their input contracts interchangeable.

MRR versus NDCG: MRR emphasizes the first relevant hit. NDCG assesses the ranked list against an ideal ordering. Use Recall at K or Fidelity when the task requires multiple facts.

Groundedness versus completeness: An answer can make only supported claims and still omit required information. Judge both when omission changes the user’s decision.

Human labels cost review effort but enable repeatable ranking comparisons. Model-based retrieval judgments work without those labels, but add judge variability and do not provide the same rank-level evidence. Match the method to the question and preserve raw scores alongside thresholded results.

Architecture / decision scenario

A benefits assistant must explain eligibility and an exception. Candidate A retrieves the eligibility passage first but omits the exception. Candidate B retrieves both passages, with the first relevant hit lower in the list. The team computes MRR at the same cutoff and reviews Recall at K plus builtin.document_retrieval Fidelity and NDCG against labels for both required passages.

Candidate A’s strong MRR cannot settle the choice because the answer requires both facts. The team confirms that candidate B’s assembled context retains the exception. It then evaluates generated responses with builtin.groundedness and builtin.relevance, adding the preview completeness evaluator only under an explicit evaluation decision.

If B retrieves both passages but the response invents an eligibility condition, the repair belongs to generation. If the exception disappears during context assembly, the repair belongs between retrieval and generation. If the ranked results never contain it, search needs attention. Separate artifacts and scores identify which experiment to run next.

Common exam traps

• Treating a rank-one relevant hit as proof of complete evidence coverage.
• Dropping negative queries because a library returns NaN for their MRR.
• Confusing model-based Retrieval with labeled Document Retrieval.
• Feeding search scores into human-label fields or using the wrong label range.
• Calling an unlabeled result irrelevant instead of recording a Hole.
• Expecting MRR among Foundry’s documented document-retrieval outputs.
• Assuming a grounded response is relevant, complete, or independently true.

What to remember

Score ranked evidence, assembled context, and generated answers independently. MRR measures the first relevant hit; NDCG measures ranking quality; groundedness measures support for response claims. Preserve enough evidence to decide whether search, assembly, or generation owns the failure.

Knowledge check

References

• Microsoft, Retrieval-augmented generation evaluators
• Microsoft, Develop a RAG solution: Information-retrieval phase
• Microsoft, rankerEval metrics
• Microsoft, rankerEval reciprocal-rank implementation