Separate retrieval failures from answer failures

Learning objective

Measure RAG quality: faithfulness, answer relevancy, context precision, and context recall.

Core idea

A retrieval-augmented generation (RAG) answer can fail because the retriever misses necessary material, ranks distractions first, or supplies sufficient information that the generator misuses. One overall quality score cannot distinguish those causes. Measure the retrieved context and the generated answer separately, keeping their different reference requirements explicit.

Faithfulness asks whether the answer’s claims follow from the retrieved material. Answer relevancy asks whether the answer addresses the question. Context precision assesses the ordering of useful retrieved chunks. Context recall asks whether the retrieved material covers the information needed for a reference answer. A high value for one does not imply high values for the others.

How it works

Preserve the evaluation inputs

Capture the question, the answer actually shown to the user, and the ordered chunks actually supplied to generation. In the Ragas metric interfaces, these are user_input, response, and retrieved_contexts. Add reference when the selected metric needs an expected answer. Do not score against a larger candidate pool that the generator never received: that measures a different context from the one responsible for its output.

The Ragas pages show collections-based metric classes under ragas.metrics.collections, with ascore(...) returning a result whose value is the score. Use the input names and class from the API you installed. Keep collections and legacy invocation styles separate when implementing the benchmark.

Measure answer support and relevance

Faithfulness decomposes response into claims and checks which claims the retrieved_contexts support. The score is supported response claims divided by total response claims. If three of four claims follow from the context, the illustrative score is 0.75. The fourth claim remains unsupported even when it happens to be true elsewhere.

That denominator matters. A response can omit half the requested information while making only supported claims. It can therefore be faithful yet incomplete. Faithfulness also cannot establish that the underlying document is authoritative or up to date: it measures consistency with the supplied context. Source validity and task correctness require their own checks.

Ragas documents answer relevancy through AnswerRelevancy. It generates questions from response, embeds them, and compares them with the embedding of user_input; the score averages their cosine similarities. It needs a judge LLM and embeddings, but no reference answer. The documented default is three generated questions, controlled by strictness. Scores usually lie between zero and one, but the page notes that cosine similarity does not guarantee that interval.

An answer that confidently repeats the question’s subject can be relevant and wrong. An accurate explanation of a different procedure can be faithful to the documents but irrelevant to the question. Interpret these metrics together rather than turning relevance into a factuality test.

Measure retrieval order and coverage

The LLM-judged collections ContextPrecision interface compares retrieved chunks with reference to assess usefulness. Ragas computes a rank-sensitive score: sum precision at each relevant position, then divide by the number of relevant retrieved chunks. With relevance indicators [0, 1, 1], the illustrative result is (1/2 + 2/3) / 2, approximately 0.583. Moving the irrelevant chunk to the end gives [1, 1, 0] and a score of 1.

This rank-sensitive score rewards placing useful material early; its denominator excludes missing reference information. The ContextUtilization variant compares chunks against the generated response instead of a reference. That changes the measurement target; an incomplete response can be a poor proxy for everything retrieval should have found.

ContextRecall uses the reference answer’s claims as its denominator. It measures the share supported by retrieved_contexts. If the reference has four required claims and retrieval supports two, recall is 0.5 even when every retrieved chunk is relevant. Context recall always requires a reference to compare against.

The pages also document alternatives under ragas.metrics that avoid judge calls. NonLLMContextPrecisionWithReference compares retrieved_contexts against labeled reference_contexts using string similarity and requires rapidfuzz; NonLLMContextRecall also compares retrieved and reference context strings. IDBasedContextPrecision and IDBasedContextRecall instead compare retrieved_context_ids with labeled reference_context_ids. ID-based precision divides matching retrieved IDs by total retrieved IDs and ignores rank; ID-based recall divides matching reference IDs by total reference IDs. These deterministic alternatives trade judge cost for labeled contexts or IDs and measure matching rather than semantic claim support.

Connect the metrics to NVIDIA evaluation

NeMo Agent Toolkit exposes a Ragas integration through evaluator _type: ragas, with metric naming the metric and llm_name selecting its judge. NVIDIA’s documented examples use AnswerAccuracy, ContextRelevance, and ResponseGroundedness. They respectively assess agreement with an expected answer, retrieved-context relevance to the question, and answer support in context.

For these three NV metrics, NVIDIA recommends max_tokens: 8 for the judge, which returns a score between zero and one. Their prompts are fixed; use tunable_rag_evaluator when the rubric needs a custom prompt. This short score output contrasts with the claim decomposition used by Faithfulness and ContextRecall.

The toolkit’s metric can be a string or a dictionary whose kwargs pass to the metric function. NVIDIA illustrates the dictionary form with FactualCorrectness, mode: precision, and a separate judge configuration requiring more tokens. These interfaces do not make the metric definitions interchangeable: ContextRelevance establishes neither reference-claim recall nor rank-sensitive precision.

Key distinctions and trade-offs

Use this comparison to decide which signal tests your suspected failure:

Keep metric results per case. The reference data and judge calibration from 3.1.1 and 3.1.2 determine whether these signals are dependable enough to guide a decision.

Architecture / decision scenario

An industrial maintenance assistant must answer a four-part shutdown question within six seconds. Its retrieved chunks support two required steps, and two distracting chunks precede the useful material. The answer repeats the question’s topic and states those two steps, plus an unsupported instruction to bypass an interlock. These are hypothetical task conditions, not equipment guidance.

The operations engineer proposes doubling retrieved context from four to eight chunks. The evaluation owner first separates the failures: recall is 2/4, so necessary information is missing; precision is reduced by the irrelevant leading chunks; faithfulness is 2/3 if the response has exactly those three claims; answer relevancy can remain high because the answer stays on topic.

The retrieval owner investigates missing procedures and ranking, while the generation owner investigates the unsupported instruction. Doubling context may improve recall but adds input tokens and threatens the six-second budget; it does not enforce faithfulness. The team requires all four signals and human review of the dangerous instruction before accepting a retrieval change. A high relevancy score alone cannot approve the result.

Common exam traps

• Equating a faithful answer with a complete or factually authoritative answer.
• Using answer relevancy as a correctness score.
• Confusing rank-sensitive ContextPrecision with the unordered ratio used by IDBasedContextPrecision.
• Claiming reference-based context recall without reference information.
• Treating NVIDIA’s ContextRelevance as a synonym for every retrieval metric.

Where the exam guide differs

What to remember

Faithfulness measures response support; relevancy measures alignment with the question. Context precision measures ranking, while context recall measures coverage against required information. Preserve the actual generation context and a valid reference so that each score answers the question you intend.

Knowledge check

References

• Ragas, Faithfulness
• Ragas, Response Relevancy
• Ragas, Context Precision
• Ragas, Context Recall
• NVIDIA, Agent Evaluation in NVIDIA NeMo Agent Toolkit
• NVIDIA, NVIDIA-Certified Professional: Agentic AI Exam Study Guide