AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Measure retrieval before judging generated answers

Completed by the knowledge check

Learning objective

Evaluate retrieval relevance, context matching, and latency independently from generated-answer quality. You need separate measurements to determine whether a retrieval change improves the evidence supplied to the generator.

Core idea

Retrieval succeeds when it supplies the right authorized evidence within the required time. Generation succeeds when it uses that evidence to produce an acceptable answer. These outcomes are related but not interchangeable: a model can answer from prior knowledge despite inadequate retrieval, or produce a wrong answer despite receiving sufficient evidence.

Amazon Bedrock RAG evaluations distinguish Retrieve only from Retrieve and generate. Retrieve-only evaluation isolates retrieved material and supplies Builtin.ContextRelevance and Builtin.ContextCoverage. Latency needs its own measurement, and exact source matching needs its own contract. None of these is a substitute for testing generated answers.

See 1.5.6 for retrieval interfaces, 1.5.11 for context assembly, and 4.2.4 for retrieval optimization. Here, the decision is whether the evidence itself passes.

How it works

Choose the evaluation boundary

A retrieve-only Bedrock evaluation can invoke an Amazon Bedrock Knowledge Base or score retrieval output that you supply from another RAG source. For a managed knowledge-base run, CreateEvaluationJob uses applicationType: RagEvaluation and inferenceConfig.ragConfigs[].knowledgeBaseConfig.retrieveConfig. That configuration names knowledgeBaseId and knowledgeBaseRetrievalConfiguration; the latter can specify vectorSearchConfiguration.numberOfResults.

The metric selection belongs in evaluationConfig.automated.datasetMetricConfigs[].metricNames. Select Builtin.ContextRelevance to assess how relevant the retrieved texts are to the question. Select Builtin.ContextCoverage to assess whether those texts cover the information in the ground truth. Both use an evaluator model rather than a binary source-identity comparison.

For output from a custom retrieval or context-assembly pipeline, capture the actual material at the boundary you want to assess and use the bring-your-own response-data path. Record whether the evaluated text is raw search output, reranked output, or the final context admitted to the generator. A high score for raw search results says little about evidence that later truncation removes.

Distinguish the dataset fields

The retrieve-only dataset is JSON Lines, with up to 1,000 prompts per dataset. Each record uses conversationTurns containing the prompt. When you select Context coverage, supply referenceResponses: this is the ground-truth answer an end-to-end RAG system should return, not the expected retrieved chunks. Bedrock judges whether the retrieved texts contain the information needed for that answer.

For supplied retrieval output, output.knowledgeBaseIdentifier identifies the RAG source and output.retrievedResults holds the results. The optional referenceContexts field represents expected ground-truth passages for your custom evaluation metrics. AWS explicitly states that the built-in metrics do not use referenceContexts. Providing expected passages there does not make Context coverage an exact passage-recall test.

Keep representative answerable and unanswerable questions, multi-document questions, ambiguous versions, and permission-filtered cases. Define a successful empty result when no authorized evidence exists. An evaluation that rewards finding any plausible text for every query can favor unauthorized or obsolete context over correct abstention.

Measure context matching explicitly

Context matching can mean semantic relevance, information sufficiency, or retrieving a specific source revision. State which meaning your gate uses. Context relevance and coverage address the first two. For the third, maintain reviewed expected document identifiers, source revisions, and required evidence spans in an application-owned test record.

The Retrieve response exposes retrievalResults, including content, location, metadata, and score. Compare the returned source identity and metadata with the expected record before generation. If your corpus stores its revision identifier in metadata, check it explicitly. Do not infer permission or freshness from the retrieval score.

For deterministic checks, your evaluation harness can count expected evidence items found among the returned results and detect forbidden document identifiers. Treat these as application-defined metrics with documented denominators. If you need a semantic custom metric, referenceContexts can supply its reference passages in the supported dataset path. Neither approach changes what Bedrock’s built-in coverage metric means.

For multi-document questions, measure whether all required evidence is present. Ten individually relevant passages can omit the one exception that decides the answer. Also inspect the final context after deduplication and token-budget selection: retrieval recall and admitted-context sufficiency can diverge.

Time the retrieval path separately

Measure elapsed time around the actual Retrieve request or your custom retrieval boundary. Decide whether that interval includes query preparation, network calls, reranking, context assembly, and retries. Record the boundary and configuration alongside every observation. Keep generation time and evaluator runtime outside the retrieval-latency measure.

Publish application-defined measurements such as RetrievalLatencyMs through CloudWatch PutMetricData, with Namespace, MetricData, MetricName, Value, Unit: Milliseconds, and bounded Dimensions identifying the configuration. This is your custom metric, not a documented Bedrock retrieve-only quality metric. Publish nonnegative raw latency samples and inspect a CloudWatch percentile statistic such as p99 on RetrievalLatencyMs under representative load. Inspect failures separately rather than reporting only the average for successful requests.

Key distinctions and trade-offs

Relevance versus coverage: Relevance asks whether returned text relates to the question. Coverage asks whether it contains the information in the ground-truth answer. Many relevant results can still have inadequate coverage.

Coverage versus exact context matching: referenceResponses supports information coverage. referenceContexts supports custom passage-based checks; built-in metrics ignore that field. Neither implies matching your required source revision without an explicit check.

Retrieval latency versus answer latency: A faster retrieval stage can still increase total response time if it supplies excessive context. Evaluate both boundaries, with different measurements.

Retrieval quality versus faithfulness: Builtin.Faithfulness in a retrieve-and-generate job assesses hallucination relative to retrieved text. A faithful answer can repeat an obsolete source. The source-quality gate and the answer-quality gate remain separate.

Architecture / decision scenario

A policy assistant tests a retrieval change that returns more passages. The team runs retrieve-only evaluation against questions whose referenceResponses include exceptions and effective dates. Context relevance stays strong, but Context coverage falls on cross-document questions.

The harness also checks retrievalResults.metadata against expected policy revisions and finds a superseded document. Separately, its timed retrieval requests show worse tail latency. A generated-answer evaluation nevertheless looks acceptable because the model often supplies a plausible answer from its prior knowledge.

The team rejects the retrieval change. It fixes source selection and evidence coverage, reruns the same retrieval tests, and only then tests answer quality. For unanswerable or unauthorized cases, an empty evidence set remains a valid outcome; the generator must handle it according to the application’s abstention policy rather than fabricate missing evidence.

Common exam traps

Using generated-answer correctness as the only retrieval test.
Treating relevance as proof that all required evidence is present.
Putting expected chunks in referenceResponses instead of ground-truth answers.
Assuming built-in metrics read referenceContexts.
Treating retrieval score as an authorization or freshness decision.
Reporting evaluator runtime as retrieval latency.

What to remember

Test evidence at an explicitly named boundary. Use Context relevance and Context coverage for their documented purposes, custom checks for exact source matching, and timed requests for latency. Require authorized, sufficient, timely context before drawing conclusions from generated-answer quality.