Locate answer defects across retrieval boundaries

Learning objective

Separate retrieval errors from generation errors using source relevance analysis, embedding diagnostics, and drift checks.

Core idea

A wrong retrieval-augmented generation (RAG) answer can be faithful to the wrong source, unsupported by a relevant source, or based on evidence that disappeared during context assembly. Diagnose the evidence path before changing either the embedding model or the generator.

Break the investigation into source availability, retrieval, assembly, and generation. At each boundary, preserve the actual input and output for the same question. A similarity score alone cannot assign responsibility. You need reviewed evidence showing whether the returned passages contain the facts required to answer and whether those facts reached the model intact.

How it works

Inspect sources without generating an answer

Use Amazon Bedrock Knowledge Bases Retrieve to inspect evidence independently of generation. Supply knowledgeBaseId and retrievalQuery.text, and preserve the actual retrievalConfiguration used by the failing path. Its vector search configuration can include numberOfResults, filter, and overrideSearchType; investigate the configured values before altering them. Search-strategy support is covered in 1.5.7.

The response’s retrievalResults contains chunk content and can include location, metadata, and score. AWS defines score as relevance to the query. It is not the probability that a generated answer is true. Read the content and its source identity, then label whether it directly supports the required fact, provides only related background, contradicts the authoritative version, or is irrelevant.

Keep authorization filters fixed during the investigation. If the expected document is unavailable to the caller, removing the filter might improve an experimental relevance score while violating the application contract. See 1.4.6. Diagnose separately whether an authorized source is missing, a metadata value is wrong, or the retrieval configuration excludes it.

Then compare returned evidence with assembled context. Relevant passages can be returned correctly but dropped, duplicated, or cut during packing. That is an assembly failure, not evidence that the vector index is ineffective. Use the context-loss checks from 5.2.1 before attributing the answer to generation.

The defect belongs to the first boundary where the required facts go missing, and generation is suspected only when they reached the model intact.

Test relevance and coverage separately

For a representative question set, annotate the authoritative passage and required facts. Ask two different questions: “Are the retrieved passages relevant?” and “Do they contain enough information to answer?” A collection of relevant introductions can still omit the decisive exception.

Amazon Bedrock retrieve-only RAG evaluation offers Builtin.ContextRelevance and Builtin.ContextCoverage. The coverage metric requires ground truth in the prompt dataset. Retrieve-and-generate evaluation offers answer metrics including Builtin.Correctness, Builtin.Completeness, and Builtin.Faithfulness. These names provide concrete measurement boundaries; evaluation setup is covered in 5.1.2.

Interpret disagreement diagnostically. Low context coverage points to absent evidence even if the answer sounds fluent. Strong coverage with poor faithfulness points toward the generator or the evidence-to-prompt boundary. High faithfulness with poor correctness can mean the model reproduced stale or incorrect source content. Review the actual passages and annotations rather than assigning a cause from one aggregate score.

A useful controlled test replaces retrieved context with a reviewed, authorized supporting passage while keeping the question and generator configuration fixed. If the answer becomes acceptable repeatedly, investigate retrieval or assembly. If it remains wrong with complete evidence, investigate instructions, generation settings, and response processing. This substitution is an application experiment, not a built-in automatic Bedrock diagnosis.

Inspect the embedding contract

For a custom embedding pipeline, compare query and document model identity, preprocessing, vector dimensions, normalization, representation, and index similarity assumptions. Matching dimensions is necessary but insufficient: two models can produce equal-length vectors with different semantic coordinates. Do not mix their vectors merely because the index accepts the shape.

For Amazon Titan Text Embeddings V2, the model identifier is amazon.titan-embed-text-v2:0. Its native request uses inputText; dimensions accepts 1024, 512, or 256, with 1024 as the default. normalize defaults to true, and embeddingTypes selects float, binary, or both representations. The response includes inputTextTokenCount and embeddingsByType. Use these concrete fields to detect configuration mismatch or unexpectedly enlarged input. See 1.5.4 for compatible publication.

Titan Text Embeddings V2 accepts non-empty input up to 8,192 tokens or 50,000 characters. Check both limits. The embedding model produces vectors; your vector database performs similarity computation and retrieval. Generation parameters such as topP do not repair an embedding mismatch, and Titan embedding models do not support those inference parameters.

Probe a few known question-passage pairs and clear negatives through the same preprocessing path used in production. Inspect whether a pipeline strips an identifier, loses table headings, or embeds boilerplate instead of substantive content. If vectors are malformed or incompatible, repair the contract before tuning relevance thresholds. A threshold can suppress defective results without making the missing evidence searchable.

Check drift against frozen controls

Freeze a baseline with source revisions, embedding configuration, query cases, retrieval settings, and generator configuration. Rerun the frozen suite as a retrieve-only evaluation job against the accepted and changed source snapshots. Compare Builtin.ContextRelevance and Builtin.ContextCoverage by category, with ground truth supplied for coverage. Record which expected passages changed rank or disappeared, and inspect source freshness before concluding that the model degraded.

For Knowledge Bases ingestion, GetIngestionJob exposes an ingestionJob with status, failureReasons, and statistics, including numberOfDocumentsFailed. Inspect this evidence when a source update stops appearing in results. A source file existing in storage does not prove its updated text is available through retrieval. Synchronization and reconciliation belong to 1.4.11.

Separate corpus drift from traffic drift. Your fixed suite can pass while production requests move into unfamiliar terminology or topics. Add representative cases to the evaluation job’s prompt dataset (see 5.1.2), retaining the frozen cases. Evaluate the expanded dataset against both snapshots and compare Builtin.ContextRelevance and Builtin.ContextCoverage by category to measure the coverage gap. Keep generation fixed while checking retrieval, then keep reviewed context fixed while comparing generator versions.

Key distinctions and trade-offs

Relevance versus entailment: A passage about cancellation is relevant but might not contain the clause needed for an eligibility decision. Read for required evidence, not topical overlap alone.

Embedding validity versus retrieval utility: A vector with the expected shape can still represent the wrong text or an incompatible model. Contract validation and labeled retrieval tests answer different questions.

Source freshness versus model quality: A faithful answer from an obsolete policy is a source-quality defect. Changing temperature cannot publish the missing policy revision.

Broader retrieval versus usable context: More results can improve coverage but add noise and exceed the assembly budget. Measure both returned evidence and evidence actually supplied to generation.

Architecture / decision scenario

An engineering assistant gives obsolete maintenance instructions after a policy update. The team suspects that the generation model has drifted.

A Retrieve replay shows the previous policy revision in metadata and related passages with strong relevance scores. GetIngestionJob shows failed documents in the update run. The engineer repairs ingestion and verifies that the authorized revised passage is retrievable; changing the generator would not fix this source defect.

A second affected question still misses a product-specific exception. Query-embedding diagnostics reveal that a custom preprocessing path strips the product qualifier from inputText. The team restores the approved input transformation, verifies the vector configuration, and checks known positive and negative pairs. Fixed queries recover coverage against the revised corpus.

For a third question, retrieved and assembled evidence are complete, but the generator invents an extra instruction even when supplied with the reviewed passage directly. That failure moves to the controlled prompt investigation in 5.2.3. The release decision requires source relevance, coverage, and answer faithfulness to pass their separate gates.

Common exam traps

• Treating a retrieval score as answer confidence or factual proof.
• Tuning the generator before checking whether required evidence was returned.
• Assuming equal vector dimensions establish embedding compatibility.
• Applying text-generation sampling parameters to Titan embedding diagnosis.
• Treating completed source upload as proof of successful retrieval ingestion.
• Blaming retrieval when correct passages were removed during context assembly.

What to remember

• Inspect Retrieve output separately from generated answers.
• Review relevance, coverage, source revision, and retained context independently.
• Compare query and document embedding contracts before tuning scores.
• Use frozen cases and source snapshots to localize drift.
• Test generation with reviewed evidence before assigning a generator defect.

Knowledge check

References

• AWS, Retrieve
• AWS, KnowledgeBaseRetrievalResult
• AWS, Use metrics to understand RAG system performance
• AWS, Amazon Titan Embeddings G1 - Text
• AWS, Amazon Titan Text Embeddings models
• AWS, GetIngestionJob