AWSAWS Certified Generative AI Developer - Professional2026-105.2 Troubleshoot GenAI applications

Repair retrieval without hiding evidence defects

Completed by the knowledge check

Learning objective

Remediate embedding model mismatches, stale indexes, chunking defects, and slow vector queries.

Core idea

A retrieval repair changes the component that lost useful evidence. Increasing the result count or changing the generation prompt can conceal the symptom while preserving its cause. First isolate retrieval from generation as in 5.2.4. Then determine whether the failure comes from vector compatibility, source freshness, document segmentation, or query execution.

For each repair, preserve a failing query, the expected source revision, the returned passages, and the configuration that produced them. Test the repaired retriever directly before asking a model to answer. A plausible final answer is weak evidence: the model might answer from prior knowledge while retrieval remains broken.

The acceptance contract has two parts. The retriever must return authorized evidence that supports the expected answer, and it must do so within the workload’s latency budget. A faster search that removes the necessary exception clause is a regression.

How it works

Restore the embedding contract

Use the embedding diagnostics in 5.2.4 and the integration contract in 1.5.4 to identify which deployed path differs from the approved configuration.

For a custom pipeline, restore the failing encoder or decoder rather than changing unrelated retrieval settings. For example, when a Titan V2 worker requests only binary in embeddingTypes, its response omits embedding; repair a decoder that still expects that field before publishing vectors. See 1.5.5 for the store integration contract.

For managed Amazon Bedrock Knowledge Bases, the embedding model is fixed in knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration.embeddingModelArn. UpdateKnowledgeBase cannot change knowledgeBaseConfiguration or storageConfiguration. To change the embedding model, create a new knowledge base, re-ingest the documents, and switch the application’s knowledgeBaseId only after regression queries pass. A sync of the existing knowledge base is not a model migration. Preserve the previous serving destination as a recovery option while validating the replacement.

If document and query embeddings differ, restore the known-good query configuration for immediate recovery or build a separate, fully regenerated index for the replacement contract. Do not pad a vector to satisfy a mapping or mix incompatible generations in one searchable collection. Validate the replacement with the same queries before changing the application’s retrieval destination.

Reconcile the source projection

For Amazon Bedrock Knowledge Bases, source edits require synchronization. StartIngestionJob takes knowledgeBaseId and dataSourceId; use its returned ingestionJobId with GetIngestionJob. Incremental synchronization processes added, modified, and deleted documents, including the parsing, chunking, embedding, and indexing required for re-ingestion.

Inspect status, failureReasons, and statistics, including numberOfDocumentsFailed, numberOfModifiedDocumentsIndexed, and numberOfDocumentsDeleted. A finished job is an operational checkpoint. It is not a document-by-document assertion that the expected revision appears in a query result. AWS documents that, for stores other than Amazon Aurora (RDS), newly ingested vectors can take a few minutes to become queryable after ingestion completes.

Use a query whose answer differs between the replaced and replacement documents. Inspect the returned evidence and any revision metadata your application maintains. Repeat the check for deleted documents: absence from the source must become absence from authorized retrieval. See 1.4.11 for reconciliation design; the troubleshooting question here is whether this specific repair reached the serving projection.

If failed-document counts remain nonzero, fix the affected ingestion inputs rather than repeatedly declaring the index refreshed. Keep the previous serving configuration available until the replacement’s freshness tests pass.

Repair evidence boundaries

Read the retrieved passage beside its source. A fragment can contain the headline rule but omit a condition, table heading, or neighboring exception. Determine whether evidence disappeared during extraction or during chunking; changing chunk size cannot restore text the parser never extracted.

Knowledge Bases fixed-size chunking lets you choose token size and overlap percentage. Hierarchical chunking retrieves child chunks and replaces them with broader parent chunks; consequently, the returned result count can be smaller than the requested count. That behavior is not automatically missing data. Semantic chunking places boundaries using sentence dissimilarity and adds foundation-model processing cost. See 1.5.1 for strategy selection rather than treating any strategy as the repair by default.

You cannot change a data source’s chunkingConfiguration in place with UpdateDataSource. Re-create the data source for a changed chunking configuration. Treat that as a controlled re-ingestion change, with an explicit plan for existing vectors and the serving destination. Re-run cases that need adjacent clauses and cases that need a precise small passage. Larger context can repair one case while adding distracting material to another.

Locate slow vector work

Measure retrieval separately from generation and compare the same query under representative concurrency. For an Amazon OpenSearch Service managed domain, ThreadpoolSearchQueue and ThreadpoolSearchRejected expose queued and rejected search work. KNNGraphMemoryUsage measures graph memory on each node. KNNGraphMemoryUsagePercentage measures native graph memory relative to knn.memory.circuit_breaker.limit; at 100%, the circuit breaker rejects new knn_vector indexing. Check this pressure before treating an indexing failure as a source defect. Scope these mechanisms to that backend rather than assuming every Knowledge Bases store exposes them.

For direct OpenSearch k-NN queries, inspect both k and size. AWS documents a maximum k of 10,000; without an explicit size, results can reflect per-shard and per-segment neighbors rather than the desired whole-query count. A post_filter can also leave fewer than k results. Increasing candidate work indiscriminately may worsen the incident without repairing its relevance defect.

Use backend pressure and per-request measurements to choose between capacity changes and query changes. Keep authorization filters intact. See 4.2.4 for optimization choices and 4.3.8 for health monitoring; the repair must demonstrate restored useful evidence as well as improved timing.

Key distinctions and trade-offs

Compatibility versus relevance: An accepted vector shape proves structural compatibility, not that ingestion and query vectors represent meaning consistently.

Synchronization versus visibility: COMPLETE establishes job completion. Retrieval tests establish whether the intended revision is serving.

Chunk repair versus context expansion: Re-segmentation changes the searchable evidence. Adding more retrieved results changes how much existing evidence you request.

Capacity versus query work: More resources can relieve measured pressure. They do not correct stale documents, incompatible vectors, or omitted clauses.

A replacement index gives you an explicit comparison and recovery boundary, but requires temporary storage and reprocessing. A small query adjustment costs less, but is appropriate only when tests locate the defect in query configuration.

Architecture / decision scenario

A policy assistant becomes slow and returns a superseded refund rule after an embedding rollout. One failing case also omits the exception for damaged products.

The team first freezes the query and inspects retrieval without generation. The custom query worker uses a different embedding model from the document-ingestion worker, although both emit 1,024-dimensional vectors. The accepted vector length conceals the incompatible semantic space. It restores the compatible query configuration, then checks ingestion statistics for the revised policy. A failed document explains the stale passage; a successful retry still requires a query-visibility test.

The exception remains absent from otherwise fresh evidence. Reviewing the source and chunks shows a segmentation defect, so the team creates a replacement data source with tested boundaries. Finally, elevated search queues and graph memory direct a separate capacity investigation. Release requires compatible vectors, visible revisions, complete exception evidence, and acceptable retrieval latency on the regression cases. One corrected answer cannot substitute for these four checks.

Common exam traps

Repairing a model mismatch by padding vectors to the index dimension.
Treating equal dimensions as proof of a shared embedding space.
Calling UpdateKnowledgeBase to switch the embedding model.
Declaring freshness solely because an ingestion job reports COMPLETE.
Updating chunkingConfiguration in place instead of re-creating the data source.
Treating hierarchical parent replacement as a guaranteed missing-result defect.
Removing access filters or raising k indiscriminately to improve retrieval.

What to remember

Repair the failed retrieval contract before tuning generation.
Align actual ingestion and query embeddings, including representation.
Verify source revisions and deletions through serving queries.
Re-create a Knowledge Bases data source when changing chunking configuration.
Accept performance repairs only when evidence and authorization tests still pass.