AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications
Monitor retrieval health beyond query speed
Completed by the knowledge check
Learning objective
Monitor vector-store query latency, index health, ingestion freshness, data quality, and optimization routines.
Core idea
A vector store can answer quickly while returning stale or irrelevant evidence. It can also contain valid embeddings while failing queries because its search resources are exhausted. Monitor serving health, ingestion progress, evidence freshness, and retrieval quality as separate dimensions.
Use the selected backend’s documented telemetry. This lesson uses Amazon OpenSearch Service managed domains for concrete vector-index signals and Amazon Bedrock Knowledge Bases for ingestion evidence. Managed-domain metrics are not a universal interface for every vector backend or OpenSearch Serverless collection. See 1.4.1 for backend selection and 4.2.4 for retrieval optimization choices; here you decide whether the evidence shows a healthy retrieval system and whether a tuning change preserved that health.
Treat source freshness as an application requirement. A completed job records ingestion activity, but it does not by itself prove that the newest authorized source revision appears in query results.
How it works
Measure the query path and backend separately
Record application retrieval duration around the actual retrieval call. Separate query embedding, network time, vector search, and any reranking when your architecture exposes those boundaries. Keep request-level observations so you can inspect slow examples rather than only an average.
For managed OpenSearch domains, SearchLatency is documented as the difference in total time taken by searches on a node between successive minutes. It is not a direct distribution of end-to-end user request durations. Use it to investigate backend search work alongside SearchRate and ThreadpoolSearchQueue, not as a substitute for the application’s retrieval latency percentile.
Publish application latency samples with CloudWatch PutMetricData. The API supports individual Value samples or Values and Counts, which support percentile statistics. CloudWatch generally needs raw data points for percentiles; arbitrary aggregate statistic sets do not preserve the distribution. Label custom metrics by backend and operation, and keep query identifiers in diagnostic records.
When latency rises, ask whether traffic increased, queues grew, or work per query changed. An unchanged service average can hide a failing tenant or a slow query class. A low retrieval latency can also accompany an empty result, so record successful evidence retrieval separately from transport completion.
Inspect index availability and vector resources
On managed domains, ClusterStatus.yellow indicates that primary shards are allocated but some replica shards are not. ClusterStatus.red indicates an allocation problem affecting primary and replica shards for at least one index. These signals describe availability and redundancy, not semantic correctness.
Inspect FreeStorageSpace, JVMMemoryPressure, and ClusterIndexWritesBlocked when indexing slows or stops. A write-block signal describes rejected writes; it does not establish the outcome of a separate query. Measure query success and evidence freshness independently.
For approximate k-nearest neighbors (k-NN) search, KNNGraphMemoryUsage reports graph cache size, and KNNGraphMemoryUsagePercentage measures native graph memory relative to knn.memory.circuit_breaker.limit. AWS documents that reaching 100% triggers the breaker and rejects new knn_vector indexing. Inspect KNNCircuitBreakerTriggered alongside KNNGraphIndexErrors and KNNGraphQueryErrors to distinguish resource pressure from successful vector operations.
Do not apply every k-NN signal to every search mode. Graph-cache metrics are scoped to approximate k-NN. Choose alarms from the documented meaning and appropriate statistic; see 4.3.1 for connecting those signals to request evidence.
Track ingestion completion and source freshness
Use the GetIngestionJob status and document-counter contract in 1.4.2 to monitor Knowledge Bases ingestion outcomes. Combine those outcomes with the changed-revision and deletion probes in 1.4.11 to measure when authoritative source changes become visible in retrieval.
Do not interpret updatedAt as the timestamp of the newest source document. It is the job’s update timestamp; for a stopped job, it is the stop time. The optional textReadyAt indicates when all text extraction finished and text became queryable, not an end-to-end quality certificate for every source modality.
Validate quality and optimization outcomes
Knowledge Bases provides per-document ingestion logs through CloudWatch Logs vended delivery. Create a delivery source with PutDeliverySource, the knowledge base ARN as resourceArn, and logType: APPLICATION_LOGS. Configure a destination with PutDeliveryDestination and link it using CreateDelivery; verify the console reports Delivery active. The documented prerequisite includes bedrock:AllowVendedLogDeliveryForResource and destination-specific permissions. These log deliveries do not support structured data stores or a Kendra GenAI Index.
Inspect StartIngestionJob.ResourceStatusChanged events, joining ingestion_job_id, knowledge_base_arn, and document_location to the source being monitored. RESOURCE_IGNORED records an ignored resource; EMBEDDING_FAILED and INDEXING_FAILED identify different failed stages. Inspect status_reasons for the documented reason rather than treating every failure as invalid source content.
Final states include PARTIALLY_INDEXED and METADATA_PARTIALLY_INDEXED; inspect chunk_statistics to understand incomplete processing. These document-level signals explain defects hidden by an aggregate failure count. They still do not certify that successfully indexed content is accurate or authorized: retain semantic and permission checks as separate quality gates.
Track application-defined defects such as missing required metadata, incorrect source revisions, empty retrieval for known-answer cases, and unauthorized evidence. Check embedding compatibility and provenance using the contracts in 1.5.4 and 1.4.5. Backend cluster health cannot establish those properties.
Retain a reviewed query set with expected evidence and compare retrieved results before and after an index change. For managed evaluation, Bedrock provides retrieve-only retrieval-augmented generation (RAG) evaluation jobs for Knowledge Bases or supplied retrieval response data. This separates retrieval assessment from generated-answer quality. Use the relevant evidence labels and acceptance thresholds rather than declaring success from a faster search alone.
Monitor optimization routines as changes with start and end times, configuration identity, and before-and-after outcomes. For OpenSearch, increasing refresh_interval improves indexing performance at the cost of a longer delay before updates become searchable. Check ingestion throughput, query latency, and revision-visibility probes together when changing it. A tuning change that reduces indexing work but violates the source freshness requirement fails the acceptance contract.
Key distinctions and trade-offs
Backend latency versus request latency: Node search work helps locate pressure. Application samples measure the retrieval experience.
Availability versus relevance: Allocated shards and successful queries do not prove the correct evidence was retrieved.
Job time versus evidence age: updatedAt describes a job update. Source revisions and query probes establish freshness.
Indexing throughput versus search visibility: A longer refresh_interval reduces refresh work but extends the delay before updates appear.
Architecture / decision scenario
An operations assistant serves an OpenSearch-backed knowledge base. After a refresh-interval change, query response times remain acceptable, but users receive an outdated escalation policy.
Investigators inspect application retrieval samples and SearchLatency separately. ClusterStatus.yellow reveals reduced redundancy; KNNGraphIndexErrors and graph-memory signals show no vector indexing failure. GetIngestionJob reports completion, while per-document ResourceStatusChanged evidence shows INDEXED for the policy. No EMBEDDING_FAILED, INDEXING_FAILED, or partial-index state explains the symptom, but the revision probe still returns the previous policy.
The team measures the interval from source change to searchable revision and finds that the optimization exceeded the approved freshness window. It restores an acceptable refresh_interval, verifies policy and deletion probes, and reruns the retrieve-only quality comparison. It also addresses the replica allocation problem independently; fixing redundancy would not repair the freshness trade-off.
Common exam traps
- Reading
SearchLatencyas the user’s p95 retrieval latency. - Treating a green or yellow cluster as proof of relevant answers.
- Ignoring write blocks because existing queries still succeed.
- Reading
updatedAtas the source-data freshness timestamp. - Applying managed-domain graph metrics to every vector backend.
- Accepting a speed improvement without checking freshness and retrieval quality.
What to remember
- Measure query latency, index health, freshness, and quality independently.
- Inspect backend-specific resource and failure signals.
- Combine ingestion status and counts with source-revision visibility probes.
- Validate optimization routines against the complete retrieval contract.
Knowledge check
A team must alarm on its p95 retrieval latency for a knowledge assistant backed by a managed OpenSearch domain. Which data should the alarm use?
SearchLatency metric with a p95 statisticSearchRate metric with a p95 statisticSearchLatency is the change in total search time on a node between minutes, a backend pressure signal rather than a distribution of request durations. Percentiles need raw samples, which statistic sets of averages do not preserve, so the application publishes its own measured durations.New vectors stop being indexed while queries keep succeeding. ClusterIndexWritesBlocked reads 0 and KNNGraphMemoryUsagePercentage reads 100%. What is the most likely cause?
knn_vector indexingknn_vector indexing is rejected. A cluster-wide write block would show in ClusterIndexWritesBlocked , a missing replica affects redundancy rather than writes, and the memory signal points at the vector index, not the embedding step.A policy document changed at 10. GetIngestionJob reports the sync complete with updatedAt at 10, yet at 11 users still receive the old policy. Which check establishes whether the new revision is retrievable?
updatedAt with the time of the policy changeClusterStatus metrics for the periodupdatedAt is when the job record last changed, and counters record ingestion activity; neither shows what a query now returns. Only a revision probe measures freshness, and cluster status describes availability, not evidence age.An ingestion job finishes with one failed document among 2,000, and the aggregate counters do not say why. Which evidence identifies the stage that failed?
ResourceStatusChanged log event and its status_reasonsstatistics field returned by GetIngestionJobKNNGraphIndexErrors metric for the job’s windowtextReadyAt timestamp compared with updatedAtEMBEDDING_FAILED , INDEXING_FAILED , ignored, and partially indexed outcomes, with the documented reason. Job statistics only count failures, and a backend metric cannot tell which source document failed or at which stage.To speed up bulk indexing, a team raises the index refresh_interval. Indexing throughput improves and query latency holds, but revision probes now show updates becoming searchable after four minutes against an approved two-minute window. What is the verdict?