AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Validate deployed journeys and answer behavior

Completed by the knowledge check

Learning objective

Validate deployments through synthetic user journeys, output drift checks, and hallucination regression tests. You need evidence that the deployed application works through its real entry point and preserves acceptable semantic behavior.

Core idea

A deployment can succeed technically while users receive wrong answers. Validate three different properties: whether the journey executes, whether outputs have changed relative to a defined baseline, and whether responses remain supported by evidence. Availability, drift, and hallucination tests answer different questions.

Amazon CloudWatch Synthetics canaries exercise endpoints and user actions on a schedule. Bedrock evaluation can assess recorded responses and retrieved contexts. Application-defined assertions connect these mechanisms to your product contract. A successful HTTP response, a changed output, and an unsupported claim each require a different interpretation.

See 5.1.7 for release gates, 4.3.9 for controlled failure diagnosis, and 3.3.6 and 4.3.5 for sustained drift baselines and alerting. This lesson validates the deployed path, including changes after the initial release comparison.

How it works

Exercise the real user journey

CloudWatch Synthetics canaries follow customer routes and actions even when no customers are using the application. They can run once or on a regular schedule, as frequently as once per minute, using cron or rate expressions. Use them to exercise the application entry point, request handling, retrieval, generation, and presentation rather than invoking only the foundation model directly.

Build a bounded synthetic journey with a dedicated test identity and known data. For a conversational assistant, that might include asking an answerable question, checking the returned citation, requesting clarification, and attempting an unanswerable question. For a transactional agent, use a test environment or controlled reversible business action. A monitoring journey should not create unreviewed production side effects on every scheduled run.

In a Puppeteer-based Node.js Synthetics canary, executeStep() wraps a named action with pass/fail logging and metrics. AWS documents that returning normally counts as a pass and throwing counts as a failure. Therefore, the step must assert the expected behavior and throw when the assertion fails. A script that receives a response and then returns normally has tested execution, not necessarily correctness.

Use executeHttpStep() for HTTP steps and explicit response validation. Keep transport assertions, response-shape assertions, and business assertions separate. For example, an HTTP success can coexist with a missing answer, a citation to the wrong policy version, or an incorrect claim that an operation completed.

Synthetics publishes SuccessPercent and Duration in the CloudWatchSynthetics namespace. Canaries using executeStep() or executeHttpStep() also publish those metrics with CanaryName and StepName dimensions. A failed citation-validation step can therefore be distinguished from a failed request step. Duration measures the canary or step interval, so label that boundary rather than presenting it as model-only latency.

Define output drift as a comparison

Output drift means that a specified output property changes relative to a specified baseline. It does not inherently mean quality has fallen. A change from an unsupported answer to a correct abstention is desirable drift; unchanged wording can still conceal an obsolete factual claim.

Maintain fixed cases for controlled deployment comparisons. Record the deployed model, prompt, application, retrieval, source, and safety-policy identities. Collect the actual responses through the same entry point the synthetic journey uses. Compare them with baseline responses using reviewed outcome labels and semantic dimensions rather than exact string equality for unconstrained text.

Bedrock judge-based model evaluation supports supplied inference response data. When you supply responses in the prompt dataset, Bedrock skips model invocation and evaluates those responses. This lets you assess observed deployment outputs without accidentally replacing them with fresh model calls. See 5.1.2 for the model-evaluation dataset contract and 5.1.4 for judge limitations.

Freeze the evaluator and rubric for the comparison. Track changes in accepted answers, correct abstentions, unsupported claims, required structured fields, and business completion separately. Compare the same cases with the same repetition policy; a single stochastic difference is insufficient to identify systematic deterioration.

For sampled production outputs, distinguish changed input composition from changed behavior on comparable inputs. Report sample sizes and risk slices. Publish application-defined measurements through PutMetricData, with a named Namespace, bounded Dimensions, and explicit counts. Metrics such as OutputDriftCases and OutputComparedCases are your definitions, not automatic Bedrock drift detectors. Preserve the comparison record for investigation.

Test hallucination against the actual evidence

A hallucination regression suite needs cases whose acceptable claims and evidence are reviewed. Include absent evidence, conflicting source versions, misleading retrieval, and questions that require abstention. Keep known incidents as regression cases, but do not allow them to replace representative coverage.

For RAG, Bedrock retrieve-and-generate evaluation exposes Builtin.Faithfulness, which measures hallucination relative to retrieved text. It also exposes Builtin.Correctness, Builtin.CitationPrecision, and Builtin.CitationCoverage. These address different failure modes: a source can be faithfully repeated yet be wrong, and a correct answer can omit required supporting citations.

The RAG evaluation supports supplying your own inference response data from an external RAG source. Capture both the deployed answer and the retrieved material that was actually used. Do not assess the answer against a repaired evidence set and then claim that the deployed run was grounded. The test must preserve the original evidence boundary.

For direct generation, Bedrock model-evaluation Builtin.Faithfulness checks whether a response introduces information absent from the prompt. That is a different evidence contract from RAG faithfulness against retrieved text. Select the evaluation type that matches where the deployed application supplied its context. Do not treat identically named metrics across evaluation types as interchangeable without checking their definitions.

Use deterministic checks for exact facts such as an approved policy identifier or a test transaction’s committed status. Use judge scores for semantic assessment, with reviewed disagreement handling. A judge’s favorable result does not override an objective contradiction in the application evidence.

Key distinctions and trade-offs

Synthetic journey versus deployment canary: A Synthetics canary is a scheduled test script. A deployment canary exposes limited traffic to a candidate release; see 5.1.5. The shared word does not imply the same function.

Execution success versus semantic success: SuccessPercent reflects the canary’s assertions. It measures answer quality only to the extent that the script explicitly checks answer behavior.

Drift versus regression: Drift is change. Regression is deterioration under a declared acceptance contract. Flagging every wording change rejects benign variation; accepting unchanged wording can preserve an unsupported answer.

Faithfulness versus factual correctness: Faithfulness evaluates agreement with supplied evidence. Correctness asks whether the answer is accurate. Stale source content can satisfy the former and fail the latter.

Architecture / decision scenario

A benefits assistant deploys a prompt revision while its policy source is also updated. A scheduled Synthetics journey reaches the public application, submits a known question, checks the response structure, and verifies the cited policy revision in a named executeStep().

Requests remain successful, but the citation assertion fails. The output-drift comparison records that the answer changed from a correct abstention to a confident coverage claim. The team supplies the captured answer and original retrieved context to RAG evaluation. Faithfulness looks acceptable because the answer repeats the retrieved passage, but the exact revision check proves that the passage is obsolete.

For this CodeDeploy-managed deployment, the team creates a CloudWatch alarm with PutMetricAlarm on the citation step’s SuccessPercent in CloudWatchSynthetics, selecting CanaryName and StepName dimensions. The CodeDeploy deployment group’s enabled alarmConfiguration includes that alarm; its enabled autoRollbackConfiguration selects DEPLOYMENT_STOP_ON_ALARM. When the configured threshold is breached during deployment, CodeDeploy stops the deployment and automatically redeploys the last known good application revision. This rollback path applies to the revision CodeDeploy manages, not independently changed policy-source data. The team corrects source selection and reruns the journey and hallucination suite, including an unsupported question to check that restoring citations does not increase fabricated answers elsewhere.

Common exam traps

Treating an HTTP success as proof of a correct GenAI journey.
Confusing Synthetics canaries with limited-traffic deployments.
Assuming Synthetics performs semantic checks without script assertions.
Treating every output difference as a quality regression.
Judging deployed outputs against evidence they never received.
Treating faithfulness as proof that source content is factually correct.

What to remember

Exercise the real deployed journey with explicit assertions. Define drift against an identified baseline, and evaluate captured outputs rather than substituting fresh generations. Test hallucination against the actual prompt or retrieved evidence, while keeping source correctness and deterministic business checks as independent acceptance gates.