AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Separate causal experiments from deployment safety

Completed by the knowledge check

Learning objective

Run controlled A/B tests, canaries, and model/prompt version comparisons.

Core idea

An A/B test estimates the effect of a candidate on a defined population. A canary limits exposure while detecting a harmful release. An offline version comparison tests the same cases against frozen candidates. For an engineer promoting GenAI changes, these are complementary controls with different evidence requirements.

A canary that shows no errors does not prove an improvement in answer quality. An offline winner does not prove that production users complete more tasks. A traffic split alone does not create a controlled experiment if users switch candidates mid-conversation or other dependencies change simultaneously.

Freeze the comparison unit, candidate package, assignment rule, primary outcome, and stop conditions before collecting results. Then choose the AWS mechanism for the resource being changed: application configuration for Bedrock routing, or endpoint variants and deployment guardrails for eligible SageMaker AI endpoints.

How it works

Establish a reproducible version comparison

Define a candidate as more than a model name. Preserve the model target, prompt version, inference settings, retrieval revision, guardrail configuration, and application code that assembles requests. If the intended experiment changes only a prompt, hold other dependencies fixed. If it tests a full package, state that the measured effect belongs to the package.

Amazon Bedrock Prompt management distinguishes an editable draft from a version that snapshots the prompt and configuration. Use saved versions for comparisons. The versioning and promotion mechanics belong to 1.6.2 and 1.6.3; this lesson uses version identity to prevent a candidate from changing during measurement.

With Converse, a Prompt management version ARN can be the modelId, and promptVariables supplies values for the prompt’s variables. When you use a Prompt management resource, the request cannot also include additionalModelRequestFields, inferenceConfig, system, or toolConfig; define those through Prompt management instead. Do not build an experiment that assumes arbitrary runtime overrides of a saved prompt’s configuration.

Run paired cases through each candidate and preserve per-case results. Repeat where nondeterministic output could change a decision. Judge evaluation boundaries belong to 5.1.3; an offline comparison should keep the evaluation dataset and rubric fixed and inspect regressions in critical slices before online exposure.

Assign A/B cohorts at the right boundary

Choose the assignment unit according to the workflow: user, tenant, conversation, or independent request. A conversational assistant usually needs a stable conversation or user assignment so a follow-up does not silently change behavior. Randomize within the eligible population, then record actual exposure alongside assignment.

AWS AppConfig multi-variant feature flags evaluate supplied context against ordered rules and return the default variant when no rule matches. Its split operator uses a consistent hash of the context value to select a percentage of traffic. Use a stable application-owned assignment identifier in that context and keep the split rule fixed during the experiment. A changed key or rule can change membership.

Return a candidate configuration from the flag and let the application invoke the selected Bedrock model or prompt version. AppConfig selects configuration; it does not itself call the model or measure business success. Log the selected candidate, configuration revision, experiment identity, and terminal outcome in your experiment record. AppConfig selects the candidate, while the application calls the model and records what it selected.

For models hosted on SageMaker AI, production variant weights distribute requests randomly across variants. InvokeEndpoint.TargetVariant targets a named production variant and overrides weighted random routing. Use explicit targeting when your application owns stable cohort assignment. Request-level weight distribution alone does not guarantee that all turns from one user stay on the same model.

UpdateEndpointWeightsAndCapacities changes the relative weights without requiring an endpoint update. This is a traffic-routing mechanism, not an automatic decision about statistical significance or acceptable model quality. Keep the planned experiment rule separate from the operational ability to change weights.

Use canaries to bound release exposure

See 1.2.8 for SageMaker deployment mechanics and eligibility. For experiment design, the deciding boundary is that CanarySize.Type: CAPACITY_PERCENT measures green-fleet capacity, not a stable user cohort; the documented canary size is at most 50% of that capacity. Do not infer user assignment from the capacity setting.

Canary alarms must detect a harmful change within the baking interval. Endpoint health can react quickly, while expert review or delayed business outcomes may arrive after that interval. Keep those outcomes in a separately controlled experiment rather than treating a quiet canary as a completed quality assessment.

See 1.2.3 and 1.6.3 for AppConfig rollout, and 1.2.8 for rollback of Bedrock routing configuration. AppConfig GrowthFactor measures the percentage of configuration targets per interval, not experiment users. Record which candidate each user actually receives instead of equating rollout progress with treatment exposure.

Key distinctions and trade-offs

Assigned cohort versus rollout target: An experiment assigns a treatment to a chosen unit, such as a conversation. A rollout updates configuration targets or fleet capacity. A rollout percentage alone cannot tell you which conversations received the treatment.

Assignment versus exposure: A user can be assigned to B but receive a fallback response from A. Preserve both facts; silently relabeling the user after failure biases the comparison.

Version versus alias or draft: A reproducible comparison names a frozen artifact. A mutable target can point to different content during the experiment even when its identifier appears unchanged.

Operational alarm versus business evidence: No rollback alarm means no detected alarm condition. It does not prove factual acceptance, task completion, or a worthwhile improvement.

Planned analysis versus repeated peeking: Predefine sample requirements, observation windows, primary outcomes, and critical guardrails. Examine uncertainty and relevant slices before promotion. Repeatedly checking until a favorable difference appears weakens the inference; plan stopping rules and record any deviation from them.

Architecture / decision scenario

A support team changes only a Prompt management version. Offline paired cases pass policy and consistency gates. The application then uses an AppConfig split rule with a stable conversation identifier to assign eligible conversations to A or B, and records the actual version used on every turn.

A gradual configuration rollout limits initial exposure and monitors configured latency and error alarms. The separate A/B analysis measures resolved tickets, corrections, and policy failures over its planned window. B improves resolution but fails the critical policy gate on one product slice.

The team returns the routing configuration to the approved version and preserves the failed cases for regression testing. A quiet canary and an improved aggregate completion rate do not override the release contract.

Common exam traps

Calling a weighted traffic split a complete experiment design.
Using per-request randomness for a multi-turn treatment without accounting for it.
Overriding a Prompt management version with prohibited Converse fields.
Reading canary capacity percentage as a stable user cohort percentage.
Treating no CloudWatch alarm as proof of improved business outcomes.
Promoting a candidate after repeated opportunistic checks of the same result.

What to remember

Freeze candidates and define the assignment and observation units.
Pair offline cases before exposing production users.
Use AppConfig context or TargetVariant for controlled application routing.
Keep canary rollback criteria separate from A/B success criteria.
Preserve actual exposure, fallback, and critical-slice failures.