AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Gate releases on complete evaluation evidence

Completed by the knowledge check

Learning objective

Automate continuous evaluation, regression testing, and release quality gates. For a production release, you need an automated decision that ties complete test evidence to the exact candidate configuration being promoted.

Core idea

Continuous evaluation is a repeatable evidence pipeline. Regression testing compares a candidate with an accepted baseline under the same contract. A release quality gate consumes those results and decides whether deployment may proceed. Running an evaluation does not implement the other two functions automatically.

Amazon Bedrock evaluation jobs produce assessment evidence; your release workflow owns the acceptance policy. In particular, GetEvaluationJob.status: Completed means that the job completed. It does not mean that your application met its quality thresholds. Keep successful execution, complete scoring, and acceptable behavior as three separate conditions.

See 5.1.1 for quality dimensions, 5.1.2 for dataset design, and 1.6.6 for prompt regression contracts. This lesson connects that evidence to automated promotion.

How it works

Bind the run to the candidate

Give each evaluation run an application-defined identity that records the code revision, model identifier, prompt version, retrieval configuration, source snapshot, dataset revision, evaluator configuration, and acceptance-policy revision. Preserve the corresponding baseline identity. If a retrieval index or judge changes while the run is executing, its results no longer describe the comparison you intended.

Use CreateEvaluationJob to submit a Bedrock model or RAG evaluation. Its applicationType distinguishes ModelEvaluation from RagEvaluation; evaluationConfig specifies the assessment, inferenceConfig specifies the inference source, and outputDataConfig.s3Uri selects the results destination. roleArn supplies the service role. These fields describe what runs and where its evidence goes; they do not express your promotion thresholds.

Set clientRequestToken consistently when retrying submission of the same logical job. AWS documents this token as an idempotency mechanism: a matching prior request is ignored without an error. Use a different logical submission identity when the candidate or dataset changes. Submission retries should recover work, not silently turn one release comparison into another.

Separate orchestration from assessment

A Step Functions workflow can call the Bedrock control plane through AWS SDK integrations. The documented resource form is arn:aws:states:::aws-sdk:bedrock:apiAction; for submission and polling, the operations are createEvaluationJob and getEvaluationJob. Step Functions expresses API parameters in PascalCase, such as JobIdentifier, even though the native Bedrock API uses jobIdentifier.

Retain the returned jobArn, wait between polls, and inspect GetEvaluationJob.status. The documented states are InProgress, Completed, Failed, Stopping, Stopped, and Deleting. Only Completed should enter the result-assessment branch. A stopped, failed, or disappearing job supplies no positive acceptance evidence. Keep a deadline for the workflow so an unresolved job cannot hold the release indefinitely.

After completion, have an application-owned assessor read the results from the configured S3 destination and verify the expected cases and metrics. Bedrock evaluation jobs write the full report to the configured S3 destination; judge-based model evaluations include scores and explanations. Count failed or missing assessments separately from low scores. An average over only the scored subset can improve when difficult cases fail to evaluate.

Encode the release decision

Return an explicit result from the assessor, such as an application-defined gatePassed boolean together with evidence completeness and failed-case counts. A Step Functions Choice state can route on that value to promotion or a failed terminal outcome. Give the Choice a Default failure path so a malformed or missing decision does not advance the release. Bedrock does not define these application fields or choose the threshold for you. Only a completed job reaches the assessor, and only an explicit passing decision reaches promotion.

Keep two complementary criteria. Absolute floors prevent a consistently poor baseline from legitimizing another poor release. Paired regression criteria prevent an otherwise acceptable average from hiding damage to previously passing cases. Apply both per consequential slice, such as permission handling, unanswerable questions, or required business steps.

Stochastic outputs require an explicit repetition policy. Decide how many candidate and baseline observations each case needs before the run starts. Store every attempt, and apply the same aggregation rule to both sides. Selectively rerunning a failed case until it passes changes the test into a search for a favorable outcome.

For CI visibility, an assessor can emit JUnit XML and fail its build command when the acceptance policy fails. AWS CodeBuild supports JUnit XML test reports and report groups configured in the buildspec. Its buildspec also supports finally commands for preserving evidence after a command fails. The report communicates the result; it does not replace the command’s failure or the workflow’s gate. CodeBuild reports expire after 30 days, so export raw report files to S3 when the release record must remain available longer.

Key distinctions and trade-offs

Continuous versus exhaustive: Run a small critical suite for frequent changes and a broader representative suite before promotion or on a schedule. EventBridge Scheduler supports Step Functions StartExecution as a templated target: specify the state machine ARN in Target.Arn, the execution role in Target.RoleArn, and evaluation-run input in Target.Input. Grant that role states:StartExecution for the target state machine. This starts the same evaluation workflow against the deployed configuration on a recurring schedule. Both suites need explicit coverage; a fast smoke suite cannot claim assurance for omitted slices.

Regression versus improvement: A candidate can improve average helpfulness while breaking a critical refusal case. Gate consequential behavior independently rather than permitting unrelated gains to cancel it out.

Assessment error versus application defect: A missing score is an incomplete measurement. An unacceptable generated answer is a measured defect. Both can block a release, but they have different recovery owners.

Automation versus approval: Automated checks enforce a predeclared policy. Human approval addresses a named judgment or exception; see 3.3.5. Neither an approval button nor an evaluation job should silently bypass the other’s required evidence.

Architecture / decision scenario

A support assistant changes its prompt and retrieval configuration. The release workflow freezes their identities, submits paired baseline and candidate RAG evaluations, and retains each jobArn. Its polling branch requires Completed for both jobs before reading their S3 reports.

The assessor finds that the candidate improves general answers but loses a previously correct abstention on an unsupported refund policy. It marks the critical regression gate false. A separate case has no usable score, so the run also fails evidence completeness. The workflow’s Choice routes to failure and retains the reports rather than promoting the candidate.

After fixing the retrieval defect and recovering the assessment failure, the team starts a fresh, identified comparison. EventBridge Scheduler invokes the evaluation state machine through its StartExecution templated target, using the same policy against the deployed configuration to detect later source changes. Its results inform a release or incident decision; they do not silently rewrite the accepted baseline.

Common exam traps

Treating Completed as a quality approval.
Promoting before all required cases have usable evidence.
Comparing candidate and baseline under different datasets or judges.
Accepting aggregate improvement despite a critical regression.
Retrying stochastic test cases until favorable output appears.
Assuming a CodeBuild report itself enforces a deployment gate.

What to remember

A repeatable run freezes the candidate, baseline, dataset, evaluator, and policy. Bedrock supplies assessment evidence; Step Functions or CI applies the release decision. Require completed execution, complete results, absolute quality floors, and acceptable paired regressions before promotion. Preserve the evidence and its identity even when the gate fails.