AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Resolve evaluator disagreement through reviewed evidence

Completed by the knowledge check

Learning objective

Account for limitations and disagreement among automatic scores, LLM-as-a-judge, and human evaluation.

Core idea

Evaluation methods observe different properties of an answer. For an engineer responsible for release approval, disagreement is diagnostic evidence: the methods may use different rubrics, lack required context, or disagree about the reference itself. A majority vote does not repair those problems.

Use automatic scores for repeatable measurements, judge models for rubric-based assessment at scale, and humans for domain interpretation and adjudication. Each method can be useful and wrong. Define what evidence each method receives, how it is calibrated, and which disagreements block a release.

The output of evaluation is an accountable acceptance decision, not a collection of scores that happens to look favorable. Preserve contradictory judgments so the team can understand what failed and reproduce the resolution.

How it works

Understand what automatic scores measure

Bedrock automatic evaluations use task-specific computed metrics. For question and answer, accuracy uses NLP-F1; summarization uses BERTScore. Those measurements provide reproducible comparison evidence, but they do not have identical semantics and are not interchangeable with a domain expert’s factual review.

Consider a reference that says a customer has thirty days to return an item. A candidate substitutes sixty days while retaining much of the wording. Similarity-based evidence can remain favorable even though the changed number determines the business decision. Conversely, a concise paraphrase may express the approved answer using different words. Interpret the score alongside the required facts and acceptable alternatives.

Automatic robustness is also task-specific. Bedrock documents F1 and deltaF1 for question answering, and BERTScore and deltaBERTScore for summarization. Do not relabel one robustness result as proof that every production perturbation, language, or adversarial input is safe. The dataset and measurement scope bound the conclusion.

Calibrate judge-model assessment

Bedrock judge jobs produce scores and explanations for prompt-response pairs. Built-in metrics use metric-specific evaluator prompts. Builtin.Correctness and Builtin.Completeness consider referenceResponse when supplied; Builtin.Faithfulness assesses whether the answer adds information absent from the prompt. Missing evidence can therefore change what a judge is capable of assessing.

Use a reviewed calibration subset containing acceptable answers, subtle factual errors, legitimate paraphrases, missing required facts, and correct abstentions. Compare judge acceptance decisions with expert labels at the threshold you intend to use. Count false acceptance and false rejection by important slice rather than only measuring correlation between numeric scores. 3.4.5 applies the same calibration to fairness slices.

Treat preference for verbosity, formatting, or familiar wording as hypotheses to test. Construct matched examples where these properties change but the approved facts stay fixed. If a judge’s rating changes enough to reverse acceptance, refine the rubric or route that slice to human review. Do not claim a particular judge is unbiased because it is a different model from the generator.

For a custom metric, Bedrock accepts evaluator instructions and an optional rating scale. The custom prompt variables include {{prompt}}, {{prediction}}, and {{ground_truth}}. Define the criterion and scoring guidelines consistently with the scale. Keep the task definition before the input variables, as AWS specifies, so evaluated content does not replace the rubric’s intended role.

A judge explanation is an explanation of its score, not a proof of correctness. Inspect it against approved evidence. Candidate text that tells an evaluator to award full marks should remain evaluated content; include such cases in calibration and verify that the resulting decision follows the rubric.

Make human review reproducible

Bedrock human evaluation supports named metrics with a ratingMethod. IndividualLikertScale rates each response; ComparisonChoice expresses preference between responses. These answer different questions. A preferred answer can still fail a mandatory requirement, so combine preference with independently defined acceptability criteria.

Use the evaluation instructions and metric descriptions to define observable criteria and examples. Bedrock lets you choose the number of workers per prompt, and its final report includes every worker’s ratings. Preserve those individual ratings before deriving an aggregate. Persistent disagreement can expose ambiguous instructions, unequal expertise, or missing source evidence.

For consequential cases, ask qualified reviewers to judge independently and then adjudicate the conflict against the approved source. Hide candidate identity in your own review process where practical and vary presentation order when testing comparative preference. These are experimental controls you own, not guarantees inferred from selecting a managed evaluation service.

The human dataset exposes prompt and referenceResponse to workers. Supply sufficient authorized evidence and train reviewers to distinguish unsupported additions from correct use of that evidence. A reviewer who lacks the policy version cannot reliably resolve a conflict about a policy exception.

Convert disagreement into action

Keep an application-owned disagreement record containing case identity, method, configuration, score, acceptance decision, and reviewer explanation. Bedrock’s full judge report is stored in the S3 destination specified for the job; use that report rather than only the console’s short explanation sample.

Route a case to adjudication when methods disagree on a consequential claim or when a judge explanation contradicts the reference. If the reference is defective, revise the dataset and rerun affected candidates. If the rubric is unclear, revise it and recalibrate. If the candidate is wrong, retain the failure as a regression case rather than weakening the threshold to remove it. Adjudication ends in a revised dataset, a revised rubric or a retained regression case, never in a weakened threshold.

Key distinctions and trade-offs

Agreement versus validity: Several evaluators can share the same bad reference. Agreement improves confidence only when the underlying evidence and rubric are valid.

Preference versus acceptability: Preference ranks alternatives. Acceptability checks a contract. A comparative winner can be unacceptable in absolute terms.

Adjudication versus averaging: Averaging is useful for graded judgments under a common scale. It cannot resolve whether an unsupported high-impact claim is permitted.

Calibration versus final testing: Calibration selects the measurement policy. A held-out evaluation then tests the candidate under that policy; using the same examples for both weakens the conclusion.

Architecture / decision scenario

A returns assistant receives a strong automatic score and a high judge rating for a response that misstates a product-specific deadline. Human reviewers split: one prefers its readability, while a specialist flags the deadline.

The team opens the full judge report and discovers that the reference omitted the product exception. It corrects the approved reference, clarifies that factual eligibility is a hard gate, and recalibrates the judge on exception cases. Reviewers then apply the same rule independently.

The candidate fails factual acceptance despite its favorable similarity and initial preference results. The team retains the disputed case, reruns both candidates against the corrected dataset, and records the evidence behind the release decision.

Common exam traps

Treating similarity to a reference as independent factual verification.
Assuming a different judge model guarantees impartiality.
Accepting a judge’s explanation without checking its evidence.
Averaging a factual disagreement into a passing presentation score.
Treating comparative preference as absolute acceptance.
Declaring human labels infallible without instructions and adjudication.

What to remember

Every evaluation method has an evidence and rubric boundary.
Calibrate judge decisions against reviewed cases and error costs.
Preserve individual human judgments and resolve important conflicts.
Repair references or rubrics before interpreting a disputed ranking.
Keep hard factual gates separate from graded preference.