AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Turn feedback into traceable reviewed labels

Completed by the knowledge check

Learning objective

Collect user ratings, feedback, and human annotations through feedback interfaces and annotation workflows.

Core idea

Feedback collection becomes evaluation evidence only when you know what was rated, why, and under which rubric. For an engineer building an improvement loop, distinguish three artifacts: a user’s reaction, a structured report of a defect, and a reviewed annotation against approved evidence.

A thumbs-down can mean the answer was false, slow, irrelevant, or unwelcome despite being correct. A thumbs-up can reward confident wording rather than factual accuracy. Preserve these signals without silently turning them into ground truth or permission to retrain a model.

Build the loop around durable response identity and authorized review context. The application owns production feedback capture; Amazon Bedrock human evaluation supplies a managed assessment workflow for prepared cases. Custom annotation workflows have their own resource and eligibility boundaries.

How it works

Capture a reaction against the displayed response

Place a small rating control next to a completed answer and offer structured reasons for a negative rating, such as incorrect facts, missing information, unclear wording, or poor relevance. Add optional free text for details the categories cannot capture. Define whether the control rates one answer or the completed conversation; mixing these units makes the aggregate ambiguous.

Bind feedback to the exact response that the person saw. Preserve response identity, conversation or request identity, model and prompt version, evidence revision, and actual outcome in an application-owned record. Keep the displayed answer snapshot or an authorized retrievable reference to it. A rating attached only to a model name cannot explain which configuration or evidence produced the defect.

Use trusted server-side context to associate the feedback with an authorized response. Do not let a submitted response identifier alone establish permission to review another tenant’s content. Authorization patterns belong to 2.3.7, and minimization and privacy transformations belong to 3.2.7. Collect only the data needed to interpret the feedback.

For a durable feedback store in DynamoDB, PutItem supports a ConditionExpression. A condition using attribute_not_exists on the partition key prevents a new write from replacing an existing item with the same key. Use an application-defined event identity to deduplicate retried submissions, and define whether changed ratings create a new event or replace an earlier choice. The database condition enforces the chosen write boundary; it does not decide the product’s rating semantics.

Keep the raw rating, reason, and comment separate from any later reviewed label. This separation lets you audit a disagreement between the person reporting the problem and the evaluator who checks the facts.

Prepare cases for human evaluation

Sample feedback-linked cases along with cases that received no feedback. Include positive reactions, negative reactions, and ordinary unreviewed responses. Voluntary feedback is a self-selected sample; its satisfaction rate is not automatically representative of all production users.

Amazon Bedrock human evaluation uses a custom S3 JSONL dataset. prompt and optional referenceResponse are visible to workers, while category supports filtering in the report card. Bring-your-own inference responses lets you evaluate the response already shown in production rather than generating a different answer for the review.

Define metrics through HumanEvaluationCustomMetric.name, description, and ratingMethod. Supported methods include ThumbsUpDown, IndividualLikertScale, ComparisonLikertScale, ComparisonChoice, and ComparisonRank. Choose a method that matches the review question: a binary approval gate, a graded individual judgment, or comparative preference. The meaning comes from the rubric and instructions, not from the widget alone.

Bedrock lets you select the number of workers per prompt; the final report retains each worker’s ratings. Use independent ratings where disagreement matters and send consequential conflicts to adjudication as described in 5.1.4. Keep worker judgments and the final approved label as separate artifacts.

Human jobs can use up to two inference sources and a custom dataset with up to 1,000 prompts. Partition larger reviews deliberately, keeping instructions and rating semantics consistent. For console-created human jobs, configure CORS on the S3 output bucket so the annotation portal can display prompts and results. This is a browser-access requirement, not an authorization policy for arbitrary reviewers.

Match the annotation workflow to the service boundary

Bedrock human evaluation uses a work team drawn from a private workforce managed through SageMaker Ground Truth. AWS documents a maximum of 50 workers per Bedrock work team. Existing workers are not automatically notified for every additional evaluation job; the workflow owner must provide the worker portal URL and notify them. Include that operational step when estimating review turnaround.

For Bedrock evaluation started through API operations, the documented integration requires a SageMaker flow definition ARN. CreateFlowDefinition uses HumanLoopRequestSource.AwsManagedHumanLoopRequestSource with AWS/Bedrock/Evaluation, and the evaluation job supplies humanWorkflowConfig.flowDefinitionArn. Follow this Bedrock-specific integration rather than substituting a generic custom human loop.

Amazon A2I is no longer open to new customers; existing customers can continue using it. Do not prescribe a new A2I dependency as universally available. The Bedrock documentation still describes its flow-definition integration; verify that path for the intended account instead of inferring general A2I eligibility from Bedrock’s human-evaluation feature.

For an existing A2I customer using a custom annotation workflow, CreateFlowDefinition identifies the work team, worker interface, and S3 output through HumanLoopConfig and OutputConfig. StartHumanLoop supplies FlowDefinitionArn, HumanLoopName, and HumanLoopInput.InputContent. Custom task types do not support built-in activation conditions: the application decides when to call StartHumanLoop, and each call initiates review. Keep that custom path separate from a Bedrock evaluation job.

Key distinctions and trade-offs

User rating versus annotation: A rating captures experience. An annotation applies a defined criterion to evidence. Retain both so satisfaction does not masquerade as factual correctness.

Feedback sample versus production estimate: Feedback-rich cases are useful for discovery. Add a planned sample of unrated traffic before estimating population-wide quality.

Managed review versus workflow ownership: A portal and work team do not define sampling, reviewer expertise, adjudication, or label promotion. The application team still owns those decisions.

Review completion versus learning: A completed annotation creates evidence. Dataset promotion, prompt changes, and model customization require separate approval and validation.

Architecture / decision scenario

A support assistant records response-linked thumbs ratings, structured reasons, and optional comments. Its DynamoDB write condition prevents a network retry from duplicating the same feedback event. A complaint about an incorrect return deadline becomes a review candidate, while the raw rating remains intact.

The team combines complaints with a sample of unrated responses, prepares approved references, and evaluates the supplied answers through a Bedrock human job. Two reviewers disagree on one policy exception, so a specialist adjudicates against the preserved policy revision.

The approved defect label enters the regression dataset; a positive rating on another factually wrong answer also becomes a defect. The team uses its eligible Bedrock workflow and does not assume that a new account can adopt A2I custom loops. Feedback informs the next experiment without automatically changing production behavior.

Common exam traps

Treating thumbs-up feedback as a factual ground-truth label.
Reviewing regenerated text instead of the answer the person rated.
Estimating all-user satisfaction from voluntary feedback alone.
Assuming a managed portal removes sampling and adjudication work.
Relying on automatic notification for every new Bedrock review job.
Prescribing A2I custom workflows without checking customer eligibility.

What to remember

Attach feedback to a durable response and configuration identity.
Preserve raw reactions separately from reviewed labels.
Give reviewers authorized context, clear metrics, and explicit instructions.
Separate Bedrock evaluation integration from custom A2I loops.
Promote reviewed cases into evaluation datasets through an accountable process.