AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Build datasets around production acceptance decisions

Completed by the knowledge check

Learning objective

Build evaluation datasets that reflect real production use-case distributions and acceptance criteria.

Core idea

An evaluation dataset is a model of the work your application must perform. For an engineer deciding whether to release a candidate, representativeness means preserving the mix of requests, contexts, and outcomes that determine production value. It does not mean collecting many unrelated prompts.

A dataset of easy, answerable English questions can make an assistant look successful while missing its actual failures: ambiguous requests, unavailable evidence, uncommon product versions, or requests that require abstention. Define the target distribution and the acceptance contract together. A prompt without an expected decision can reveal output variation, but cannot reliably distinguish an improvement from a regression.

Keep a production-weighted set for estimating ordinary performance and a separately reported challenge set for rare, consequential failures. This separation preserves both business realism and risk visibility. Neither set should silently replace the other.

How it works

Define cases and their expected outcomes

Start with the workflow rather than a generic benchmark. Identify request families, languages, input lengths, customer groups, evidence availability, and risk classes. Estimate their production frequencies from authorized application records. Amazon Bedrock model invocation logging collects request data, response data, and metadata for supported bedrock-runtime calls in an account and Region, with delivery to Amazon S3 or CloudWatch Logs; an S3 destination bucket must be in the same account and Region as the logging configuration. Logging is disabled by default, so it is a sampling source only after it is enabled. See 4.3.2 for configuration and 3.2.8 for privacy boundaries. Record the observation window and sampling method so another engineer can explain why each slice has its weight.

For each case, preserve the request and the context needed to interpret it. A benefits question may depend on country, plan year, and policy revision; a support question may depend on product version and an earlier clarification. Removing those dependencies creates a different task, even when the visible question stays unchanged.

Write acceptance criteria before generating candidate answers. State required facts, prohibited claims, required output fields, and acceptable terminal behavior. Include correct abstention when evidence is missing and useful clarification when a specific missing input could resolve ambiguity. Do not label every refusal a failure or every answer a success.

The quality dimensions in 5.1.1 supply the rubric. Your dataset determines which cases exercise that rubric and what evidence the evaluator receives. Review reference responses with subject-matter experts, including exceptions and permitted alternative wording. A reference should express an approved answer, not copy a candidate’s response and declare it correct.

Map the case to Bedrock’s dataset contract

For model-as-a-judge jobs, Amazon Bedrock requires a custom dataset in Amazon S3 using JSON Lines and the .jsonl extension. Each line is a valid JSON object. A dataset supports up to 1,000 prompts per evaluation job, so larger suites need planned partitions and consistent aggregation outside the job.

When Bedrock generates responses, each record contains prompt, optional referenceResponse, and optional category. Supplying category produces evaluation scores for each category. Use it for a meaningful reporting slice, such as locale plus risk class; do not assume this one field automatically implements your entire sampling model.

The optional reference influences Builtin.Correctness and Builtin.Completeness. Those metrics also work without a reference, but a no-reference assessment is different evidence from comparison with approved ground truth. Record whether references are present when comparing runs.

For existing responses, add modelResponses entries with response and modelIdentifier. Judge evaluation supports one response per prompt and only one unique modelIdentifier in the job. Keep each candidate’s responses in its own aligned dataset. Maintain the same case order and an external case-to-record mapping so differences can be inspected by case rather than only by aggregate score.

Store richer metadata in an accompanying application-owned dataset manifest, rather than inventing undocumented Bedrock input fields. That manifest can record case identity, source revision, sampling stratum, production weight, annotation revision, and expected terminal behavior. The S3 dataset is the service input; the manifest is your reproducibility contract.

Protect the comparison from leakage

Separate prompt-development examples from the held-out acceptance set. If engineers repeatedly tune against the same failures, that set becomes a development tool. Keep a held-out suite for promotion and add independently reviewed production cases over time. Track dataset revisions so an apparent improvement caused by changed questions is not attributed to the model.

Prevent duplicates from crossing the boundary through near-identical conversations or document revisions. Split related cases together when their shared content would expose the answer. If a reference depends on retrieved material, freeze the evidence snapshot for an offline comparison or explicitly declare that source freshness is part of the experiment.

When evaluating complete application outputs, generate them through the actual pipeline and use Bedrock’s bring-your-own-response path. Bedrock skips model invocation and scores the supplied responses. This lets your test harness preserve retrieval, authorization, and prompt assembly rather than replacing them with isolated raw-model calls.

For human jobs, the custom dataset also uses S3 JSON Lines. prompt and referenceResponse are visible in the worker interface. Remove unnecessary personal data and provide only evidence the authorized evaluator needs. Privacy handling is covered in 3.2.7; evaluation does not create permission to disclose production conversations.

Key distinctions and trade-offs

Production weighting versus challenge coverage: A production estimate reflects expected traffic proportions. A challenge set deliberately overrepresents hard cases. Report both, and avoid presenting the challenge-set failure rate as a population estimate.

Reference answer versus acceptance rubric: A reference provides approved content. A rubric defines what makes an answer acceptable, including alternative wording, clarification, and abstention. Literal equality is often too narrow for natural-language tasks.

Case count versus independent evidence: Many paraphrases of one question can improve stress coverage without adding many independent business situations. Count both records and distinct cases when judging the strength of the result.

Dataset partition versus experiment change: Splitting a large suite across jobs is an operational choice. Changing candidate settings, references, or evaluator instructions between partitions changes the measurement and needs explicit accounting.

Architecture / decision scenario

An assistant receives mostly product-help requests, with a smaller share of refund disputes and multilingual questions. The team samples ordinary traffic proportionally and creates a separate challenge set for expired policies, unsupported locales, and missing purchase dates.

Each case records approved policy evidence and acceptable outcomes. A missing date expects clarification; an obsolete policy expects abstention or escalation. The team emits S3 JSONL records with category and reviewed referenceResponse, while a separate manifest retains sampling weights and evidence revisions.

Candidate outputs come from the full application and are scored in aligned judge jobs. Promotion requires acceptable production-weighted results and passing the dispute-risk gate. Increasing ordinary helpfulness cannot hide a regression on policy exceptions.

Common exam traps

Assuming a public benchmark reproduces your production distribution.
Omitting unanswerable or ambiguous requests from the acceptance set.
Treating a reference response as a requirement for exact wording.
Combining several candidate identifiers in one judge evaluation job.
Adding undocumented metadata fields to the Bedrock dataset format.
Reusing tuning examples as independent evidence for release approval.

What to remember

Sample from the workflow you intend to support.
Define acceptable answers, clarification, and abstention before scoring.
Separate production weights from deliberate challenge coverage.
Preserve case identity and evidence revisions outside the service input.
Partition beyond 1,000 prompts without changing the measurement contract.