AWSAWS Certified Generative AI Developer - Professional2026-101.2 Select and configure FMs

Benchmark the complete model operating envelope

Completed by the knowledge check

Learning objective

Benchmark candidate FMs on representative quality, throughput, latency, and cost criteria. Use the eligible shortlist from 1.2.1 to select a model that meets the complete production contract.

Core idea

A benchmark is a controlled comparison of outcomes under specified conditions. The winner is not necessarily the model with the highest average quality score or the lowest token price. You need acceptable results delivered within the latency and capacity envelope, at a cost the business can sustain.

Compare candidates on the same task population and disclose any deliberate differences in prompts or inference settings. Keep quality gates separate from optimization targets. A model that fails a critical task slice does not become acceptable because it is inexpensive on the easiest requests.

How it works

Freeze the comparison contract

Start with the requirements from 1.1.1 rather than inventing a benchmark around convenient examples. Assemble a versioned set of realistic inputs, reviewed expected outcomes, and a scoring rubric. Include common requests, long inputs, difficult cases, missing evidence, and requests that should produce clarification or abstention.

Amazon Bedrock evaluations provides programmatic, human-based, and model-as-judge evaluation paths. Custom prompt datasets let you test your own task instead of relying entirely on a built-in dataset. Use the applicable evaluation path to produce quality evidence, while retaining your definition of an acceptable result. A score supplied by a service does not choose the business threshold.

Freeze the model ID or inference profile, prompt version, dataset version, response limits, and inference settings for each candidate. Run repeated trials where output variability matters. If one model needs a different prompt to meet the contract, record it as a candidate configuration; do not describe the experiment as a pure model-only comparison.

Quality includes response validity and task correctness. An answer that parses successfully can still invent a policy. A grounded refusal can be correct for an unanswerable request. Keep these outcomes distinct so a model is not rewarded merely for returning fluent text every time.

Measure response time at both boundaries

Instrument the application from request acceptance to a usable, validated result. Separately measure the model call so you can explain where time goes. If the product streams responses, measure time to first useful output and time to complete output independently. A fast first token does not guarantee a fast complete document.

For Bedrock’s bedrock-runtime endpoint, CloudWatch’s InvocationLatency measures the interval from sending the request to receiving the last token. For ConverseStream and InvokeModelWithResponseStream, TimeToFirstToken measures the interval to the first token in milliseconds. Converse also returns metrics.latencyMs, and its response includes usage and stopReason. Your benchmark harness should retain these diagnostics with the candidate identity.

Use latency distributions, including tail behavior, rather than one average. Separate short and long outputs before interpreting a change. A model that generates a much longer answer can have higher completion latency even when its generation throughput is strong. Limit response length consistently with the task, then evaluate whether the shorter output remains useful. See 4.2.1 for the latency breakdown and 4.2.3 for benchmarking latency-optimized inference.

Exercise representative load

Run load cases with the production mix of prompt sizes, output sizes, and simultaneous requests. Measure accepted completions per unit of time as well as attempted calls. A load generator can submit more requests while producing fewer usable answers because retries or throttling consume the budget.

Use CloudWatch’s InvocationThrottles, InvocationClientErrors, and InvocationServerErrors to investigate capacity pressure, request defects, and service failures. InvocationClientErrors also includes requests rejected for exceeding a service quota; those rejections are not counted as throttles. EstimatedTPMQuotaUsage approximates token-per-minute quota consumption, but does not represent the reservation-based accounting that drives throttling, so do not use it alone for capacity planning. Retry settings affect observed throttles. Record the SDK retry configuration and the application’s deadline so one candidate does not receive hidden extra attempts.

An isolated request establishes neither burst tolerance nor sustainable throughput. Increase load within the approved test envelope and observe when latency, errors, or task quality violates a gate. Report that boundary alongside the tested account and routing configuration. It is an experiment result, not a universal AWS capacity guarantee.

Price the accepted outcome

Retain Converse token usage and compare it with the applicable Amazon Bedrock pricing mode. The pricing page names Standard, Flex, Priority, and Reserved service tiers, plus batch inference; check which offerings apply to the selected model. Pricing also differs by model and input or output type. Do not multiply a single assumed token rate across every modality or purchase mode.

CloudWatch’s InputTokenCount and OutputTokenCount support aggregate usage analysis. Request-level usage helps explain why an apparently cheaper model costs more for the task: it might need longer instructions, produce longer outputs, or require a second attempt. Include retrieval, validation, and review in the workflow’s total cost when those components differ between candidates.

The decision record should report cost per accepted result, not only cost per request. If half the outputs need expensive manual repair, a low invocation price can conceal a poor business outcome. Detailed attribution belongs to 4.1.1; this lesson owns the selection comparison.

Key distinctions and trade-offs

Model latency versus workflow latency: Bedrock runtime metrics describe the model request boundary. Your application measurement includes retrieval, network overhead, retries, and validation.

Token throughput versus task throughput: Tokens describe generated volume. Accepted completions describe useful work. A verbose model can lead on the first measure and lose on the second.

Benchmark result versus service promise: Measured capacity applies to the documented experiment conditions. Changing account quotas, model versions, request lengths, or routing can change the result.

A larger model might justify its price by reducing rework on difficult cases. A smaller model might meet the same quality gate for a narrower task. Consider separate task routes only after measuring them; model-tier routing is developed in 4.1.3.

Architecture / decision scenario

A claims team compares two eligible models for drafting an explanation from supplied evidence. One produces attractive answers quickly on short demonstrations. The other costs more per token but follows the required evidence and output contract more consistently.

You run a held-out dataset with long evidence, conflicting documents, and unanswerable cases. Bedrock evaluations supplies reviewed quality results. The load harness records complete response latency, token usage, stop reasons, and throttles under the expected concurrency.

The lower-priced model needs frequent repair on disputed claims and exceeds the tail-latency gate after retries. The other passes the consequential slices and has lower cost per accepted draft. You select that configuration and preserve the rejected candidate’s evidence rather than describing the choice as a preference for a provider.

Common exam traps

Choosing from a public leaderboard without testing the application task.
Treating InvocationLatency as time to first token.
Comparing throughput without recording prompt and output lengths.
Ignoring retries when interpreting throttles and cost.
Using token price as the complete cost of a successful workflow.
Averaging critical quality failures into an attractive overall score.

What to remember

Freeze the task population and candidate configurations.
Use Bedrock evaluations for quality evidence and a load harness for operating behavior.
Measure first output, complete output, and accepted task throughput separately.
Retain usage, stop reasons, errors, and retry settings.
Optimize cost among configurations that pass the required quality gates.