Compare models through accepted business outcomes

Learning objective

Compare models with Amazon Bedrock Model Evaluations and quality, token, latency, and business outcomes.

Core idea

A model comparison is a constrained product decision. For an architect selecting a production candidate, the winner is the model that delivers acceptable business outcomes within the operating envelope. The highest quality score, fewest tokens, or shortest inference time alone does not identify that winner.

Amazon Bedrock evaluation jobs measure selected aspects of model responses. Your application must supply the surrounding evidence: the full request path, usage across attempts, accepted outcomes, and the costs of correction or escalation. Join these measurements by candidate and case so you can explain the trade-off instead of presenting four unrelated dashboards.

Use the benchmark contract in 1.2.2 and the representative dataset in 5.1.2. This lesson adds the evaluation-job boundaries and the decision that combines quality with operational and business evidence.

How it works

Choose the evaluation mode deliberately

In an automatic evaluation job, you choose one task type, which determines the available metrics and built-in datasets. Builtin.Accuracy has task-specific computed semantics; see 5.1.4.

An LLM-as-a-judge job uses a generator and an evaluator. The evaluator scores prompt-response pairs using selected built-in or custom metrics and supplies explanations. For example, choose Builtin.Correctness, Builtin.Completeness, and Builtin.Relevance when those dimensions match the contract. Keep the evaluator and rubric fixed across candidates. Each built-in metric uses a different evaluator prompt, so selecting a different metric set changes the questions used to judge the response. Record both the evaluator model and selected metric names before comparing candidate scores.

Bedrock supports evaluating supplied inference responses and skips generation on that path. This is useful for comparing outputs from a complete application, including models outside Bedrock. Judge datasets support one response per prompt and one unique modelIdentifier per job. Run aligned candidate jobs rather than pretending that a single judge job performs an arbitrary multi-model tournament.

Human evaluation can compare up to two inference sources, including Bedrock-generated and supplied responses. Use it for calibrated preference or domain review when the deciding question requires human interpretation. Evaluator limitations belong to 5.1.4; preference alone still does not establish that a response meets a factual gate.

Bedrock’s console summarizes judge scores in a histogram and displays score explanations for the first five dataset prompts. The full report is available in the job’s S3 destination. Inspect that report for matched-case regressions and category differences rather than approving a candidate from the short console sample.

The CreateEvaluationJob operation records the service role, evaluationConfig, inferenceConfig, and outputDataConfig. Retain that configuration with the result. A changed task, reference, inference setting, or evaluator creates a different comparison even if the candidate model name remains the same.

Join quality to usage and latency

For calls through Converse, retain usage.inputTokens, usage.outputTokens, usage.totalTokens, and metrics.latencyMs with the candidate and request identity. These are response fields for an invocation, not complete end-to-end application economics. Track retries, retrieval, tool execution, and follow-up calls in your own attempt records.

Use the latency definitions and endpoint boundaries in 1.2.2 and the runtime metric accounting in 4.3.3 when joining operational evidence to evaluation results.

Measure both service latency and the product’s elapsed time from accepted request to usable outcome. Compare distributions rather than averages alone. A model that produces long explanations can have an acceptable median and a poor tail for long-context cases. A model that responds quickly but repeatedly requires clarification can make the completed workflow slower.

Token counts explain consumption but do not themselves determine a bill. Use the applicable pricing terms for the selected model and inference path when calculating cost. Include repeated attempts and downstream work in the cost-per-accepted-outcome calculation; detailed attribution belongs to 4.1.1. Avoid assigning a fixed price to a generic token independent of model and usage category.

Define business evidence independently

Choose a business outcome that the workflow can verify: a correctly resolved ticket, an accepted document extraction, or a completed task without human correction. Record the acceptance decision separately from whether the model returned text. An eloquent answer can be a completed invocation and a failed business result.

For offline comparisons, reviewed task acceptance is a proxy for production success. Label it as such. After offline gates pass, a controlled release can measure real completion, correction, escalation, and repeat-contact rates; see 5.1.5. Application-defined outcome metric publication is covered in 4.3.4; do not assume Bedrock emits a built-in ticket-resolution metric.

Compare candidates on the same cases, preserving failed requests and abstentions in the accounting. Report quality by category, usage per attempt and completed task, latency tails, and acceptance rate. Inspect cases where candidate rankings differ. A small gain in generic helpfulness can conceal expensive failures concentrated in one business-critical slice.

Key distinctions and trade-offs

Job score versus application outcome: Evaluation measures responses under a defined test contract. Business success measures the workflow’s accepted result. Connect them empirically; do not declare them interchangeable.

Score distribution versus average: A favorable mean can hide a cluster of poor responses. Inspect the judge-score histogram and matched cases, especially where the failures affect a mandatory business gate.

Tokens per invocation versus cost per accepted task: A compact model can require more attempts or corrections. Count those attempts before concluding that fewer output tokens mean better economics.

Aggregate winner versus eligible candidate: Apply mandatory quality and latency constraints first. Among candidates that pass, compare business value and operating cost. A failed critical gate cannot be offset by cheap inference. Evidence joined by candidate and case meets the mandatory constraints before any value and cost comparison.

Architecture / decision scenario

A document assistant compares two models on aligned, reviewed cases. Candidate A produces stronger relevance scores but emits longer answers. Candidate B uses fewer tokens and returns faster, yet misses a required exception on complex documents.

The team retains judge-job scores and explanations, joins them to Converse usage and latency records, and measures end-to-end accepted extraction results. B fails the exception gate despite its token and latency advantage. A passes, but its long-document latency tail exceeds the product deadline.

The decision is to revise the application contract and rerun the comparison before release. Neither candidate wins by averaging away a failed constraint. A subsequent candidate must pass quality, meet the deadline, and demonstrate acceptable cost per reviewed business result before entering a controlled production test.

Common exam traps

• Selecting a built-in benchmark without checking the business cases it represents.
• Assuming a judge job accepts many candidate identifiers in one dataset.
• Comparing judge scores from jobs that used different evaluator models or metric sets.
• Comparing token counts without retries and accepted outcomes.
• Assuming Bedrock supplies application-specific business metrics.
• Selecting a cheap candidate that fails a mandatory quality gate.

What to remember

• Freeze cases, evaluator settings, and candidate configurations.
• Use aligned jobs when the evaluation mode limits candidate sources.
• Join quality scores to per-invocation usage and application latency.
• Measure business acceptance independently from text generation.
• Choose among candidates that satisfy the operating constraints.

Knowledge check

References

• AWS, Model evaluation task types in Amazon Bedrock
• AWS, Evaluate model performance using another LLM as a judge
• AWS, Create a prompt dataset for a model evaluation job that uses a model as judge
• AWS, Create a human-based model evaluation job
• AWS, CreateEvaluationJob
• AWS, Converse
• AWS, Starting an automatic model evaluation job in Amazon Bedrock