Define quality gates before selecting scores

Learning objective

Define relevant GenAI quality metrics for factual accuracy, fluency, relevance, and output consistency.

Core idea

Quality is a contract between the application and the person relying on its output. For an architect defining release gates, four separate questions matter: is the answer true, is it readable, does it address the request, and does the same task produce dependable behavior? A single score cannot answer all four.

Define the acceptance rule before choosing the evaluator. A fluent answer with an invented refund deadline fails a support assistant even if its relevance score is high. Two differently worded answers can both pass if they preserve the same policy, required fields, and safe outcome. Evaluate whether the answer preserves the required facts and decisions, rather than whether it resembles one attractive example.

Amazon Bedrock provides named built-in metrics for LLM-as-a-judge evaluation and a mechanism for custom metrics. These scores are evidence for your acceptance policy. They do not create that policy or establish that a consequential claim is true.

How it works

Separate the four quality dimensions

For factual accuracy, identify the claims that change a decision and compare them with approved evidence. Bedrock’s Builtin.Correctness measures whether a response is correct. If you supply referenceResponse in the evaluation dataset, the evaluator considers it when scoring correctness. Builtin.Completeness separately measures whether the response answers every question in the prompt and also considers a supplied reference response.

A correct but incomplete answer can omit an exception that determines eligibility. Define a correctness gate for false claims and a completeness gate for missing required facts. Preserve the evidence revision used to approve the reference; an obsolete reference can reward an obsolete answer. Reference responses and production-distribution design belong to 5.1.2.

Bedrock’s Builtin.Faithfulness checks whether a response contains information not found in the prompt. This measures adherence to available context, not independent truth about the world. If you evaluate a retrieval-augmented answer, the evaluation input must include the evidence needed to assess it. A response can faithfully repeat a stale policy and still fail factual accuracy.

For fluency, define what a reader must understand without repair: grammatical sentences, clear references, coherent ordering, and appropriate terminology. Builtin.Coherence identifies logical gaps, inconsistencies, and contradictions within a response. Builtin.ProfessionalStyleAndTone measures suitability of style, formatting, and tone for a professional setting. These mechanisms cover useful parts of fluency, but neither name means that every language-specific grammatical requirement is checked.

Use a custom metric when the product requires a narrower fluency rubric, such as readable instructions in a particular language. Bedrock custom judge metrics accept detailed evaluator instructions and an optional rating scale. The instructions should distinguish a minor editorial defect from an ambiguity that changes the action a reader takes.

For relevance, use Builtin.Relevance to assess how closely the answer addresses the prompt. Pair it with completeness when the request has several parts. An answer can be entirely on topic while answering only the first question. Builtin.FollowingInstructions assesses compliance with explicit directions, so it can help identify an answer that ignores the requested format despite discussing the right subject.

For output consistency, distinguish contradictions inside one answer from variation across runs. Builtin.Coherence addresses the former. It does not establish that repeated generations preserve the same conclusion. Run a fixed case multiple times, retain the outputs, and apply application-defined checks for stable decisions, required fields, and evidence-supported facts. For Converse calls, fix and record the supported inferenceConfig.temperature and inferenceConfig.topP settings with the repetition policy. temperature ranges from 0 to 1; a lower value makes higher-probability choices more likely. Changing sampling settings changes the consistency measurement; see 4.2.6 for tuning. Measure the fraction of repeated cases that satisfy the same acceptance contract.

Make metric interpretation explicit

For each dimension, record the evaluated unit, evidence, rubric, failure severity, and release decision. The following mapping keeps the measurements separate:

Define hard failures separately from graded quality. For example, a fabricated deadline can block release, while a minor wording defect contributes to a fluency distribution. Do not average a safety-critical factual error away with three strong presentation scores.

Bedrock permits up to 10 custom metrics per judge evaluation job. Its custom metric configuration names the metric, instructions, rating scale, and evaluator. The rating scale must agree with the instructions; otherwise, identical labels can represent different judgments. Record the judge model, metric configuration, and rubric revision with every score.

Key distinctions and trade-offs

Correctness versus faithfulness: Correctness concerns whether an answer is correct; faithfulness concerns support in the supplied prompt. Evaluate both when source quality and unsupported additions can fail independently.

Fluency versus usefulness: Elegant prose can be irrelevant, incomplete, or false. Fluency matters when it makes correct guidance usable, but cannot compensate for a defective recommendation.

Consistency versus identical text: Exact text equality is appropriate for a fixed identifier, not for every natural-language explanation. Define invariants at the decision and contract level so legitimate paraphrases do not become regressions.

Metric versus gate: A metric produces evidence on a defined scale. A gate converts evidence into accept, revise, or reject using the product’s error costs. The team owns this conversion, including exceptions and review rules.

Architecture / decision scenario

A support assistant explains warranty coverage and returns a recommended next step. The team defines an approved reference for each product version, a fluency rubric for customer-facing English, required answers to coverage and claim-submission questions, and a stable eligibility decision across repeated generations.

Candidate A has stronger Builtin.Relevance and polished wording, but one repeated output invents a coverage exception. Candidate B uses simpler prose and consistently preserves the approved policy. The team rejects A on the factual gate, checks B’s fluency rubric, and verifies B’s completeness before approval. The decision uses all four dimensions rather than the highest average.

Common exam traps

• Equating Builtin.Faithfulness with independent factual verification.
• Treating Builtin.Coherence as a repeated-run consistency measure.
• Rewarding readable prose that omits a required answer.
• Requiring identical wording when the contract requires identical decisions.
• Combining unrelated scales without preserving hard failure gates.
• Assuming an optional reference response cannot change the judge’s assessment.

What to remember

• Define four separate quality contracts before choosing scores.
• Preserve approved evidence for factual and completeness judgments.
• Use custom metrics for requirements the built-in rubric does not cover.
• Evaluate consistency through repeated behavior, not prose identity alone.
• Keep critical factual failures outside compensating averages.

Knowledge check

References

• AWS, Use metrics to understand model performance
• AWS, Create a prompt dataset for a model evaluation job that uses a model as judge
• AWS, Create a prompt for a custom metric
• AWS, Create a model evaluation job using custom metrics
• AWS, InferenceConfiguration