Combine deterministic metrics with calibrated autoraters

Learning objective

Combine deterministic metrics with calibrated model-based autoraters so that an agent evaluation covers both exact requirements and qualitative criteria.

Core idea

Use hard gates plus a qualitative scorecard:

acceptable case = every required deterministic check passes
                  AND every required autorater threshold passes

A deterministic metric applies fixed code to recorded evidence. Given the same inputs and metric version, it returns the same result. It fits questions such as whether a tool name is allowed, arguments satisfy constraints, a required field exists, or a side effect occurred.

A model-based autorater, also called a judge model, interprets evidence by using natural-language criteria and a rating rubric. It fits qualities with several valid expressions, such as clarity, completeness, appropriate tone, or whether an explanation reflects the tool trace. Google Cloud distinguishes computation-based metrics from rubric-based metrics for this reason. The latter are suited to qualities that deterministic algorithms do not capture well. See the Gen AI evaluation service overview.

Do not collapse the two layers into one compensating average. A high clarity score must not cancel an unauthorized tool call. Report each layer separately, then apply explicit pass rules.

Build the combined evaluation

Route each criterion to the right scorer

Use deterministic code whenever the requirement can be expressed as an exact match, set membership, schema check, predicate, or observed postcondition. Use an autorater only when acceptable answers vary and semantic interpretation is necessary. Agent Platform supports both custom code metrics and custom large language model (LLM) metrics with natural-language rubrics, as described in Manage evaluation metrics.

Give the autorater all evidence needed for its criterion, but no irrelevant signals. For example, judging whether a final answer accurately reflects an agent’s actions requires the user request, final answer, and tool trace. Judging tone might require only the request and answer.

Make the rubric observable

Define one quality dimension at a time. State the criterion, allowed evidence, score scale, anchored description for every score, and behavior when evidence is insufficient. A label such as “good” is not an anchor. “Explains the refund decision using only the supplied policy and order facts” is testable.

Google Cloud’s model-based metric templates separate the judge instruction, criteria, rating rubric, evaluation steps, and input variables. They support both pointwise scoring of one response and pairwise preference between two responses. See Metric prompt templates for model-based evaluation.

Calibrate against expert judgment

Here, calibration means checking and adjusting the autorater so that its decisions match the intended domain judgment closely enough for the evaluation use case. It is not the judge’s own confidence score.

Use the following loop:

1. Have domain experts label representative, boundary, and failure cases by using the same rubric. Resolve ambiguous policy language before blaming the judge for human disagreement.
2. Tune the rubric, examples, judge configuration, and pass threshold on a calibration split.
3. Measure the frozen autorater on a separate held-out split. Inspect the confusion matrix and important slices, not only overall agreement.
4. Set an abstain or human-review path for missing evidence, borderline cases, and high-impact disagreements.
5. Version the rubric, judge model, configuration, dataset, and threshold with the agent evaluation result.

Google Cloud’s judge-evaluation workflow uses human ratings as ground truth and reports balanced accuracy, balanced F1, and a confusion matrix for pointwise or pairwise judges. These measures reveal false passes and false failures that a single average can hide. See Evaluate a judge model.

Key distinctions and trade-offs

Calibration versus evaluation: Calibration changes the evaluator. Final evaluation measures the agent with a frozen evaluator. Tuning and reporting on the same labeled cases overstates judge quality.

Pointwise versus pairwise: Pointwise scoring asks whether one result meets an absolute rubric, so it fits acceptance thresholds. Pairwise scoring asks which of two results is better, so it fits candidate-versus-baseline selection. A pairwise winner can still be unacceptable. Pairwise judges should also be tested with response order swapped because model judges can show positional bias; Google Research discusses this risk in Judging with Confidence.

Coverage versus certainty: Autoraters scale qualitative review, but their judgments remain model outputs. Human review is slower and more expensive, but it is appropriate for calibration, sampled audits, and high-impact ambiguous cases. Deterministic checks are repeatable, but overusing exact text matching rejects valid language variation.

Architecture / decision scenario

A retailer evaluates an agent that issues policy-compliant refunds and explains the decision to the customer.

Experts label examples that include partial refunds, tool failures, ambiguous requests, and correct answers with different wording. During calibration, the confusion matrix shows that the judge often passes replies that omit a required time-limit caveat. The team adds that caveat as a separate rubric criterion, provides the applicable policy text as evidence, and chooses a threshold from the cost of false passes and false failures. It then validates the frozen judge on held-out labels.

The release report keeps exact failures and qualitative scores in separate columns. A response with excellent tone still fails if the agent refunded the wrong amount. A policy-compliant refund can also fail the response-quality gate if its explanation makes an unsupported promise. Cases with missing policy evidence go to human review instead of receiving a guessed score.

A case is acceptable only when both layers pass, and either layer alone can fail it.

Common exam traps

• Using an LLM judge for a schema, exact value, or prohibited-action check that deterministic code can enforce.
• Treating an autorater’s rationale or confidence as proof that it agrees with domain experts.
• Tuning the judge and reporting its quality on the same human-labeled cases.
• Averaging hard-check failures with qualitative scores so that a critical violation can still pass.
• Treating a pairwise winner as proof that the winning response meets an absolute quality bar.
• Reporting only mean agreement and missing false-pass rates or weak slices.

What to remember

• Use deterministic metrics for exact requirements and autoraters for genuine semantic judgment.
• Calibrate autoraters against expert labels, then validate them on held-out cases.
• Keep hard gates and qualitative scores separate in reports and pass rules.
• Use pointwise judges for absolute thresholds and pairwise judges for relative preference.
• Route uncertain, missing-evidence, or high-impact cases to human review.

Knowledge check

References

• Google Cloud, Gen AI evaluation service overview
• Google Cloud, Manage evaluation metrics
• Google Cloud, Metric prompt templates for model-based evaluation
• Google Cloud, Evaluate a judge model
• Google Research, Judging with Confidence: Calibrating Autoraters to Preference Distributions