Resolve disagreements before trusting model graders

Learning objective

Select and combine code-based, model-based, and human grading, calibrating model graders against human labels and adjudicating disagreements

Core idea

Code validators, independent review, and labeled calibration provide different checks. The Professional decision is how to combine their verdicts into a repeatable evaluation framework, measure where they disagree, and assign authority to resolve those disagreements.

Anthropic recommends deterministic grading where possible, model grading where needed, and human grading for validation. A task can have several graders, with weighted, binary, or hybrid scoring. Choose that composition before comparing candidates: an improved writing score must not compensate for an unauthorized action.

How it works

Assign one criterion to its strongest grader

For each success criterion, write the observable evidence, grading method, and consequence of failure. Use code for exact values, calculations, required fields, or a verifiable business-state condition. Use model grading for meaning that tolerates multiple valid phrasings. Reserve expert judgment for reference labels, unresolved interpretation, calibration, and sampled audits.

A code grader can implement the documentation’s output == golden_answer check, after explicitly justified normalization. A model grader is a separate inference call; Anthropic’s evaluation examples use client.messages.create() with a rubric and candidate answer. This does not make the verdict authoritative. Its reliability must be established against expert judgments for the actual criteria.

Record each criterion, grader revision, verdict, and adjudication status separately rather than reducing everything to one number during collection. That preserves the information needed to distinguish semantic disagreement from a broken parser.

Specify what the model is allowed to judge

Give the judge the task, authoritative reference material, candidate output, and an explicit rubric. Anthropic recommends clear rubrics and specific scores; its agent-evaluation guidance also recommends an uncertainty outcome, such as Unknown, when information is insufficient. In this design, Unknown is a rubric value, not an API stop reason.

Use isolated judgments for distinct dimensions when a broad overall rating hides which criterion failed. For example, factual support and clarity need different questions: a readable explanation can still contain an unsupported claim. Supply only the material needed to judge each criterion.

Anthropic recommends using a grader model with thinking on so it reasons before producing a score; configure the model-supported thinking behavior as described in 2.1.3. Its evaluation examples also recommend using a different model to evaluate the output from the one that generated it.

Use pairwise comparison to rank two candidate outputs; use absolute rubric-based scoring to decide whether one output meets a fixed acceptance bar. Reference-based evaluation compares an output with a supplied reference, while natural language assertions express individual criteria the judge checks.

Define borderline examples in the rubric. “Adequate explanation” leaves reviewers guessing. “Identifies the applicable exception and cites the supplied policy passage; fails if the exception is omitted” is assessable. A reference answer can illustrate correctness without requiring identical wording.

Calibrate the judge’s errors, not its confidence

Use subject-matter expert (SME) review to label candidate outputs without first showing reviewers the model verdicts. Include acceptable outputs, realistic failures, boundary cases, and every important task slice. Measure inter-annotator agreement and resolve human-human disagreements before using those labels as the comparison target.

Run the judge on the same frozen outputs, then calculate agreement per criterion and inspect the disagreements. Aggregate agreement alone can hide a dangerous asymmetry. If most outputs are correct, a judge that passes everything can appear accurate while missing every meaningful defect.

Record false passes, false failures, and uncertainty separately. Inspect whether a false pass comes from an incomplete rubric, missing source material, an invalid human label, or a model judgment error. Adjust only the defective part, version it, and confirm the result on labeled outputs not used to edit the rubric. Otherwise, you measure adaptation to known disagreements rather than general grading reliability.

This extends calibration from a confidence threshold to the grading process itself. The artifact is a calibration report naming the judge configuration, rubric revision, labeled sample, confusion counts, and acceptance rule. Anthropic calls for close calibration with human experts.

Adjudicate rather than average

Route model-human disagreement and Unknown to the named domain adjudicator with the source, criterion, and both judgments. The adjudicator records the accepted label and reason. A disputed label remains unresolved until that review completes; it must not silently become a pass or disappear from the denominator.

Expert labels and judge verdicts on the same frozen outputs meet when agreement is calculated, and model-human disagreement and Unknown go to the named domain adjudicator.

When code and a model disagree, examine what each claims. A source-state check can establish that a required record was never created even when a model approves the final explanation. Conversely, an exact-string grader may reject a valid alternative phrasing. Repair the faulty grader through review instead of letting majority vote decide factual truth.

After calibration, use model graders at scale with expert spot-check sampling. Revalidate when the rubric, judge configuration, or task population changes. Keep a route for an adjudicator to revise a reference label; human authority needs a documented reason, not immunity from correction.

Key distinctions and trade-offs

Weighted score versus critical requirement: Weighted scoring supports partial progress. A binary requirement preserves an obligation that cannot be traded away. A hybrid can use both, but report them separately.

Agreement versus false-pass risk: Agreement measures alignment over the sampled outputs. False-pass counts expose cases where the automated judge would approve an actual defect. Choose the acceptance rule from the consequence of that error.

Multi-judge consensus versus expert adjudication: Multiple judgments can reveal instability, but correlated model opinions do not resolve an ambiguous policy. An accountable specialist settles the interpretation and updates the rubric.

Architecture / decision scenario

A healthcare operations team evaluates administrative referral summaries, with an eight-hour expert-review budget per release. The clinical-operations owner requires every summary to preserve the requested department and distinguish missing information from confirmed facts.

The proposed shortcut is a single model rating for each summary. The architect instead uses code for required identifiers, separate model rubrics for source fidelity and clarity, and experts for calibration and unresolved cases. Source fidelity is mandatory; clarity contributes a secondary score.

On 100 expert-labeled outputs, 90 are acceptable and 10 contain unsupported claims. A candidate judge agrees on 94 overall, but passes 6 of the 10 defective outputs. Its 94% agreement conceals a 60% false-pass rate among actual defects. The operations owner rejects deployment of that judge as the sole fidelity check.

Experts adjudicate those six cases and find that the rubric accepts plausible additions absent from the referral. The team revises the criterion, confirms performance on untouched labeled outputs, and budgets expert time for every Unknown and sampled accepted summaries. If the review budget cannot support unresolved volume, the owner reduces automation scope rather than declaring those cases passed.

Common exam traps

• Treating a calibrated extraction confidence score as calibration of the judge.
• Using one holistic rating for unrelated success criteria.
• Averaging a failed critical requirement into an acceptable overall score.
• Reporting agreement without examining false passes and sample composition.
• Letting model-human disagreements vanish from the reported population.
• Editing a rubric on known failures and calling the same cases independent validation.

What to remember

Assign graders per criterion, preserve their separate results, and calibrate model judgments against adjudicated human labels. Investigate error direction, validate rubric changes independently, and give unresolved cases an explicit owner and status. Grading scale is useful only when the release decision remains defensible.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Demystifying evals for AI agents