Calibrate judges before trusting their scores

Learning objective

Implement calibrated LLM-as-a-judge evaluation with explicit rubrics and bias checks.

Core idea

For an architect selecting a quality gate, an LLM judge is a measurement instrument that also needs evaluation. A numerical score does not establish that the judge applies your business standard consistently. Calibration compares its decisions with independently reviewed examples; bias checks test whether irrelevant presentation changes alter those decisions.

Make the rubric explicit enough that a reviewer can identify the evidence for each score. Keep deterministic checks for properties such as exact IDs and valid schemas. A fluent explanation from a judge cannot override a failed business constraint.

See 3.1.2 for cloud evaluation setup and input mappings, and 3.1.3 for independent regression gates. This lesson defines how to decide whether a semantic judge is reliable enough to contribute to those gates.

How it works

Encode the quality standard

Microsoft Foundry rubric evaluators score custom weighted dimensions on individual responses or multi-turn conversations. The evaluator catalog labels rubric evaluators as preview. Each dimension has an id, description, and weight; always_applicable controls whether the judge can skip it as irrelevant. Applicable dimensions receive scores from 1 to 5. The overall weighted result is normalized to a 0–1 range, with a documented default pass threshold of 0.5.

Write each description around observable behavior. “Good customer support” leaves the judge to invent the standard. “Identifies the requested account from the supplied evidence and asks for clarification when two accounts remain plausible” states the decision. Define anchors for low, intermediate, and high performance, including cases where a polished answer must fail.

Use always_applicable for a criterion that must be assessed on every response. Otherwise, inspect applicability decisions: a skipped dimension does not contribute to the aggregate score. A candidate can appear stronger because the judge stopped applying the difficult criterion, rather than because the response improved.

Weights express relative importance, not hard prohibitions. A weighted average can compensate for a weak dimension with strong scores elsewhere. Keep a separate deterministic rejection for a prohibited action or wrong entity ID. A higher weight alone does not make that failure impossible to pass.

Choose the evaluator interface deliberately

Foundry also documents custom prompt-based evaluators, a preview capability. In Python, project_client.beta.evaluators.create_version() registers the definition. Its prompt_text contains the judge instructions, data_schema declares expected inputs, metrics declares the scoring method and range, and init_parameters declares runtime configuration.

Template variables such as {{response}} map to evaluator input fields. A prompt-based evaluator requires deployment_name and threshold at initialization and returns JSON with result and reason. Its supported scoring methods are ordinal, continuous, and binary. A code-based custom evaluator instead requires deployment_name and pass_threshold; it still needs a deployment name for service orchestration even though it calls no LLM. Match the threshold parameter to the evaluator type, and do not infer that a custom score uses the rubric evaluator’s normalized scale.

Use an explicit judge model deployment and record the evaluator version with each run. A change to the model, instructions, threshold, input mapping, or rubric changes the measurement instrument. Recalibrate that instrument before treating its results as comparable with a previous baseline.

Calibrate on reviewed examples

Build a calibration set containing clear successes, clear failures, and borderline cases. Have qualified reviewers apply the written rubric without seeing candidate identities or judge scores. Resolve reviewer disagreement about the business standard before using majority labels as truth. Split calibration examples from the held-out cases used to confirm the resulting judge.

Foundry’s Human Evaluation templates support thumbs up/down, slider, multiple-choice, and free-form questions. Use questions that correspond to rubric dimensions rather than one overall satisfaction question. Download Results exports reviewer answers with the prompt, agent response, questions, and timestamp as CSV. See 3.1.1 for reviewer access and evidence handling.

Compare judge and reviewer pass/fail decisions, not only average scores. Count false passes and false failures separately. A false pass accepts an unacceptable response; a false failure rejects an acceptable one. Their costs can differ substantially. Inspect disagreements by task type and risk level before setting the acceptance threshold.

For example, your release policy might permit no false passes on the reviewed wrong-account cases. That is an application gate, not a Foundry guarantee or a universal sample-size rule. Adjust the descriptions or threshold on the calibration set, then check the held-out set without further tuning. If the judge fails there, retain human review for that decision.

The judge is tuned on the calibration set and confirmed on the held-out set without further tuning.

Probe bias with controlled variations

Create matched examples that preserve the task evidence while varying an irrelevant property. Compare concise and verbose correct responses, swap the order of candidates in pairwise comparisons, and remove model or vendor names. Where task semantics permit, vary names or demographic references without changing the required decision. A change in score should be explainable by the rubric, rather than by presentation alone.

These are application-designed experiments; the evaluator API does not turn them into automatic proof of fairness. Track disagreement rates for each variation and inspect the returned reason. Test repeated judgments too, because one favorable score can conceal instability near the pass threshold.

Keep candidate content separate from judge instructions. Include examples whose response text asks the judge to award full marks, and require the rubric to treat that text as evidence to assess. A judge vulnerable to instructions inside the assessed answer can become an optimization target instead of a quality gate. Escalate inconsistent or suspicious decisions for human review.

Key distinctions and trade-offs

Rubric versus calibration: A rubric defines the standard. Calibration measures whether the configured judge follows it on reviewed examples. Neither replaces the other.

Threshold versus quality: Raising a threshold changes the pass decision. It does not repair inaccurate labels, missing criteria, or presentation bias.

Judge reasons versus ground truth: Reasons help locate disagreement. They are generated output that reviewers must check against the evidence.

A richer rubric exposes more failure modes but adds inference cost and review work. A single aggregate score is convenient for ranking, while per-dimension results are better for deciding what failed. Use the aggregate for comparison only alongside noncompensating business and safety gates.

Architecture / decision scenario

A team compares two support agents. The candidate writes longer answers and receives higher judge scores, but human reviewers find that both versions sometimes select the wrong customer account.

The team creates rubric dimensions for subject resolution, evidence use, clarification, and communication. Subject resolution receives explicit anchors, and required dimensions use always_applicable. Exact customer IDs remain a separate deterministic check. Reviewers label calibration cases through matching Human Evaluation questions and export their answers.

The evaluation owner tests shortened and expanded versions of the same correct answer, blinded candidate names, swapped pairwise order, and equivalent-name cases. Verbosity still drives false passes, so the owner revises the rubric’s evidence requirements and creates a separate evaluator version. The held-out set then measures whether the revision reduced false passes without excessive false failures. Promotion depends on that evidence and the exact-ID gate, rather than the candidate’s higher average alone.

Common exam traps

• Calling a judge calibrated because its prompt contains a rubric.
• Treating a 1–5 dimension score as the normalized 0–1 overall result.
• Assuming a high weight creates an unconditional rejection rule.
• Counting skipped dimensions as passed dimensions.
• Tuning and validating on the same reviewed examples.
• Passing pass_threshold to a prompt-based evaluator, which requires threshold.
• Assuming judge explanations prove unbiased or accurate decisions.

What to remember

Specify the criterion, version the judge, compare its decisions with reviewed labels, and test irrelevant variations. Use held-out validation and independent hard checks before trusting a semantic score as a release gate.

Knowledge check

References

• Microsoft, Rubric evaluators
• Microsoft, Custom evaluators
• Microsoft, Built-in evaluators
• Microsoft, Human evaluation for Microsoft Foundry agents