NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.1 Implement evaluation pipelines and task benchmarks to measure performance.
Calibrate judges against human task labels
Completed by the knowledge check
Learning objective
Apply LLM-as-a-judge rubrics and calibrate judges against human labels.
Core idea
An LLM judge is a measurement component with its own failure modes. It scores the input and output presented to it according to a rubric; a fluent explanation does not establish that the score matches the business requirement. Calibrate the judge by comparing its decisions with adjudicated human labels before relying on it to score a larger benchmark.
Dataset construction belongs to 3.1.1. Here, the question is whether your scoring implementation recognizes the outcomes those labels define. Keep the judge’s rubric, model, examples, and decision threshold identifiable so that changing the measurement does not look like an improvement in the agent.
How it works
Turn the rubric into a scoring interface
Separate dimensions that have different consequences. A meeting summary might preserve all decisions but assign an action to the wrong employee. Scoring “overall quality” on one vague scale makes that error easy to overlook. Define which facts must appear, which transformations are allowed, and which errors force a failing label.
NeMo Agent Toolkit provides _type: langsmith_judge under eval.evaluators. Its prompt accepts a prebuilt OpenEvals prompt name, such as correctness, or a custom template with fields including {inputs}, {outputs}, and {reference_outputs}. llm_name selects the judge configured under llms, and feedback_key names the resulting metric. NVIDIA documents this integration as requiring the nvidia-nat[langchain] package extra; the full nat eval runtime also needs the evaluation installation described in 3.1.4.
For independent dimension scores, configure one langsmith_judge evaluator per dimension, each with its own feedback_key; each evaluator adds a judge call per case before retries. Alternatively, output_schema names a TypedDict or Pydantic model by Python dotted path for a combined structured judgment, and score_field selects the score path to report. A combined judgment uses one call before retries but requires a larger structured response; it does not emit every schema field as a separate metric.
Make the score domain explicit. choices can restrict scores to specified values; continuous: true instead permits a score between zero and one. These options are mutually exclusive. Use discrete labels when the outcome is a decision, and a continuous score only when intermediate values have a defined interpretation. A value of 0.8 is not an 80% probability that the answer is correct merely because it lies between zero and one.
The judge model must support structured output in JSON schema mode. NVIDIA identifies parsing errors as the consequence when it does not. Resolve that incompatibility by selecting a compatible judge model or correcting the scoring interface; do not interpret an unparseable judge response as a legitimate zero-quality agent answer.
Calibrate with reviewed examples
Start with human-scored input/output pairs that include clear passes, clear failures, and borderline cases. Have reviewers apply the same written rubric, reconcile disagreements, and preserve the adjudicated label. LangSmith’s annotation queues support prescribed rubrics and multiple reviewers; these are useful when you need a review process rather than an informal spreadsheet of opinions.
The toolkit’s few_shot_examples parameter supplies calibration examples to langsmith_judge. Each example contains inputs, outputs, and score, with optional reasoning. Select examples that explain the rubric’s dividing line, such as a faithful paraphrase that passes and an invented deadline that fails. Keep a separate set of human-labeled cases for checking judge agreement after tuning. Reusing the demonstrations as the entire validation set tests familiarity with the demonstrations.
For a binary acceptance rule, produce a confusion table: judge pass or fail against human pass or fail. Count false accepts separately from false rejects. A false accept allows an inadequate result to pass the evaluator; a false reject causes unnecessary investigation or blocks an acceptable result. The task owner decides their relative costs, and the evaluation owner implements the associated threshold and review rule.
Inspect disagreements by rubric dimension and case category. If the judge repeatedly accepts invented action owners, add explicit attribution requirements and suitable examples, then rerun the untouched validation set. If humans disagree on the same cases, clarify the rubric first. Tuning the model to imitate inconsistent labels cannot establish a dependable measurement.
The judge is tuned with few_shot_examples, checked against a separate set of human-labeled cases, and a fix is followed by a rerun of that untouched set.
Keep scoring failures visible
Retain evaluator outputs with per-case scores and explanations. NeMo Agent Toolkit writes <evaluator-name>_output.json for configured evaluators, so you can inspect whether a high aggregate score hides one recurring misclassification. use_reasoning, enabled by default, controls the judge reasoning available for disagreement analysis and adds output-token cost. The explanation helps locate a mismatch; the adjudicated task label remains the comparison target.
Judge calls inherit RetryMixin settings: do_auto_retry defaults to true, while num_retries and retry_on_status_codes bound retries for eligible transient failures. Those failures consume retry attempts rather than becoming rubric scores; account for the additional calls in the nightly budget.
Judge calls add model latency and token cost beyond generating the agent answer. Repeated judging can reveal unstable classifications, but also increases that cost. Preserve the generated answer when studying judge variation so that changes in the agent’s answer do not become a second uncontrolled variable. The harness modes that separate generation from scoring are covered in 3.1.4.
Key distinctions and trade-offs
Choose the evaluator by the criterion you need to enforce:
| Approach | Deciding condition | Mechanism | Trade-off |
|---|---|---|---|
| Deterministic scoring | Exact values or structure decide correctness | Code evaluator; toolkit langsmith with exact_match where appropriate | Cheap and repeatable, but exact text matching rejects acceptable paraphrases |
| Reference-based judge | Semantic agreement with a known outcome matters | langsmith_judge prompt using {reference_outputs} | Flexible wording; depends on reference quality and judge calibration |
| Reference-free judge | A rubric can be applied without a reference answer | langsmith_judge custom prompt | Useful for properties such as clarity; cannot replace missing task truth |
| Human review | Consequences or ambiguity exceed validated judge reliability | Rubric-based annotation queue and adjudication | Expert effort and slower throughput |
NVIDIA also provides tunable_rag_evaluator, whose judge_llm_prompt can be customized. Its default dimensions are coverage, correctness, and relevance; default_score_weights changes their weights. Its dataset answer describes the expected content. Select this interface when that expected-answer-description model fits the task. Changing weights changes the definition of success, so validate the resulting decisions against human labels too.
Architecture / decision scenario
A consulting firm generates meeting summaries for account teams. Its critical rule is that every named action owner and deadline must be supported by the transcript. Reviewers adjudicate 200 summary/transcript pairs, of which 40 contain unsupported assignments. These are illustrative calibration results for this deployment.
The team proposes accepting summaries whenever a general helpfulness judge scores at least 0.8. That judge agrees with humans on 180 of 200 cases, but accepts 12 of the 40 unsupported summaries. The 90% overall agreement conceals a 30% false-accept rate among the known failures. The delivery owner permits at most two such false accepts on this validation set.
The evaluation engineer replaces the vague prompt with an attribution rubric and adds few_shot_examples drawn from a separate demonstration pool outside the 200 validation pairs, covering invented owners and deadlines. The engineer validates on the held-out pairs, reports the confusion table, and keeps human review for disputed attribution. The delivery owner approves the acceptance rule; the engineer owns judge configuration and measured agreement. Buying a larger judge alone is the tempting alternative, but it does not resolve the unspecified rubric or demonstrate compliance with the two-error limit.
Common exam traps
- Treating judge confidence or explanation length as measured agreement.
- Reporting overall agreement while hiding false accepts on critical failures.
- Using calibration demonstrations as the entire validation set.
- Setting both
choicesandcontinuousforlangsmith_judge. - Counting judge parsing failures as valid task scores.
Where the exam guide differs
The study guide’s readings use “NVIDIA Agent Intelligence Toolkit.” The documented evaluation interfaces belong to NeMo Agent Toolkit. Recognize the earlier product name in an exam item, but identify the actual judge through its evaluator type and configuration, such as langsmith_judge, prompt, and llm_name.
What to remember
A judge needs a task rubric, a supported scoring interface, and validation against adjudicated human outcomes. Examine the errors that matter to the business, not only aggregate agreement. Changing the rubric or judge changes the measuring instrument and requires checking its behavior again.
Knowledge check
A legal team’s judge must fail contract summaries that omit a termination clause. On 250 adjudicated pairs the judge agrees with the reviewers on 92% of cases, and the contract owner allows at most three accepted omissions on this validation set. Which evidence decides whether the judge meets that rule?
To reduce false accepts, an engineer adds 20 failing cases from the validation set as few_shot_examples for langsmith_judge, reruns the same validation set, and reports zero false accepts. What should happen before the judge is trusted?
continuous: true to expose borderline scoresWhile inspecting judge disagreements on meeting summaries, an engineer finds 15 cases where the two human reviewers themselves split on whether a paraphrased deadline counts as supported by the transcript. What should the engineer do before tuning the judge on these cases?
After switching langsmith_judge’s llm_name to a new model, a nightly run shows many cases with parsing errors, which the dashboard records as a score of zero. The new model does not support structured output in JSON schema mode. What should the team do?
llm_name at a judge that supports JSON schema modenum_retries for the judge’s calls that failed to parsecontinuous: true so the judge returns a plain numbercontinuous changes the score domain, not the model’s structured-output support.