NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.1 Implement evaluation pipelines and task benchmarks to measure performance.
Turn task requirements into benchmark cases
Completed by the knowledge check
Learning objective
Define task-specific success criteria and build labeled or synthetic datasets with hard cases.
Core idea
A benchmark measures whether an agent completes a defined business task under defined conditions. An answer that sounds helpful can still fail because it cites an inapplicable policy, omits a required exception, or acts when it should request clarification. Start with the observable outcome, then choose the labels and evaluator that distinguish success from those failures.
The benchmark’s central artifact is a case specification: an input, the information available to the agent, an expected outcome, and a rule for scoring it. LangSmith calls these evaluation examples and separates inputs, reference outputs, and metadata. NeMo Agent Toolkit represents a structured dataset using id, question, and answer. A reference answer belongs to evaluation, not to the agent’s input.
How it works
Write success criteria before collecting examples
Translate the task into independently checkable requirements. For a policy assistant, these might be selecting the applicable policy version, answering all required clauses, and declining to infer a missing eligibility fact. A latency requirement belongs beside these quality requirements, but a fast wrong answer still fails the task.
Choose which requirements are mandatory. Use deterministic checks where the result is exact, such as a classification label or required output structure. Use a semantic rubric where several phrasings can satisfy the same outcome. LangSmith documents code evaluators for exact or structural checks and LLM judges for prompt-defined criteria. Judge calibration belongs to 3.1.2; this Bite defines what gets labeled.
Write the rubric as an annotation instruction with positive examples, counterexamples, and the treatment of missing information. If two subject-matter reviewers cannot agree whether a response succeeds, revise the criterion before expanding the dataset. Their disagreement might reveal an ambiguous business rule rather than an agent defect.
Preserve the case’s reference and context
Assign every case a stable identifier and an expected result. For structured NeMo Agent Toolkit datasets, eval.general.dataset._type selects the reader and file_path selects the file. Supported readers include json, jsonl, csv, xls, and parquet. Use eval.general.dataset.structure to map different source-column names onto id, question, and answer.
Keep case metadata such as policy revision, difficulty, provenance, and expected clarification behavior with the record. NVIDIA documents EvalInputItem.full_dataset_entry as the route through which custom evaluators access additional dataset fields. These fields can support scoring without becoming workflow inputs. This separation matters when a label describes a hidden requirement that the agent must discover from the supplied documents.
A reference need not be one canonical sentence. The toolkit’s tunable_rag_evaluator accepts an answer field describing what the generated answer must contain. That is useful for answers with several acceptable formulations. Keep exact expected values for arithmetic, identifiers, or categorical decisions; replacing them with a broad semantic description makes avoidable errors harder to detect.
Combine observed and synthetic cases deliberately
Human-labeled cases drawn from reviewed examples reveal actual user phrasing and operational ambiguity. Synthetic cases expand coverage when you lack enough examples of a particular condition. Neither source makes the labels correct automatically: reviewers must confirm the expected outcome against the authoritative material.
Ragas documents TestsetGenerator with generate_with_langchain_docs(docs, testset_size=...) for generating cases from documents. Its generate(..., query_distribution=...) interface controls the mixture of query synthesizers. Generation consumes model and embedding calls; a requested testset_size controls quantity, not representativeness or label validity. Exporting with to_pandas() lets you inspect the candidates and select cases for the final benchmark.
Design hard cases around a specific failure opportunity: near-identical policy names with different eligibility rules, an exception several sections away, a question with insufficient information, or a document containing a plausible but superseded answer. A longer prompt is not inherently a harder case. Record which requirement each case challenges so that adding many easy paraphrases does not masquerade as broader coverage.
Key distinctions and trade-offs
Choose the case source by the gap it addresses:
| Case source | Deciding condition | Mechanism | Cost or limitation |
|---|---|---|---|
| Human-labeled examples | You need realistic phrasing and expert interpretation | Reviewed input/reference pairs with metadata | Annotation time; observed examples can miss rare failures |
| Synthetic examples | You need controlled coverage of a missing condition | Ragas TestsetGenerator, testset_size, query_distribution | Generation and validation cost; plausible labels can be wrong |
| Curated hard cases | A consequential failure is rare in ordinary traffic | Explicit case requirement and reviewed expected outcome | Oversampling changes the meaning of an aggregate success rate |
Hard cases are a coverage category, so either human-authored or synthetic examples can supply them. In LangSmith, use dataset splits for these high-level evaluation groups and metadata for per-example provenance. In NeMo Agent Toolkit, store a category column and select its values through eval.general.dataset.filter.allowlist.field.category; run the representative and challenge sets separately so their scores are never averaged together. Passing a challenge set demonstrates the tested capability; it does not estimate how often ordinary production requests succeed.
Protect the distinction between task input and scoring information. Putting the reference answer into question makes the task easier than production. Conversely, hiding necessary policy context from the agent while keeping it in the reference creates an impossible test. Inspect both sides before interpreting a low score.
Architecture / decision scenario
An insurance company is building an employee assistant for benefits eligibility. The business owner requires at least 95% task success on 400 representative cases and zero invented eligibility decisions on a separate 80-case missing-information set. These are proposed acceptance criteria for this deployment, not product defaults.
The team proposes generating 5,000 easy questions from the policy handbook and accepting a high average score. The evaluation lead rejects that shortcut: the disputed failures concern dependent status and policy exceptions, not handbook vocabulary. Instead, benefits specialists label the representative cases and approve synthetic additions that target missing dependent data and conflicting policy revisions.
Each NeMo Agent Toolkit record has an id, question, and answer, with case category and policy revision retained as additional metadata. The 80-case set runs separately with eval.general.dataset.filter.allowlist.field.category selecting the value missing_information. For those cases, the expected outcome requires the precise clarification rather than an eligibility guess. The evaluator reads the extra fields through full_dataset_entry; the workflow receives only the intended input.
If all 400 representative cases pass but six missing-information cases invent eligibility, the assistant fails the stated acceptance rule. The benefits owner owns the rule and label adjudication; the evaluation lead owns dataset composition and scoring implementation. More synthetic volume cannot compensate for the failed mandatory behavior.
Common exam traps
- Choosing a scoring metric before defining successful task completion.
- Treating synthetic answers as validated ground truth.
- Passing reference outputs into the agent as task inputs.
- Calling a benchmark representative because it contains many examples.
- Hiding a failed mandatory requirement inside a high average score.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit.” NVIDIA documents evaluation under NeMo Agent Toolkit, with nat eval, eval.general.dataset, and eval.evaluators. Read the guide’s toolkit name as the earlier name when identifying the evaluation pipeline; use the documented NeMo Agent Toolkit interfaces when implementing it.
What to remember
Define the business outcome, label the expected behavior, and preserve the information needed to score each case. Use reviewed synthetic examples to fill coverage gaps, not to replace label validation. A benchmark should expose the failure that matters even when ordinary cases score well.
Knowledge check
A benefits assistant must reach at least 95% success on 400 representative cases and make zero invented eligibility decisions on 80 curated missing-information cases. The team’s draft report averages all 480 cases into one success rate and presents it as the expected production rate. How should the benchmark be reported?
eval.general.dataset.filter.allowlist.field . Any average across the two sets, however its threshold is tuned, can let invented decisions that fail the mandatory zero rule disappear into a high number, and dropping or diluting the curated cases removes the very failure the benchmark exists to expose.A NeMo Agent Toolkit dataset holds id, question and answer for each case. A custom evaluator also needs each case’s policy revision and whether a clarification is expected, while the workflow must receive only the employee’s question. Where should the two extra fields live?
full_dataset_entryquestion that the agent is told to ignoreEvalInputItem.full_dataset_entry as the route by which a custom evaluator reaches dataset fields the workflow does not consume, so scoring information stays out of the task input. Appending labels to question or the system prompt hands the agent hints it will not have in production, and joining by row position breaks as soon as a filter or reordering changes the rows; the case id is the join key.A team generates 2,000 candidate cases from its policy handbook with Ragas TestsetGenerator and proposes using the generated reference answers directly as the expected outcomes of a release benchmark. What must happen before those cases can gate a release?
testset_size so that a few wrong labels carry less weightquery_distribution toward the harder query synthesizerstestset_size nor query_distribution makes a label correct: one controls quantity and the other the mixture of query types. A judge scoring the references is another model’s opinion, not the authoritative material, so reviewers who own the policy must confirm each expected outcome before the case decides anything.The disputed failures of an eligibility assistant are answers that miss an exception stated several sections away from the main rule. The team already has 400 reviewed representative cases. Which addition most directly extends the benchmark’s coverage of this failure?