NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.2 Compare agent performance across tasks and datasets.
Check task slices across repeated runs
Completed by the knowledge check
Learning objective
Compare agents across task types, difficulty slices, and user cohorts, accounting for run-to-run variance.
Core idea
An overall score answers a question about the dataset’s mixture of cases. It does not establish that an agent performs acceptably for every task or user group. A large, easy cohort can overwhelm a regression in a smaller, difficult cohort. Compare candidates within defined slices and then aggregate using weights that represent the intended deployment.
A second problem is variation between executions. One successful run on a difficult case does not make that case reliably solved. Keep the controls from 3.2.1, repeat execution, and distinguish performance across different cases from instability on repeated attempts at the same case.
How it works
Label the dimensions that change the decision
Start with three separate dimensions. A task type names the requested operation, such as retrieval, calculation, or a multi-step update. Difficulty describes the challenge within a task, such as one source versus conflicting sources. A user cohort groups the intended audience, such as language or permission tier. Do not define difficulty after seeing which candidate fails: that makes the comparison depend on its result.
Maintain a case registry containing the dataset id and team-defined fields such as task_type, difficulty, and cohort. These field names are your schema, not reserved toolkit keys. Keep reference outcomes separate from workflow input so that a success label cannot leak into the agent’s question.
NeMo Agent Toolkit exposes additional dataset fields to custom evaluators through EvalInputItem.full_dataset_entry. To select a cohort or task column, use eval.general.dataset.filter.allowlist.field with that column’s name and a list of accepted values; denylist.field excludes selected values. The column need not be an ID.
For nested data, NVIDIA’s custom-parser example uses eval.general.dataset._type: custom, a function naming the parser, and kwargs containing difficulty: "medium" and max_rows: 5. That parser extracts questions, filters their difficulty, and returns up to five rows. Those arguments belong to the example parser, not every dataset reader.
Prefer the complete run when comparing deployment-wide latency or load: running a small filtered slice changes the submitted workload. Filtered evaluations are useful for investigating a known cohort, but candidates must receive the same filter and execution settings. Preserve each slice’s case count as well as its score.
Compare within slices before combining them
Read the configured evaluator’s <evaluator-name>_output.json, which contains per-entry scores and an average. Join those entries to the registry by case identity. Treat a missing entry or unusable generated_answer in workflow_output.json as an unsuccessful execution for this comparison, retaining it in the declared denominator. With the profiler enabled, inspect per-request error flags in standardized_data_all.csv to investigate execution errors. A missing evaluator score needs separate classification.
Use task-specific success criteria from 3.1.1. Retrieval-answer scores and authorized-update success rates measure different outcomes; report each task’s candidate difference on its own scale before combining them.
Three summaries answer different questions:
| Summary | Deciding condition | Mechanism | Cost or limitation |
|---|---|---|---|
| Case-weighted aggregate | The dataset reflects the deployment mixture | Average all eligible case scores | Large groups dominate |
| Equal-slice aggregate | Every defined slice needs equal influence | Average the slice means | Rare slices can receive disproportionate deployment weight |
| Deployment-weighted aggregate | Your test sample deliberately oversamples hard or rare cases | Apply predeclared traffic weights to slice means | Requires defensible weights and revision tracking |
Predeclare important intersections, such as difficult calculations for restricted-permission users. Publish their counts and gather distinct cases when a decision rests on a tiny group.
Repeat the experiment without inflating coverage
The documented command nat eval --config_file=EVAL_CONFIG_PATH --reps=5 runs five repetitions. Replace EVAL_CONFIG_PATH with the evaluation configuration path. NVIDIA describes repetitions as a way to average scores and analyze variation in generated outputs. Five is an example, not a guarantee of sufficient statistical precision.
For each candidate and slice, report the number of distinct cases, repetitions, mean result, and spread across repetitions. Also inspect per-case consistency: a mean of 80% can describe stable success on four out of five cases or intermittent success on every case. Those behaviors imply different support burdens.
Do not describe 50 cases repeated five times as 250 independent user scenarios. Repetitions expose execution instability; additional distinct cases expand task coverage. If the two candidates differ by one percentage point while their slice results move by several points across repetitions, the comparison does not justify a confident winner. Predeclare a practical improvement threshold and collect more runs or cases according to the uncertainty you need to resolve. Any confidence interval added by your analysis must identify its sampling unit and method; an observed minimum-to-maximum range is not a confidence interval.
See 3.1.4 for offline rescoring with --skip_workflow. Rescoring fixed answers separates judge instability from agent execution variability; it does not add another agent run.
Compare datasets without merging their meanings
Run the same candidate configuration against each suite using nat eval --config_file=EVAL_CONFIG_PATH --dataset=DATASET_FILE. Replace DATASET_FILE with each JSON dataset’s path; --dataset overrides the configured dataset path. Report results side by side for the in-house suite and a second suite or dataset revision. Preserve each suite’s label definitions, case mix, and scoring rubric. Do not pool scores when “success” means different things; a lower score on a harder suite is not by itself an agent regression.
Key distinctions and trade-offs
Task variety, difficulty, and cohort membership are complementary views. A candidate can improve retrieval while degrading calculation, or improve English results while degrading another language. A single ranking is useful only after those losses have been judged acceptable by the product owner.
Repetitions consume workflow and judge calls, while broader datasets consume labeling effort too. Spend repeats on unstable cases and add distinct cases where coverage is weak. Keep the final comparison set separate from targeted debugging cases so that repeated investigation does not become unreported tuning against the selection dataset.
Architecture / decision scenario
An employee benefits assistant serves two language cohorts. In a fixed 200-case suite, 160 cases are in English and 40 are in French. Across five repetitions, the baseline averages 90% success in English and 85% in French; the challenger averages 94% and 70%. The case-weighted totals are 89% and 89.2%.
A team proposes the challenger because its aggregate is higher. The benefits product owner requires at least 80% in each language and in hard calculation cases. The suite distinguishes policy lookup from entitlement calculation and simple from conflicting-policy cases. Its 10 hard French calculation cases average 80% success for the baseline and 40% for the challenger; repetition ranges are 60–100% and 20–60%. That task × difficulty × cohort intersection confirms a severe regression despite variability.
The challenger’s overall repetition scores range from 88% to 91%, so its 0.2-point aggregate advantage is also weak compared with observed variation. The team retains the baseline: the French mean violates the cohort requirement, and the aggregate does not establish a meaningful gain. It expands the 10-case intersection before further selection trials and uses fixed-output rescoring to investigate judge instability separately. These are illustrative results and product thresholds, not vendor benchmarks.
Common exam traps
- Accepting an aggregate improvement that conceals a required cohort’s regression.
- Averaging unlike task metrics without defining their combined meaning.
- Defining hard cases from the challenger’s failures after the run.
- Treating repetitions as additional independent task coverage.
- Calling a run range a confidence interval.
- Treating offline rescoring as another execution of the agent.
Where the exam guide differs
The study guide’s suggested readings say “NVIDIA Agent Intelligence Toolkit.” NeMo Agent Toolkit documents the nat eval repetition and offline-scoring interfaces used here. Read that product name in an exam item as the toolkit family.
What to remember
Compare matched cases within task, difficulty, and cohort slices before accepting a combined score. Report slice sizes, intended weights, and repeat-run variation. More attempts at the same question reveal instability; they do not replace broader coverage or repair a cohort that misses its acceptance threshold.
Knowledge check
An IT help agent serves 300 standard users and 60 restricted-permission users in a fixed 360-case suite. The product owner requires at least 85% success in each cohort. Averaged over five repetitions, the baseline scores 88% and 87%; the challenger scores 93% and 72%, for a case-weighted total of 89.5% against 87.8%. What should the team decide?
A decision rests on 10 hard calculation cases for French users. To make that slice “large enough”, an engineer proposes running nat eval --reps=20 and reporting it as 200 scenarios. What expands the slice’s coverage?
--reps=20 and reporting the slice as 200 scenarios--skip_workflowOver five repetitions of the same suite, a challenger’s aggregate success ranges from 86% to 91% and the baseline’s from 85% to 90%. The challenger’s mean is 0.4 points higher, and the team predeclared a minimum useful improvement of two points. What should the team report?
The same candidate scores 88% on an in-house suite and 71% on a second suite whose labels and rubric define success differently. A manager wants one figure for the release note and asks whether the agent regressed. How should the team report the results?
After a comparison run, an engineer proposes labeling as “hard” the cases the challenger failed, so that the report can show hard-case performance for both candidates. Which basis should the difficulty label use instead?