NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.2 Compare agent performance across tasks and datasets.
Isolate changes before comparing agent results
Completed by the knowledge check
Learning objective
Design controlled comparisons with fixed datasets and configurations.
Core idea
A comparison supports a decision only when you can explain what changed between its candidates. If you change the agent, the test questions, and the judge together, a higher score cannot tell you which change helped. Keep the evaluation conditions fixed and vary the factor the decision concerns.
See 3.1.4 for assembling a repeatable evaluation harness and 2.6.2 for comparing action-selection strategies. Here, the design artifact is a comparison specification: the candidate difference, matched case IDs, fixed dependencies, scoring configuration, and acceptance rule. The harness executes that specification; it does not decide whether the experiment isolates the right variable.
How it works
Define the comparison before running it
Write a concrete question such as, “Does replacing the planning model improve completed purchase-order investigations without increasing unauthorized tool use?” State whether you are comparing one component or an entire deployable configuration. Changing a model and its prompt together can be a legitimate system comparison, but it cannot attribute the result to the model alone.
For each candidate, preserve the same test cases and expected outcomes. In NeMo Agent Toolkit, eval.general.dataset selects the dataset; _type selects its reader and file_path selects the file. For structured records, the documented default columns are id, question, and answer. Use eval.general.dataset.structure when your dataset uses different column names. The case identifier is the join key for comparing outputs, not the row’s position in a summary table.
Consume the approved dataset revision from 3.1.1 and the versioned run inputs from 3.1.4. Confirm that both candidates select the same cases and filters. A shared filename does not establish that the contents stayed fixed between candidates.
Fix both execution and measurement
Run the candidates through nat eval --config_file=EVAL_CONFIG_PATH, replacing EVAL_CONFIG_PATH with the candidate’s evaluation configuration. The full configuration-driven evaluation runtime requires the nvidia-nat[eval] installation extra. Keep the candidate configurations identical except for the declared experimental change.
The controls extend beyond the workflow’s YAML. Preserve tool permissions, tool implementations, retrieval corpus and index revisions, starting conversation state, and external response fixtures where the test requires fixed answers. If live external data must remain live, record that limitation and interleave candidate runs over the same time window. Treat unexplained changes in upstream responses as a reason to repeat the comparison, not as model improvement.
Keep eval.evaluators and each judge’s llm_name fixed. NVIDIA’s Ragas evaluator configuration references a judge defined under llms; changing that judge changes the measuring instrument. Calibration against human labels belongs to 3.1.2. For this experiment, preserve the calibrated rubric, judge configuration, and scoring version across both candidates.
Set eval.general.max_concurrency explicitly. Different parallelism changes endpoint pressure and can change observed latency or failure rates. The evaluation documentation identifies HTTP 429 rate limiting as a reason to reduce concurrency, including an override to one concurrent request. If throttling invalidates a comparison, rerun both candidates under the same revised setting. Reducing it only for the challenger changes the experiment.
Audit what actually ran
See 2.1.4 and 3.1.4 for configuration artifacts and versioned run records. For this comparison, diff the candidates’ config_effective.yml files and confirm that only the declared change differs. For remote evaluation, retain the server’s deployed revision and configuration per candidate as well.
The comparison owner and product owner should approve the comparison specification before results arrive. The former confirms that the controls isolate the proposed change; the latter confirms that the acceptance rule answers the deployment question. If a prompt requires a particular model interface, comparing independently deployable bundles may be more useful than forcing both models through one incompatible prompt. Declare that bundle comparison explicitly, retain each bundle’s revisions, and avoid claiming that its score identifies the contribution of any one component.
Inspect workflow_output.json alongside each configured evaluator’s <evaluator-name>_output.json. Compare the same case IDs and reconcile missing entries or unusable generated_answer values before using averages. A candidate that completes fewer difficult cases must not gain an advantage because only its successful outputs reach the report. Declare the treatment of execution failures in the comparison specification before seeing which candidate produces them.
Key distinctions and trade-offs
Choose the experiment by the claim you need to make:
| Comparison | Deciding condition | Mechanism | Cost or limitation |
|---|---|---|---|
| Single-factor comparison | You need to attribute a change to one component | Candidate configurations differ in one declared setting | May understate the benefit of jointly tuned components |
| Whole-system comparison | You must select a deployable bundle | Freeze and identify each bundle; keep evaluation conditions shared | Establishes bundle performance, not individual causes |
| Live-environment comparison | External behavior is part of the deployment question | Interleave runs and retain upstream responses with case results | Environmental variation remains a competing explanation |
A fixed dataset improves comparability, but repeated tuning against it can turn it into a development set. Keep final selection cases separate from cases used to revise the candidates. More controls also require more fixtures and revision tracking. Spend that effort on dependencies capable of changing the decision, especially mutable tool data and judge behavior.
Architecture / decision scenario
A manufacturing procurement team compares two purchase-order investigation agents on 240 approved cases. The baseline resolves 82% and the challenger resolves 89%, but the challenger used a refreshed supplier index and a different judge. The product owner proposes shipping the challenger on that seven-point gain.
The comparison owner rejects that attribution: three factors changed. The immediate decision is whether the challenger model helps on the deployed corpus, so both candidates receive the same index snapshot, case file, tool fixtures, and judge configuration. Both use eval.general.max_concurrency of four, and their effective configurations are retained. These figures are scenario requirements, not toolkit limits.
On the controlled rerun, the baseline resolves 83% and the challenger 84%. The team’s predeclared minimum useful improvement is three percentage points, so it retains the baseline pending repeated trials; see 3.2.2. It can separately compare the refreshed index or evaluate the complete challenger bundle, but must name that different question. The procurement product owner owns usefulness; the evaluation engineer owns whether the reported comparison supports that decision.
Common exam traps
- Treating a fixed filename as proof that dataset contents stayed fixed.
- Attributing a whole-system gain to the model alone.
- Changing the judge while claiming an unchanged quality scale.
- Comparing source YAML while ignoring command-line overrides.
- Assuming remote evaluation artifacts describe the deployed workflow.
- Dropping failed cases before calculating candidate success rates.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit.” The documented evaluation interface is NeMo Agent Toolkit and nat eval. Read that product name in an exam item as the toolkit family.
What to remember
Fix the cases, dependencies, scoring, and execution conditions around a declared candidate change. Audit the effective configuration and compare matched case outputs. A controlled experiment supports the question it isolates; a higher aggregate score alone does not establish its cause.
Knowledge check
A support-ticket agent’s challenger resolves 88% of 300 cases against the baseline’s 81%. The challenger used a new model, a refreshed knowledge index and a different judge. The decision is whether the new model helps on the current index. Which run answers that?
During a comparison, the challenger receives HTTP 429 responses at eval.general.max_concurrency 8 while the baseline does not. An engineer reruns only the challenger at concurrency 1 and compares the result with the baseline’s run at 8. What should the team do?
A comparison specification declares that execution failures count as unsuccessful. On 240 matched cases, the baseline resolves 204 (85%). The challenger times out on 30 cases and resolves 189 of the remaining 210, which its report shows as 90%. What success rate should the challenger’s report show?
Two candidates’ source YAML files differ only in the model. The challenger was launched with an --override flag. Before reporting, the engineer must confirm that the runs differed only in the model, including any settings passed on the command line. What should the engineer compare?
config_effective.yml filesconfig_effective.yml records the configuration after command-line overrides, so diffing it shows whether the --override introduced a second difference. The source YAML is exactly what the overrides bypassed, a shared file name does not prove the contents stayed fixed, and the averages are the result being audited, not its conditions.