NVIDIANVIDIA-Certified Professional: Agentic AI2026-102.6 Evaluate and refine agent decision-making strategies.
Score tool choices and trajectories separately
Completed by the knowledge check
Learning objective
Evaluate tool-selection accuracy and trajectory quality.
Core idea
An agent can produce a correct answer after choosing an unnecessary tool, using the wrong account identifier, or repeating a successful operation. Answer correctness measures the destination. Tool-selection accuracy measures a decision at a particular state. Trajectory quality measures whether the sequence of decisions reaches the destination appropriately.
Score all three separately. Otherwise, a higher answer score can hide a deterioration in action selection, and a low trajectory score cannot tell you whether to repair the tool description, the arguments, or the stopping behavior. See 3.1 for evaluation-pipeline design; this lesson defines what you measure inside an execution.
How it works
Define a correct choice at a known state
Create evaluation cases that specify the request, available tools, observations already received, and acceptable next actions. Include cases where the correct choice is to ask for missing information or finish without a tool. If the denominator contains only executed calls, an agent that always calls something escapes the test of whether a call was needed.
Define your tool-selection score as the number of evaluated states with an acceptable selected action divided by the number of evaluated states. This is a team-defined metric, not a built-in NeMo Agent Toolkit metric name. Record an explicit no-call outcome alongside tool names. Define argument correctness separately: selecting lookup_invoice with another customer’s identifier counts as a selection success under a name-only definition, yet it is an operational failure.
Where several actions are valid, annotate an acceptable set and the conditions that distinguish its members. Two independent lookups can occur in either order. A refund before checking eligibility cannot be made acceptable by treating every permutation as equivalent. The reference is a set of constraints on decisions, rather than a single transcript every successful run must imitate.
Connect the rubric to execution records
NeMo Agent Toolkit runs configuration-driven evaluation with nat eval --config_file=EVAL_CONFIG_PATH; replace EVAL_CONFIG_PATH with your evaluation configuration. The full command runtime needs nvidia-nat[eval]. Its workflow_output.json contains per-case question, answer, generated_answer, and intermediate_steps, so you can compare the returned answer with the actions that produced it.
For explicit checks, implement a custom evaluator registered with @register_evaluator. Its asynchronous evaluate_item receives an EvalInputItem and returns an EvalOutputItem. The input exposes trajectory, optional expected_trajectory, input_obj, output_obj, and expected_output_obj. Return the case id, a numeric score, and reasoning explaining the failed condition. Use separate evaluators for tool identity and arguments when you need independent aggregate scores.
For canonical ATIF trajectories, subclass AtifBaseEvaluator and implement evaluate_atif_item; trajectory steps expose tool_calls for tool-level scoring. These helper classes and ATIF-specific models remain evaluation subsystem APIs, while registration, configuration, and EvaluatorInfo are stable public plugin APIs. Registered ATIF evaluators require nvidia-nat-eval[full]; standalone ATIF harness usage needs only nvidia-nat-eval.
Store case annotations such as acceptable tools and required prerequisites in dataset metadata. Set eval.general.dataset.pass_full_entry: true to expose those fields through EvalInputItem.full_dataset_entry. Your evaluator translates them into assertions; adding metadata alone does not make the toolkit enforce the rubric.
Check that your selected workflow integration actually captures the tool events the evaluator needs. An empty trajectory is missing observation data, not proof that no tool ran. The output filter eval.general.output.workflow_output_step_filter controls which intermediate steps appear in the output file. The default is [LLM_END, TOOL_END]; the documented example adds TOOL_START. Keep the records needed to inspect arguments and correlate returned results before interpreting a score.
Judge the path without hiding exact failures
The toolkit’s built-in _type: trajectory evaluator uses a judge LLM, selected through llm_name under eval.evaluators, to assess intermediate steps while considering available tools. NVIDIA documents a score from zero to one and describes this evaluator as LangChain/LangGraph agent trajectory evaluation. Do not assume that selecting this evaluator gives every external agent integration equivalent trajectory coverage.
Use the judge for qualities that need contextual judgment: whether a lookup was useful, whether the agent acted on the observation, and whether it stopped with a sufficient answer. Keep exact prerequisites, tool names, and argument checks deterministic in your custom evaluators. A judge score of 0.9 is not a statement that 90% of tool selections were correct.
Inspect disagreements between the judge and the explicit checks. A judge may approve an economical answer while a prerequisite assertion catches a skipped eligibility lookup. Preserve that failure even if the final answer happens to match. Your evaluator’s reasoning should identify the failing step, not merely describe the answer as poor.
The custom evaluators and the judge score the trajectory separately, and you inspect where they disagree.
Key distinctions and trade-offs
These measurements answer different questions:
| Measurement | Unit being scored | What it detects | What it can miss |
|---|---|---|---|
| Tool-selection accuracy | A decision at a specified state | Wrong tool or unnecessary call | Incorrect arguments and later misuse |
| Argument correctness | A selected tool’s inputs | Wrong identifier, missing field, invalid value | Whether the tool was needed |
| Trajectory quality | The execution sequence | Redundancy, missed prerequisites, failure to use results | Exact failures absent from its rubric |
| Answer correctness | The final result | Wrong or incomplete business outcome | Wasteful or inappropriate intermediate actions |
More detailed annotation costs domain-expert time, but gives a targeted repair. A single judge score is cheaper to introduce and harder to diagnose. An exact transcript match is easy to compute but rejects valid alternative paths. Prefer invariant prerequisites and allowed alternatives when the workflow permits them.
Architecture / decision scenario
In an illustrative replay of 200 invoice questions, an accounts-payable assistant returns 180 correct answers, but only 150 first actions match the acceptable tool set. Thirty cases ask about status yet call a payment-mutation tool; eight of those also use an incorrect invoice identifier. Successful answers do not erase those selections.
The finance automation owner rejects the proposal to release on the 90% answer score alone. The evaluation engineer adds separate custom evaluators for first-action identity and argument scope, using metadata exposed through pass_full_entry. A trajectory judge assesses redundant status checks and stopping behavior. These outputs distinguish description confusion from identifier extraction and repeated calls.
The competing proposal is to require one exact reference sequence for every question. The team rejects it because two read-only lookups can legitimately be reversed. Instead, the reference annotation permits both orders but requires the eligibility result before any mutation. The owner chooses a description repair for the wrong-tool cases and an argument-validation repair for the eight identifier failures, then reruns the same cases. The release decision considers tool identity, argument scope, trajectory quality, and answer correctness separately.
Common exam traps
- Treating a correct final answer as proof of correct action selection.
- Calling a judge’s trajectory score a tool-selection percentage.
- Penalizing every difference from one reference sequence.
- Counting only executed calls and omitting no-call decisions.
- Assuming an absent tool event proves that the agent did not execute a tool.
Where the exam guide differs
The study guide’s suggested readings say “NVIDIA Agent Intelligence Toolkit (1.1.0).” The evaluation documentation uses NeMo Agent Toolkit and the nat eval command. Read an exam reference to the earlier toolkit name as the workflow evaluation role; use the documented NeMo Agent Toolkit interfaces when implementing it.
What to remember
Evaluate the selected action, its arguments, the execution sequence, and the final answer as distinct objects. Combine explicit custom assertions with a trajectory judge, preserve valid alternative paths, and inspect missing instrumentation before trusting a score.
Knowledge check
A team checks executed calls only for valid tool names and reports 98% tool-selection accuracy. Support logs show the agent calling a tool on requests where the right move is to ask the user for a missing account number. How should the team redefine the score?
trajectory judge’s score, averaged over every evaluated caseA release report says 90% of tool selections are correct, citing the built-in trajectory evaluator’s mean score of 0.9. The release board needs the share of annotated states at which the agent chose an acceptable action. What should the team do?
workflow_output.json in its placetrajectory evaluator with --reps=5 and report the meantrajectory evaluator is a judge LLM rating the whole path from zero to one; 0.9 is not a statement that 90% of selections were correct. Exact selection checks belong in a custom evaluator that returns a score and a reasoning per case, and a correct final answer can hide wrong selections.In a billing workflow, two read-only lookups can run in either order, but a refund must follow an eligibility check. The reference requires one exact step sequence and rejects many runs reviewers consider correct. How should the reference change?
After a team switches its agent integration, its custom tool evaluator finds empty trajectories in most cases, and the wrong-tool rate falls to almost zero. What should the team suspect first?
trajectory evaluator needs replacingexpected_trajectory , so no steps were comparedexpected_trajectory cannot empty the recorded one, and no judge is involved in a custom evaluator’s result.A team adds acceptable_tools and required_prerequisites fields to every dataset entry, reruns nat eval, and sees no score change. What completes the rubric?
expected_trajectory so the toolkit applies themTOOL_START to workflow_output_step_filter to load the fieldspass_full_entry: true and assert the fields in your own evaluatortrajectory evaluator’s configuration blockpass_full_entry: true exposes the whole entry as EvalInputItem.full_dataset_entry , and the evaluator turns those fields into assertions; metadata alone makes the toolkit enforce nothing. The step filter decides which intermediate steps reach the output file, not which dataset fields reach an evaluator.