NVIDIANVIDIA-Certified Professional: Agentic AI2026-102.6 Evaluate and refine agent decision-making strategies.
Compare strategies on identical task cases
Completed by the knowledge check
Learning objective
Compare decision strategies (ReAct, tool calling, planning, routing) on the same task.
Core idea
A strategy comparison asks which decision process earns its cost on your workload. It does not ask which framework produces the most impressive demonstration. Give each candidate the same requests, tools, observations, and business acceptance criteria, then compare successful outcomes and execution cost together.
See 1.2.3 for ReAct, tool calling, and ReWOO mechanics, and 1.6.2 for router construction. Here you control the experiment: what changes between candidates, what remains fixed, and what result justifies choosing one strategy over another.
How it works
Define what each candidate includes
In NeMo Agent Toolkit, select the candidate through workflow._type. The comparison can use react_agent, tool_calling_agent, rewoo_agent as an upfront-planning implementation, and router_agent as a branch-selection implementation. These documented agents require the nvidia-nat[langchain] plugin.
ReAct and native tool calling are not mutually exclusive concepts across the entire agent ecosystem: one describes a reasoning/action pattern and the other an interface for requesting tools. Your experiment compares these specific toolkit implementations. The tool calling agent requires a model that supports tool or function calling. Choose a compatible model before holding the model fixed across candidates.
Set tool_names and llm_name for the first three candidates. The router uses branches and llm_name; those branches must provide the work needed to finish the task. Include branch execution in the router’s cost and correctness measurements. A router selecting a specialist that performs ten calls is not a one-call solution.
Match cases and observations
Use identical task identifiers and initial state. If the task reads mutable records, replay the same recorded responses or use resettable test fixtures. Otherwise, one strategy might encounter an available item while another encounters an unavailable item, and the difference gets misattributed to reasoning.
Keep the tool implementations, permissions, data snapshot, result rubric, and endpoint conditions fixed. Strategy-specific prompts must satisfy their implementation’s required format; equal treatment does not mean pasting the ReAct prompt into the ReWOO planner. Save the candidate configurations so the result identifies precisely what was compared.
Set and record each candidate’s action and retry budgets explicitly. ReAct defaults are max_tool_calls: 15, tool_call_max_retries: 1, and parse_agent_response_max_retries: 1; tool calling uses max_iterations: 15 for its tool-call cap; ReWOO defaults to tool_call_max_retries: 3; routing defaults to max_router_retries: 3. Align comparable budgets and document differences between parsing, tool-execution, and branch-selection retries, so an extra recovery attempt is not mistaken for a strategy advantage.
Run the configurations against the same eval.general.dataset using nat eval --config_file=EVAL_CONFIG_PATH. The full evaluation runtime requires nvidia-nat[eval]. Control eval.general.max_concurrency across runs so different admission pressure does not masquerade as a strategy effect. Use the nat eval option --reps=5 for five repeated executions when variable choices can change the outcome.
Hold llm_name fixed for an initial strategy comparison. Then evaluate a separately labeled model change if necessary. Changing the strategy, model, and tool descriptions together can identify a good complete system, but cannot isolate why its decisions improved.
Score the same business result
Apply the measurements from 2.6.1: tool-selection accuracy, argument correctness, trajectory quality, and answer correctness. Compare the paths with equivalent prerequisites, rather than requiring every candidate to reproduce the same steps. A router’s correct branch selection and a ReAct agent’s correct next tool are intermediate measurements; neither replaces successful completion.
Include latency, model-token consumption, tool-call counts, and failure outcomes in the candidate result sheet. Configure eval.general.profiler and install nvidia-nat-profiler to obtain per-request latency and token columns in standardized_data_all.csv; workflow_profiling_metrics.json supplies aggregate means and percentiles. Retain per-case results in workflow_output.json and the evaluator-specific output files. Compare disagreements by task identifier: the requests one strategy fixes may be different from those another strategy fixes, even when their averages agree.
Include a slice in which an observation changes the required next action. That tests the benefit of adaptive decision-making. Include simple cases too, so planning overhead cannot hide inside a benchmark made entirely of difficult requests. A candidate that fails or exceeds the application deadline remains in the results; omitting those cases rewards expensive strategies that abandon the hardest work.
Key distinctions and trade-offs
The decision separates these implementations:
| Strategy | Deciding task condition | Toolkit mechanism | Cost or failure to measure | Mechanism owner |
|---|---|---|---|---|
| ReAct | The next useful action depends on a returned observation | react_agent, tool_names | Repeated reasoning and tool calls; parsing failures | 1.2.2 |
| Tool calling | Structured tool selection is sufficient | tool_calling_agent, compatible llm_name | Wrong function or arguments; unnecessary calls | 1.2.3 |
| Planning | Dependencies can be specified before execution | rewoo_agent, planner and solver phases | Upfront planning and poor adaptation to surprises | 1.2.3 |
| Routing | One branch can own the complete request | router_agent, branches | Misrouting and full downstream branch cost | 1.6.2 |
NVIDIA documents the tool calling agent as limited in complex reasoning between calls. ReWOO separates planning, execution, and solution, but its upfront plan limits adaptation to unexpected results. The router selects one branch per request; it does not itself combine branches or perform multi-step routing. Design the benchmark to expose those limits, rather than treating each label as a universal capability claim.
The cheapest inference path may still produce the highest cost per accepted result if it causes repeated failures. Conversely, a small quality gain can be irrelevant when its latency misses the product’s deadline. Make the minimum acceptable result explicit before ranking cost among candidates that meet it.
Architecture / decision scenario
A travel company’s itinerary-repair assistant must resolve disrupted bookings within a 10-second response target and achieve at least 95% task success in its test set. The set contains 300 fixed requests, including 60 where a supplier response changes the feasible next step. All four candidates use the same compatible model and replayed supplier responses.
Illustrative measurements show ReAct at 96% success and 9 seconds p95, tool calling at 89% and 5 seconds, ReWOO at 92% and 7 seconds, and routing at 93% and 6 seconds. These are scenario results, not vendor performance figures. Each router branch includes its complete booking-repair execution in the measured time.
The product owner chooses ReAct for this workload. Tool calling is tempting because it responds four seconds faster, but its 89% success rate misses the 95% requirement. ReWOO loses cases where the supplier observation invalidates its plan. The router’s single branch does not complete enough requests that cross specialist responsibilities.
The evaluation engineer verifies those explanations through matched failed cases rather than guessing from the averages. The team records token and call costs before rollout and retains the simpler candidates as comparison points for a narrower future workload. It does not claim that ReAct is intrinsically better; the deciding condition is successful adaptation inside this workload’s deadline.
Common exam traps
- Comparing strategies on different tasks or mutable tool responses.
- Excluding downstream specialist work from routing cost.
- Assuming native tool calling always means one inference request.
- Treating an upfront plan as automatic adaptation after every observation.
- Choosing on average accuracy while discarding failed or timed-out runs.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit (1.1.0)” for the toolkit documented here as NeMo Agent Toolkit. Read an exam’s “ReAct,” “planning,” or “routing” as a decision pattern, then distinguish the specific implementation’s tool interface and control flow; the pattern name alone does not establish its measured quality.
What to remember
Compare complete executions on matched cases. Hold tools, state, model, concurrency, and acceptance criteria fixed first. Choose among strategies that meet the required result and latency, using per-case failures to explain where each strategy earns or wastes its cost.
Knowledge check
A team compares react_agent and rewoo_agent on hotel-rebooking cases whose tools query live room availability. ReAct wins mostly on cases where ReWOO found the target hotel sold out. The team must attribute the difference to the strategy. What should change first?
--reps=5 and average the scorestool_call_max_retries budget than ReActA router_agent is measured at one LLM call and under a second per request, against six seconds for a ReAct agent on the same cases. Its branches are specialist agents that each run their own tool loop. What must the router’s measurements include?
max_router_retries budget raised to match ReAct’s limitsCandidate B switches the strategy to ReAct, changes llm_name to a newer model and rewrites the tool descriptions; it beats candidate A by eight points. The architect must report whether the strategy caused the gain. What is the next run?
--reps=5 to confirm that the gain is stablellm_name and tool descriptionsA warehouse-exception assistant needs at least 90% task success with p95 latency of 8 seconds or less. On matched cases, illustrative results are: tool calling 88% at 4 seconds, ReAct 93% at 7 seconds, ReWOO 94% at 11 seconds, and routing 87% at 6 seconds. Which candidate should the team choose for this workload?
One candidate’s accuracy is computed after excluding 40 runs that exceeded the application deadline; the other candidate is scored on every case. How should the comparison be fixed?