NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.5 Analyze evaluation results to guide targeted optimization.
Attribute failed cases through their traces
Completed by the knowledge check
Learning objective
Attribute failures to retrieval, reasoning, prompts, tools, or orchestration using per-case traces.
Core idea
A failed answer identifies an outcome, not the component that caused it. Follow the case from its input through retrieved material, model requests, tool executions, and workflow transitions. Find the first consequential divergence from the expected task behavior, then test whether correcting it changes the outcome.
Treat attribution as a hypothesis supported by observable execution. A model’s explanation of its own mistake is not a causal diagnosis. You need the actual inputs, outputs, errors, and ordering at the boundary you propose to change. See 3.1.3 for retrieval and answer metrics; here those scores select cases for investigation rather than settle ownership.
How it works
Join the score to the execution
NeMo Agent Toolkit writes each evaluator’s results to <evaluator_name>_output.json. The documented output includes average_score and eval_output_items, with per-entry id, score, and reasoning. Join a failed entry by its case identifier to workflow_output.json, which contains the expected answer, generated_answer, and intermediate_steps. Verify the identifier correspondence rather than joining by row position.
For a dashboard view, install nvidia-nat[weave] and set general.telemetry.tracing.<name>._type: weave, with a project, in the evaluation configuration. A nat eval run logs a Weave trace timeline; its Dataset Results tab exposes per-entry scores. With LangSmith tracing enabled, nat eval creates an Experiment whose per-example runs carry evaluator scores as feedback, with OpenTelemetry (OTel) span traces for each LLM call. Use the file-based join when investigating exported artifacts without a dashboard.
Read config_effective.yml alongside the case. It contains the configuration after command-line overrides, so it helps distinguish a defective prompt from a run that used the wrong prompt configuration. Dataset, evaluator, and execution preservation are covered in 3.1.4; the diagnostic question is which configuration actually produced this failure.
Check trace completeness before interpreting missing events. The documented default for eval.general.output.workflow_output_step_filter is [LLM_END, TOOL_END]. A missing TOOL_START in that export does not prove the tool was never invoked. Include TOOL_START when investigating dispatch, and retain profiler traces when you need the execution timeline. Distinguish a filtered export from a genuinely absent operation.
Locate the first divergence
Compare the expected task sequence with what crossed each boundary. Use these five attribution categories as a team-defined investigation rubric, not as automatic labels supplied by the toolkit:
| Suspected cause | Deciding observation | Mechanism to inspect | Targeted intervention and owner |
|---|---|---|---|
| Retrieval | Required supporting material is absent or irrelevant before generation | Retrieved context in the tool result and the corresponding model input; a ragas evaluator can supply ContextRelevance | Repair retrieval or context delivery with the retrieval owner |
| Reasoning | Adequate facts and usable tools reach the model, but it selects an invalid action or conclusion | Model outputs and tool sequence; _type: trajectory evaluates LangChain/LangGraph agent trajectories | Test the decision policy with the agent owner |
| Prompts | The effective instruction or tool description directs the wrong behavior | config_effective.yml and the recorded model input; ReAct system_prompt and additional_instructions | Revise the conflicting instruction with the prompt owner |
| Tools | An appropriate request reaches the tool, but execution returns an error or incorrect result | Tool input, output, and execution boundary in intermediate_steps | Repair the adapter or dependency with the tool owner |
| Orchestration | A valid result is discarded, associated with the wrong step, or used before its dependency finishes | Function boundaries and ordering in the trace | Repair result handoff or workflow control with the orchestration owner |
The ReAct implementation requires nvidia-nat[langchain]; its prompt settings are not universal configuration keys for every agent framework. A retrieval problem can originate after search: the retriever returns the right passage, but context assembly omits it. Attribute that to the handoff that lost the passage, not automatically to the index. Likewise, a tool returning “not authorized” can be correct behavior; inspect whether the agent chose an operation the task permits before blaming the tool.
Test the attribution without rewriting everything
For one representative failed case, propose a bounded intervention at the suspected boundary. Supply the missing passage in a controlled test, correct one conflicting instruction, or replace a faulty tool result with its verified expected value. Record the changed boundary and rerun the case. This is a diagnostic experiment, not permission to replace production results with fabricated success.
If the case still fails, inspect the next divergence. Multiple defects can coexist: correcting retrieval can expose invalid reasoning that was previously hidden behind missing context. Record a primary cause and contributing causes rather than forcing every failure into one mutually exclusive label.
Each bounded intervention ends in a rerun, and a case that still fails sends the investigation to the next divergence.
NeMo Agent Toolkit’s IntermediateStepManager publishes events tracking function boundaries, LLM calls, tool usage, and intermediate operations. Its profiler collects framework callbacks; explicitly select framework_wrappers on @register_function for supported framework instrumentation. Verify that the suspected inner operation is represented before assigning a cause. Missing instrumentation is an investigation gap, not proof that the inner operation succeeded.
Key distinctions and trade-offs
Instruction defect versus decision defect: A trace can show an invalid choice without proving why it occurred. Compare the effective instruction and available facts, then test one change. Do not label every incorrect model output a prompt problem.
Tool selection versus tool execution: A well-functioning tool can receive the wrong request. A correctly selected tool can fail during execution. The repair owners and experiments differ; see 2.6.1 for scoring action selection.
Diagnostic coverage versus trace volume: Detailed model and tool payloads make context-loss defects visible but increase retained data and review work. The profiler’s all_requests_profiler_traces.json includes raw LLM and tool inputs. Keep a deliberately selected diagnostic corpus rather than assuming a summary score contains enough detail.
Architecture / decision scenario
An insurer’s employee policy assistant must answer within 6 seconds and cite the applicable policy edition. In a 100-case review, 20 answers fail, and baseline p95 completion is 5.1 seconds. The model team proposes replacing the model across the workflow, but its trial raises p95 by 1.4 seconds to 6.5 seconds, missing the 6-second deadline.
The evaluation lead joins each failure to its trace. Seven cases lack the required policy passage at retrieval. Four contain the passage and correct instructions but choose an invalid exception. Three use a prompt that explicitly favors the superseded edition. Two reach the correct lookup tool, which returns an incorrect edition identifier. Four retrieve and interpret the right edition but lose it when the workflow hands results to final generation.
The lead assigns five repair batches: retrieval, agent decision policy, prompt, lookup adapter, and result handoff. These are distinct interventions despite the same visible symptom. For the four handoff cases, replaying the correct intermediate result repairs the answer without changing the model; for the four decision cases, it does not. The orchestration owner therefore repairs the handoff first for that batch, while the agent owner tests the reasoning cases separately. A model replacement remains a candidate only where its measured benefit addresses the observed cause and fits the deadline.
Common exam traps
- Inferring retrieval failure from answer accuracy alone.
- Treating evaluator
reasoningas a complete execution trace. - Interpreting a filtered-out start event as an unexecuted tool.
- Blaming a tool for an invalid action selected by the agent.
- Calling every bad decision a prompt defect without checking the effective input.
- Changing retrieval, prompt, and model together and claiming one caused the improvement.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit” and “AIQ Toolkit.” The documentation calls the product NeMo Agent Toolkit and uses nat eval for evaluation. Read those guide names as references to the toolkit’s evaluation and tracing role; use the documented NeMo Agent Toolkit artifacts when diagnosing a case.
What to remember
Join each failed score to its actual execution and effective configuration. Inspect the earliest consequential divergence across all five cause categories, verify trace coverage, and test a narrowly scoped intervention. Attribution becomes useful when it identifies a repair owner and a falsifiable hypothesis.
Knowledge check
A retail agent’s evaluation ran with the default eval.general.output.workflow_output_step_filter. The failure is reproducible, and the stock-lookup tool’s start/end instrumentation has been verified. In one failed case, workflow_output.json shows the model’s LLM_END events but no TOOL_START or TOOL_END for the stock-lookup tool. You must decide whether the agent never dispatched the tool or dispatched it and the call never completed. What should you do first?
TOOL_START included in the step filterreasoning for the case to confirm the dispatchLLM_END and TOOL_END , so a missing start event says nothing about dispatch, and a call that never completed leaves no end event either. For the reproduced failure, including TOOL_START distinguishes an undispatched call from one that started but never completed, provided the tool boundary is instrumented. Retained profiler traces can supply the same evidence for the original run. The evaluator’s reasoning explains a score; it is not an execution trace.A support assistant fails a case because its answer cites no warranty terms. The trace shows that the retriever’s tool result contains the correct warranty passage, but the recorded input to the final generation call does not. The effective instructions are correct. Where should the repair be targeted?
An IT assistant’s task permits only reading a user’s account record. In a failed case, the trace shows the agent called the disable_account tool, which returned “not authorized”. config_effective.yml and the recorded model input show read-only instructions, accurate tool descriptions, and the account data the task needed. Which owner should receive the case?
disable_accountIn four failed cases of a travel-booking agent, the trace shows the correct fare retrieved and interpreted, then absent from the input to final generation. You suspect the result handoff. Which experiment tests that attribution?
A policy assistant answers from a superseded policy edition. The prompt file committed to the repository tells the model to use the current edition, but the evaluation was launched with command-line overrides. Which artifact shows the prompt configuration that actually produced the failure?
config_effective.yml written by the evaluation runreasoning for the failed entryconfig_effective.yml holds the configuration after command-line overrides — the one the run actually used — so it separates a defective prompt from a run that used the wrong prompt configuration. The committed file shows only the intended configuration, and the evaluator’s reasoning explains a score, not the configuration behind it.