NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.5 Analyze evaluation results to guide targeted optimization.
Prioritize repairs and verify their effect
Completed by the knowledge check
Learning objective
Prioritize fixes by impact and verify them with re-evaluation.
Core idea
Choose a fix because it removes a consequential failure at an acceptable cost, then require a result that could disprove your choice. The largest error category is not automatically the highest priority, and a repaired demonstration is not proof that the workflow improved.
Turn the attribution from 3.5.1 and the bottleneck analysis from 3.5.2 into a ranked repair backlog. Each entry needs a failure population, consequence, proposed intervention, owner, expected benefit, and acceptance test. See 3.2.1–3.2.3 for controlled comparisons and budget-aware selection; this lesson concerns which diagnosed defect to address and what closes it.
How it works
Rank consequences before convenience
NIST AI RMF 1.0, MANAGE 1.2, prioritizes treatment of documented AI risks using impact, likelihood, and available resources or methods. MANAGE 1.3 calls for planned, documented responses, and MANAGE 1.4 addresses residual risks. Use those dimensions in the team’s repair record rather than ranking only by the count of failed answers.
For each diagnosed cluster, record how often the failure appears in the relevant workload and what happens when it occurs. Include affected task types, user groups, and consequences. A rare unauthorized disclosure can outrank frequent formatting defects. The benchmark frequency is an observation about that dataset; do not equate a deliberately oversampled risk slice with production prevalence.
Then record how much of the problem the proposed intervention can plausibly remove. Twenty failed cases do not imply twenty recoverable cases: some might lack required source material, depend on an unavailable service, or have a different underlying cause. Use the per-case traces and diagnostic intervention to distinguish repairable failures from the broader symptom count.
Make effort and trade-offs explicit. A low-effort prompt revision is attractive when the traces show a conflicting instruction; it is less credible when correct instructions already reach the model and the tool returns bad data. Name the artifact that changes, the responsible owner, the targeted case identifiers, and the expected direction of change. These are team-defined backlog fields, not built-in NeMo Agent Toolkit configuration keys.
Define a repair test that can fail
Before implementing the candidate, write acceptance criteria for three views: the targeted failure cluster, the rest of the regression set, and delivery budgets. For example, require the ten known context-handoff cases to pass, no additional unauthorized actions on the safety slice, and completion latency within the agreed target. Use task-level outcomes as well as the component metric the fix directly targets.
This prevents a local success from hiding a global regression. A retriever can return more relevant context while the longer input makes the assistant miss its deadline. A shorter response can reduce token use while omitting the required decision. The intended optimization includes constraints on quality and delivery, not only one improved scalar.
Keep a record of residual failures and who accepts them. NIST’s MANAGE function permits different response types, including mitigation, avoidance, transfer, and acceptance; it does not prescribe your organization’s tolerance. A release owner must decide whether the observed residual risk fits the deployment, rather than allowing a higher average score to make that decision implicitly.
Generate fresh results for the changed path
Use nat eval to execute the changed workflow and score its outputs. The full config-driven workflow-evaluation runtime requires the documented nvidia-nat[eval] installation. Keep the baseline and candidate configurations distinct and inspect config_effective.yml to verify which intervention each run actually applied.
NeMo Agent Toolkit writes per-case results in workflow_output.json and each evaluator’s results in <evaluator_name>_output.json. Compare the same case identifiers across baseline and candidate. Count repaired failures, retained failures, and newly introduced failures; inspect the associated traces for the repaired cluster. A different answer is not necessarily a repaired answer, and a net gain can conceal an unacceptable regression.
Using the Weave tracing setup from 3.5.1, log both runs to the same project and distinguish them with eval.general.workflow_alias. Select the runs and use Compare for metric summaries and Dataset Results for per-entry scores; plotting those scores supports only two runs. With LangSmith tracing, each evaluation becomes an Experiment named <project> (Run #N) linked to the same dataset.
Set eval.general.output.job_management.append_job_id_to_output_dir: true to preserve separate run directories; see 3.1.4. For repeated trials, nat eval --reps runs multiple repetitions. Apply the same repetition policy to both configurations and inspect whether the targeted improvement persists rather than selecting the candidate’s most favorable run. Detailed repeat-run comparison belongs to 3.2.2.
The --skip_workflow option evaluates previously generated answers. Use it when investigating evaluator behavior on a fixed output set, not as proof that a changed retriever, prompt, tool, or workflow produces better results. Similarly, --skip_completed_entries resumes incomplete evaluation work; reusing completed baseline answers does not exercise a modified workflow on those cases. Execute the candidate on every case needed to establish the repair.
Key distinctions and trade-offs
Use the observed consequence to distinguish repair priorities:
| Candidate | Deciding condition | Verification artifact | Cost or risk to examine |
|---|---|---|---|
| Correct a severe but infrequent failure | Consequence exceeds the deployment’s tolerance | Targeted risk-slice outcomes and traces | Residual exposure while the repair is incomplete |
| Repair a frequent answer defect | Many relevant tasks fail for the same diagnosed cause | Per-case evaluator results joined to workflow outputs | Regressions on previously passing tasks |
| Optimize a slow step | The step delays requests that miss the delivery target | Profiler timeline and complete-request measurements | Lost quality or contention at required concurrency |
| Improve scoring or labels | The apparent defect comes from measurement | Rescored fixed outputs, then a confirmed baseline | Mistaking judge changes for agent improvements |
Prefer a bounded intervention when it can address the highest-priority cause and make the experiment interpretable. A broader redesign can still be warranted, but its benefit must be measured across the additional behavior it changes. Automated parameter and prompt search belongs to 3.4.3; diagnosing a cluster does not require searching every possible model configuration.
Architecture / decision scenario
A university’s research assistant prepares literature summaries for staff. In a 500-case evaluation, 40 summaries miss relevant citations, 15 omit a requested section, and 5 use restricted material in an answer to an unauthorized role. The team has two engineer-days before a release decision, and the release owner requires zero observed restricted-material disclosures on its dedicated risk slice and p95 completion within 10 seconds.
One proposal rewrites the summary prompt to address the 40 citation failures. Another repairs the context handoff that the five disclosure traces identify as bypassing the approved access-filtered result. The security owner prioritizes the latter because consequence outranks raw frequency. The retrieval owner keeps citation repair next in the backlog; the missing-section prompt fix follows it.
After the handoff repair, the evaluation engineer runs fresh candidate executions for all 500 cases and the dedicated risk slice. Across matched repeated trials, the five known disclosures disappear, but three previously passing summaries now fail because required permitted context is also dropped. Measured p95 rises from 9.0 to 10.8 seconds. The repair therefore remains open despite the improved disclosure count.
The workflow owner narrows the handoff change to preserve permitted context and removes duplicated context processing. A second fresh evaluation must verify the targeted cases, introduced regressions, and latency criterion together. The release owner decides from that record; passing a finite risk slice is a release observation, not proof that every possible disclosure has been eliminated.
Common exam traps
- Prioritizing the most frequent failure without considering its consequence.
- Treating every case in a symptom cluster as recoverable by one fix.
- Using a development example as the entire repair test.
- Accepting an average-score gain while previously passing critical cases regress.
- Using
--skip_workflowto validate a changed execution path. - Resuming saved baseline answers and presenting them as candidate executions.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit” and “AIQ Toolkit.” The documented product is NeMo Agent Toolkit, whose nat eval command executes and evaluates workflows. Read an exam reference to those toolkit names as the evaluation role; distinguish fresh workflow execution from its documented offline rescoring option.
What to remember
Prioritize by consequence, likelihood, recoverable benefit, and available resources. Define a repair’s acceptance criteria before making the change, then execute the candidate on the relevant cases. Close the fix only when targeted failures improve without violating regression and delivery constraints, and record the residual risk.
Knowledge check
A bank’s assistant shows three clusters in a 400-case evaluation: 30 answers with formatting defects, 12 incomplete summaries, and 3 answers that disclose another customer’s balance. Traces attribute each cluster to one diagnosed cause with a bounded fix. The release owner tolerates zero disclosures, and the team has capacity for one repair before the release decision. Which repair should come first?
A team changes how its retriever filters documents and must show that the change repairs 12 known failures. An engineer proposes running nat eval --skip_workflow with the candidate configuration on the baseline run’s workflow_output.json. What should the team do instead?
--skip_completed_entriesnat eval on every case--skip_workflow scores answers that were already generated, so the changed retriever never runs and the scores still describe the baseline. Resuming with --skip_completed_entries likewise reuses saved baseline answers. Fresh candidate executions on the targeted cases test the repair, and the regression set checks for newly introduced failures. Running only previously passing cases would omit the 12 failures; rescoring fixed outputs tests the evaluator, not the changed workflow.Before a repair, the team set acceptance criteria for the targeted cluster, the regression set, and the latency budget. In a fresh evaluation, the candidate passes all ten targeted context-handoff cases and raises the average score, but two previously passing cases on the safety slice now take an unauthorized action, and p95 completion rises above the agreed target. What should happen to the repair?
Two diagnosed clusters have the same consequence and similar repair effort, and only one fix fits this cycle. Fix A targets a 20-case cluster, but diagnostic interventions recover only 6 of those cases; the rest lack the required source material. Fix B targets a 12-case cluster, and its diagnostic interventions recover 11. Which fix should rank higher?