NVIDIANVIDIA-Certified Professional: Agentic AI2026-1010.2 Design structured feedback loops that guide iterative agent improvements.
Route feedback to a testable repair
Completed by the knowledge check
Learning objective
Design a feedback taxonomy tied to specific improvement actions.
Core idea
A feedback taxonomy is useful when a category changes what your team does next. “Bad answer” groups together missing source material, an unsupported claim, a failed tool, and an appropriate refusal that disappointed someone. Sending all four to prompt tuning spends engineering time without establishing the cause.
Separate the reporter’s observation from the team’s diagnosis. A reader can report an incorrect total; only investigation establishes whether the source, calculation, or presentation failed. Keep both in an action registry so adjudication does not erase the original complaint.
NIST AI RMF GOVERN 5.1 describes collecting, considering, prioritizing, and integrating feedback from people outside the development team about potential individual and societal impacts of AI risks; GOVERN 5.2 calls for incorporating adjudicated feedback into system design and implementation. Your taxonomy operationalizes that distinction. See 3.3.1 for response-linked collection and 3.3.2 for privacy-aware dataset curation. Here you decide which improvement a reviewed report warrants and how its owner proves completion.
How it works
Define categories by their next action
Write a category specification with an inclusion rule, an exclusion rule, an accountable owner, a repair artifact, and a closure test. The following is a proposed application taxonomy, not a NeMo Agent Toolkit enumeration:
| Category | Deciding observation | Improvement action and mechanism | Owner and cost |
|---|---|---|---|
| Source coverage | Required authoritative material is missing from the retrieved context | Repair source coverage; require an expected-source custom check and use ContextRelevance diagnostically | Knowledge/retrieval owner; ingestion or retrieval work |
| Unsupported response | Context contains the needed facts, but the answer invents or contradicts them | Revise the scoped instructions; measure ResponseGroundedness and answer quality | Agent owner; extra evaluation and possible prompt tokens |
| Tool selection or arguments | Execution takes an inappropriate tool path or supplies invalid inputs | Repair the tool interface or agent tool exposure; inspect intermediate_steps and a trajectory evaluator | Tool/agent owner; integration review and replay cost |
| Workflow or presentation | Correct content violates a confirmed task instruction or output requirement | Change a specific instruction and add a requirement-specific custom evaluator | Product/agent owner; risk of changing other tasks |
| Safety or authorization | The reported outcome violates a policy, or a permitted task is blocked | Open policy adjudication and a targeted policy regression case; see 9.1.1 and 9.3.2 | Security/policy owner; expert review before adaptation |
| Service performance | A completed task misses the agreed latency or cost target | Route to the service-target investigation in 8.1.1 rather than content training | Operations owner; capacity and performance analysis |
| Unresolved report | The report lacks enough information to distinguish causes | Request clarification or investigate a replay; retain an explicit unresolved disposition | Triage owner; review capacity rather than speculative tuning |
Several reports can share a repair, and one report can have several diagnosed causes. Preserve a primary routing category for accountability and secondary labels for analysis. Do not force a tool timeout followed by an invented answer into whichever label the reviewer happened to select first.
Make the action registry executable by a team
Use application-defined fields such as report_id, observed_issue, diagnosed_category, review_status, owner, repair_id, and closure_test_id. These names describe your registry, not toolkit API fields. The registry joins the collection record to a specific engineering task and, eventually, a test result. A category without an owner becomes a dashboard count; a task without a closure test becomes an unverifiable claim of improvement.
For a NeMo Agent Toolkit workflow with _type: react_agent, route an instruction-category repair to workflow.additional_instructions; see 9.3.3 for prompt repairs. Route a permission-category defect to an authorization repair, rather than changing workflow.tool_names; see 9.1.1.
Do not turn every category into a training label. A permission defect needs an authorization repair. A missing document needs a knowledge repair. A preference for shorter answers needs product confirmation before it changes a shared instruction. See 8.4.1 for choosing tuning versus training after diagnosis; the taxonomy supplies that decision’s reviewed input.
Attach a category-specific acceptance test
NIST MEASURE 3.3 connects feedback and appeals to evaluation metrics. Translate each accepted report into a test that distinguishes the reported failure from a passing outcome. State the expected result and any prohibited behavior before proposing the repair.
In NeMo Agent Toolkit, configure evaluators under eval.evaluators. Config-driven evaluation needs the full runtime: nvidia-nat[eval] for nat eval, and nvidia-nat-eval[full] for registered custom evaluators. The ragas metrics ContextRelevance, ResponseGroundedness, and AnswerAccuracy answer different diagnostic questions; _type: trajectory assesses the tool path. A response can be grounded in an irrelevant passage, and relevant context can still omit a required source. Require the expected-source check for coverage; a single score cannot establish that both retrieval and answering improved.
When a category concerns a business requirement that those evaluators do not capture, implement a custom evaluator using register_evaluator and EvaluatorInfo, then configure that evaluator explicitly. For an analytics assistant’s output, a custom check might verify the requested reporting period and required units. Keep its result beside the domain expert’s reviewed expectation. An automated check cannot settle an ambiguous business definition merely by returning a number.
Key distinctions and trade-offs
Observation versus cause: Keep “wrong total” as the user’s observation even when the cause is a stale table. Changing the diagnosis should update routing, not rewrite what the reporter experienced.
Category versus severity: Category chooses the repair owner; severity chooses urgency and response. An uncommon authorization defect can outrank a frequent formatting complaint. Apply severity independently, with an explicit policy owner for contested cases.
Granularity versus agreement: A fine taxonomy is costly if reviewers cannot distinguish its labels. Maintain a labeled example set and an adjudication record for disagreements. Split a category when the split changes owner, intervention, or acceptance test; merge labels that always produce the same decision.
Continuity versus refinement: Add an application-defined taxonomy_version to each registry record. When you split or merge categories, retain the original labels and a versioned mapping to their replacements. Compare category shares within one version or through that mapping. Otherwise, a classification change can look like a rise or fall in failures even when behavior stays the same.
Architecture / decision scenario
A manufacturing analytics assistant receives 200 reviewed complaints in a month. The triage team diagnoses 110 as missing maintenance records, 45 as incorrect time-period selection, 25 as unsupported conclusions despite adequate context, and 20 as unresolved. Two engineers can complete only one major intervention this sprint. A proposed blanket prompt rewrite treats all 200 as answer-quality defects.
The product owner rejects that proposal. The 110 coverage cases go to the knowledge owner with retrieval acceptance cases, while the 45 time-period cases receive an explicit reporting-period check and a scoped instruction ticket. The 25 unsupported conclusions receive grounding tests. The 20 unresolved reports stay pending investigation rather than becoming assumed model errors.
Coverage accounts for 55% of reviewed complaints, versus 12.5% for unsupported conclusions, so the team prioritizes the knowledge repair unless severity changes the ordering. Those percentages describe this reviewed complaint set, not all production requests. The owner closes the repair only after the coverage cases and unrelated control tasks pass; 10.2.2 carries the accepted change through the release loop.
Common exam traps
- Using satisfaction as a diagnosis of model failure.
- Treating a category, a severity level, and a product preference as interchangeable.
- Routing missing knowledge, tool permissions, and presentation defects to the same training job.
- Treating an unresolved complaint as a confirmed failure or silently dropping it.
- Closing a feedback item when a change is merged instead of when its acceptance test passes.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit.” The cited documentation calls the product NeMo Agent Toolkit. Read the guide’s name as that product lineage; use the documented toolkit identifiers when assigning repairs and evaluation checks.
What to remember
A category earns its place by choosing an owner, a repair, and a closure test. Preserve observations separately from adjudicated causes, distinguish severity from category, and measure the failure the reporter actually experienced. A taxonomy directs change; it does not automatically authorize training or establish that a release improved.
Knowledge check
A reader of a manufacturing analytics assistant reports that “the quarterly downtime total is wrong.” Investigation finds that the retrieved context lacked two required maintenance records, and the model summed the records it did receive correctly. How should the action registry record this report?
A month of reviewed feedback holds 300 formatting complaints and 3 reports that the assistant exported records the requester was not authorized to see. The policy owner rates the disclosures as critical and the formatting as minor. Two engineers can take one repair this sprint. How should the team order the work?
A source coverage repair adds a missing maintenance manual to the index. Before closing the related feedback items, the owner needs an acceptance test that distinguishes the reported failure from a passing outcome. Which test fits this category?
ResponseGroundedness on the reported questionstrajectory evaluator over the workflow’s tool-call sequenceContextRelevance is diagnostic beside it. A response can be grounded in an irrelevant passage, so groundedness does not show coverage, and a trajectory evaluator assesses the tool path. Fewer complaints can reflect less use rather than a repair.Of 200 reviewed complaints, 20 lack enough detail to tell whether retrieval, a tool, or the answer failed. The dashboard owner proposes assigning them to categories in proportion to the 180 diagnosed reports. What should happen to the 20 instead?