NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.3 Collect and integrate structured user feedback for iterative improvements.
Attach actionable feedback to exact responses
Completed by the knowledge check
Learning objective
Capture structured ratings, labels, and corrections linked to traces and session context.
Core idea
A negative rating tells you that someone disliked an outcome. A diagnostic label states what they believe failed; a correction proposes what the answer should have been. These signals have different authority. Preserve them separately and attach each to the exact response that prompted it, so an evaluator can reconstruct the task instead of guessing from a session-wide satisfaction score.
See 2.5.3 for feedback that changes an active conversation. Here the artifact is a durable feedback record for later evaluation. The taxonomy and release-improvement loop belong to 10.2.1 and 10.2.2; this lesson establishes what those processes can reliably consume.
How it works
Separate the three signals
Use three independently interpretable fields in your application-owned feedback schema. Record a rating with its scale and question: a binary helpfulness reaction and a five-point correctness score are different measurements. Record a label using an enumerated vocabulary and rubric version, so a change in the meaning of “incorrect” does not silently change historical counts. Record a correction as a proposed replacement or corrected fact, together with its review status and source.
A user can reasonably rate a refusal as unhelpful even when it follows policy. A user can also supply an incorrect replacement. Keep the submitted label separate from the adjudicated label, and the proposed correction separate from the approved reference answer. NIST AI RMF subcategory MEASURE 3.3 connects reports from users and affected communities to evaluation metrics; domain review makes those reports usable without treating every report as established truth.
Use NVIDIA’s trace-linked reaction path
NeMo Agent Toolkit documents POST /feedback for reactions on an assistant message using observability_trace_id and reaction_type; its request example uses "👍". This route is available with the Weave FastAPI plugin worker. Install nvidia-nat[weave] and configure the worker and tracing exporter:
general:
front_end:
_type: fastapi
runner_class: nat.plugins.weave.fastapi_plugin_worker.WeaveFastAPIPluginWorker
telemetry:
tracing:
weave:
_type: weave
project: "nat-demo"Keep that reaction path distinct from your richer collection schema. Persist diagnostic labels, proposed corrections, rubric versions, and review decisions in your application’s feedback store. Join them to the same response and trace rather than treating arbitrary extra application fields as part of the documented reaction request. This also lets the product evolve its annotation rubric without changing the inference interface.
NeMo Agent Toolkit’s Weave integration supports explicit user attribution through context metadata fields trace_user_name, trace_user_email, and trace_user_id, set from an authentication callback. The documented priority is name, then email, then ID, with anonymous as fallback. Every attribution field you set is written to the Weave call summary. If attribution needs only an opaque ID, set trace_user_id and avoid populating the identifying alternatives. User attribution helps analysis; authorization to submit feedback on a particular tenant’s response still belongs to your authenticated application boundary.
Preserve the response’s context
Define application fields for feedback_id, response_id, session_id, turn number, submission time, trace reference, and workflow and prompt revisions. These are your schema choices, not additional toolkit API parameters. Have the client submit your application’s response_id; your authenticated application service checks ownership, maps it to the stored trace ID, and calls /feedback itself. The documented request carries only observability_trace_id and reaction_type, so exposing the worker route to clients unmediated lets the client choose which trace it rates. Mediated this way, your application service, not the client, chooses the trace that /feedback rates.
Capture the version of the response the user actually saw. If regeneration creates a second answer, give it a different response ID. Otherwise a delayed thumbs-down can be joined to the replacement answer and turn a successful repair into an apparent failure. For a multi-turn retrieval assistant, retain controlled references to the relevant turn history, retrieval source revisions, and tool outcomes. Store enough to explain the answer; copying an entire conversation into every feedback record increases privacy exposure and storage cost.
Feedback can arrive after execution finishes or after a trace expires. Keep a durable association record with the necessary response and release identifiers. Mark an unavailable trace explicitly; do not substitute the session’s most recent trace. A feedback record with incomplete context can still count toward a reported experience problem, but should not become a replayable regression case until its context is reconstructed and reviewed.
Make collection measurable
Keep a collection ledger that distinguishes displayed feedback opportunities, submitted responses, accepted records, and records awaiting review. Give a submission a stable application feedback_id so a transport retry does not become another vote. If your interface permits changing a rating, retain its revision history and define which revision counts in analysis.
Route records with missing associations or conflicting corrections to a review queue with an explicit status. Check accepted records for a resolvable response reference and rubric version. A high reaction count with many unresolved references measures collection activity, not an actionable improvement dataset. See 3.3.2 for privacy review and sampling before dataset export.
Key distinctions and trade-offs
Choose the signal according to the question it can answer:
| Signal | Deciding condition | Mechanism | Cost or limitation |
|---|---|---|---|
| Rating | You need a low-effort experience signal | Weave POST /feedback reaction, or an application-defined rating scale | High volume can still leave the cause unknown |
| Diagnostic label | You need to group reported failure types | Versioned application enum and annotation rubric | Reviewer disagreement requires adjudication |
| Correction | You need a candidate expected outcome | Proposed replacement, supporting source, and review status | Domain verification costs more than collecting a click |
Trace identity identifies execution; session identity groups a conversation; response identity identifies the artifact being judged. None substitutes for the other. More context improves diagnosis but raises retention and access-control costs, so use controlled references and an explicit export decision.
Architecture / decision scenario
A telecom customer assistant answers 40,000 billing questions a day. Its existing application rating store records 800 negative reactions, but only 240 carry a reason and only 90 include a proposed correction. The product team proposes turning all 800 reactions into “wrong answer” labels to accelerate prompt tuning.
The evaluation owner rejects that conversion: a negative reaction does not identify whether the invoice calculation, explanation, latency, or refusal caused dissatisfaction. The platform team enables the Weave worker for trace-linked reactions and stores the separate label and correction records against an application response_id. A delayed rating on answer version one remains attached to that answer after a regenerated version two appears.
Billing specialists review the 90 proposed corrections against the referenced invoice rules. Approved corrections become candidate expected answers; rejected and disputed proposals retain their submitted form and review status. The collection ledger shows 800 reactions, 240 diagnostic submissions, and the actual number of approved corrections separately. This costs specialist time, but avoids optimizing toward unsupported labels. The platform owner investigates unresolvable trace joins; the evaluation owner accepts only records with sufficient context for the intended analysis.
Common exam traps
- Treating a thumbs-down as proof of factual error.
- Treating a user-supplied correction as an approved reference answer.
- Assuming the Weave reaction endpoint is available on every FastAPI worker.
- Associating delayed feedback with the most recent answer in a session.
- Confusing user attribution with authorization to access a response.
- Counting duplicate submissions as independent opinions.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit” and “AIQ Toolkit.” The documented product is NeMo Agent Toolkit. When an exam item uses the guide’s name, recognize the toolkit role, then distinguish the documented Weave reaction mechanism from application-owned labels and corrections.
What to remember
Capture sentiment, diagnosis, and proposed truth separately. Join each to the response actually displayed, its execution trace, and sufficient session context. Review corrections before promoting them into evaluation references, and measure whether collected records are usable rather than counting clicks alone.
Knowledge check
A billing assistant’s feedback store holds 1,200 thumbs-down reactions. Of them, 300 also carry a reason chosen from the product’s versioned failure vocabulary; the other 900 carry nothing else. The team is building a set of factual-error cases for prompt tuning. How should the 900 bare reactions be recorded?
incorrect answer labels under the current rubric versionWith a thumbs-down, a user types what they say the correct refund amount was. The evaluation team builds regression cases only from approved reference answers. How should the application store the typed amount?
A multi-tenant web client receives NeMo Agent Toolkit answers through your authenticated application service. You enable the Weave FastAPI plugin worker so that reactions reach POST /feedback linked to traces. A tenant must be able to rate only its own responses. Which flow meets the requirement?
response_id ’s owner, then calls /feedback/feedback with the trace ID returned alongside each answer/feedback after the auth callback sets trace_user_id/feedback with the trace ID of the session’s latest answerobservability_trace_id and reaction_type , so a client that calls the route itself chooses which trace it rates. Mediated by the service, ownership is checked and the response is mapped to its stored trace ID before the call. trace_user_id attributes feedback to a user without authorizing access to a response, and the session’s latest answer need not be the one the user rated.A user gives a thumbs-down to answer version one, then regenerates, and version two resolves the issue. The rating reaches your feedback service 30 seconds after version two is displayed, in the same session. Which design keeps the rating on the answer it judged?
response_id of the displayed versionA NeMo Agent Toolkit deployment with Weave tracing must attribute feedback to users for analysis, and the privacy policy allows Weave to hold only an opaque user identifier. Which configuration fits?
trace_user_* fields in the authentication callbacktrace_user_id as an extra field in each /feedback requesttrace_user_id in the authentication callbacktrace_user_* field and let feedback fall back to anonymous/feedback uses, so setting name and email still sends them. The documented reaction request carries only the trace ID and reaction type, and leaving every field unset gives up the attribution the analysis needs.