NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.3 Collect and integrate structured user feedback for iterative improvements.

Curate feedback without hiding missing populations

Completed by the knowledge check

Learning objective

Curate privacy-aware feedback datasets and control sampling bias.

Core idea

Production feedback is a selected view of production behavior. People who respond, tasks that reach a visible answer, and cases that survive privacy filtering can differ from the workload you need to evaluate. A large dataset can therefore give a precise answer to the wrong question.

Use the trace-linked records from 3.3.1 as inputs to a curated dataset, with explicit inclusion rules, privacy transformations, and population coverage. NIST AI RMF MEASURE 2.2 addresses representation in human-subject evaluations, while MEASURE 2.10 and 2.11 address privacy risk and bias. Turn those concerns into a dataset manifest and a sampling ledger that an evaluation owner can inspect. See 10.2.2 for the broader production-to-release loop.

How it works

Define the permitted dataset before exporting

Specify the dataset’s purpose, authorized readers, approved storage location, retention period, and permitted fields in an application-owned dataset manifest. Record who approves use of feedback for evaluation and whether training is separately permitted. An operational need to diagnose an answer does not automatically justify copying its conversation into a training corpus. NIST’s Generative AI Profile describes purpose and participation risks around personal data, including risks from combining information across sources.

Separate restricted source records from derived evaluation examples. Keep only the task context needed to test the intended behavior and controlled references for authorized investigation. A pseudonymous session key can preserve duplicate detection without putting an employee name in the evaluation input. Because a join to the restricted source may remain possible, treat that key as sensitive rather than declaring the dataset anonymous.

Record transformations and exclusions by reason. If privacy filtering removes most examples from one task class, that is both a privacy outcome and a coverage change. Preserve aggregate counts in the sampling ledger so the evaluation report does not silently generalize from the surviving tasks.

Redact before telemetry export, then review the dataset

NeMo Agent Toolkit’s Weave tracing configuration exposes redact_pii, redact_pii_fields, and redact_keys under the configured general.telemetry.tracing exporter. redact_pii defaults to false; enabling it redacts recognized personally identifiable information before traces are sent to Weave. redact_pii_fields defaults to all supported entities; to narrow it, select types such as EMAIL_ADDRESS, PHONE_NUMBER, and PERSON. redact_keys extends sensitive-key redaction beyond the defaults api_key, auth_headers, and authorization.

This integration uses presidio-analyzer and presidio-anonymizer, installed with nvidia-nat[weave]. Configure the exporter before collecting the traces you intend to analyze. Do not wait until a dataset download to remove secrets that have already left the runtime.

Review exposure of attribution fields such as trace_user_id separately; see 3.3.1. Apply the dataset’s field allowlist and privacy checks to corrections, source excerpts, and trace attachments across their separate data paths. Redaction can also remove information required to judge correctness: replace a sensitive case with an approved surrogate preserving the failure mechanism, or exclude it and declare the coverage gap. Do not label an unusable example as a model failure.

Track selection and nonresponse

Define the target population by task, language, interaction surface, release, and time window using dimensions your organization is permitted to collect. Count eligible interactions, feedback invitations, submissions, privacy-approved records, and accepted examples for each relevant slice. These denominators expose where data disappears.

Voluntary reactions are useful for finding failure cases, but response propensity can differ across groups. Randomly sampling only volunteers does not recover users who never submitted feedback. Supplement feedback with an approved random sample of eligible production interactions and domain review, or a representative user study when collection rules require it. NIST’s Generative AI Profile recommends examining the representativeness and balance of evaluation sources, including coverage affected by the digital divide.

Keep a population-oriented evaluation set separate from a failure-enriched challenge set. Oversample rare failures to investigate them, but retain the sampling design. Report slice results and counts before aggregating. See 3.2.2 for slice comparisons, deployment weighting, and uncertainty. If you weight slice rates by deployment proportions, document the proportions and assumptions; weighting observed language groups does not correct unknown differences between responders and nonresponders within a language.

Produce an executable, reviewable dataset

Assign stable case IDs and keep related examples together when separating development cases from a held-out evaluation set. Repeated feedback on one meeting or document can otherwise make one underlying case appear in both sets. Store the split assignment, source release, annotation rubric, reviewer status, and transformation version in the dataset manifest. These are application artifacts, not NeMo Agent Toolkit configuration keys.

For an approved question-and-answer dataset, NeMo Agent Toolkit’s evaluation documentation illustrates entries with id, question, and answer. Select the exported file through eval.general.dataset, using _type: json and file_path. Install nvidia-nat[eval] for the full evaluation runtime and invoke nat eval --config_file=EVAL_CONFIG_PATH, replacing EVAL_CONFIG_PATH with your evaluation configuration path. The dataset path should point to the approved export rather than unrestricted production logs.

Inspect workflow_output.json, which includes the expected answer, generated_answer, and intermediate_steps. Apply the same storage and access review to these outputs: a sanitized input file does not establish that generated output or tool observations are suitable for broad distribution. The dataset owner signs off on the export; the evaluation owner signs off on what population its results describe.

Key distinctions and trade-offs

Choose a collection according to the conclusion you need:

Collection	Deciding condition	Control artifact	Cost or limitation
Voluntary feedback	Discover reported experience problems	Submission and response-rate ledger	Nonresponders remain unobserved
Population-oriented sample	Estimate behavior across an intended workload	Sampling frame, slice counts, and inclusion rules	Requires approved access and annotation effort
Failure-enriched set	Stress a known weakness	Failure labels and explicit enrichment criteria	Raw aggregate is not a production failure rate

Privacy transformations reduce exposure but may also change the task being tested. Representative coverage and useful diagnostic detail compete with collection restrictions. Make that trade-off visible in the manifest rather than assuming a larger export is a better dataset.

Architecture / decision scenario

An enterprise meeting companion processes 10,000 meetings weekly: 8,000 in English and 2,000 in Spanish. Voluntary feedback arrives for 1,600 English meetings and 40 Spanish meetings: response rates are 20% and 2%. A team proposes evaluating on 1,000 randomly selected feedback records because random selection sounds representative.

The evaluation owner rejects that workload estimate: its sampling pool already underrepresents Spanish meetings. Instead, an approved review sample covers 200 eligible meetings per language, selected independently of whether feedback was submitted. The owner chooses the per-language sample size for the precision the comparison needs and reports each slice rate’s uncertainty with meetings as the sampling unit; see 3.2.2. If reviewed error rates are 5% for English and 20% for Spanish, their equal-sized sample averages 12.5%; deployment weighting gives 8%. Report both slice rates and the 8% estimate with their sampling basis; the balanced-set average is not workload performance.

The privacy owner permits sanitized meeting-summary cases but excludes raw attendee identities and confidential agenda details. The dataset manifest records exclusions by language and task; cases that lose essential meaning are replaced with approved surrogates or marked uncovered. Reported failures form a separate challenge set. This uses more reviewer time than replaying volunteers’ logs, but exposes the language gap while keeping data use within the approved purpose.

Common exam traps

Assuming a random sample of volunteers represents all eligible interactions.
Reporting an enriched failure set’s average as the production failure rate.
Assuming pseudonymous trace references make a dataset anonymous.
Assuming Weave PII redaction is enabled by default or covers every application data path.
Removing sensitive cases without reporting which slices lost coverage.
Mixing related source cases across development and held-out evaluation sets.

Where the exam guide differs

The study guide’s readings name “NVIDIA Agent Intelligence Toolkit” and “AIQ Toolkit.” NeMo Agent Toolkit documents the tracing-redaction and evaluation mechanisms used here. Interpret the guide’s terminology as that toolkit role; distinguish exporter redaction from the dataset curation and population analysis your team must own.

What to remember

A feedback dataset needs an approved purpose, privacy controls, and a declared sampling basis. Track what was eligible, submitted, excluded, and reviewed. Separate population estimates from failure discovery, and preserve enough context to test the same task after sanitization.