NVIDIANVIDIA-Certified Professional: Agentic AI2026-1010.2 Design structured feedback loops that guide iterative agent improvements.

Carry production feedback into re-evaluated releases

Completed by the knowledge check

Learning objective

Run a data-flywheel loop from production feedback to re-evaluated releases.

Core idea

A data flywheel turns production experience into a change whose outcome you measure. Logging more conversations is only the input. The loop closes when an accountable owner checks whether the released change repairs the targeted problem and whether the next feedback cycle confirms that result.

NVIDIA’s Data Flywheel Foundational Blueprint uses production logs, user feedback, and expert labeling to find smaller, cheaper candidate models that match production-model outputs for a given workload_id. Its focus is reducing inference cost and latency while preserving quality. The repository is deprecated, retained for reference, and advises against new production use. Use its experiment structure to reason about the loop; the evaluation stage can use NeMo Agent Toolkit’s nat eval, as described below.

See 10.2.1 for the diagnostic taxonomy, 3.3.2 for dataset curation, and 8.4.3 for release artifact promotion. Connect their outputs into one cycle: feedback supports an experiment, results support a release candidate, and production verification supports closure.

How it works

Carry a reviewed problem into an experiment

Start with an adjudicated feedback cohort and a repair hypothesis. For example: “The assistant omits the refund deadline because its task instructions do not require one.” Name the affected workload, accountable owner, proposed intervention, expected outcome, and stop condition in an experiment record. These are application-defined artifacts.

The reference blueprint’s log schema uses timestamp, workload_id, client_id, request, and response. Its workload_id identifies a single request type or agent node, while client_id identifies the application or deployment producing traffic. Preserve that distinction when joining feedback to experiments: pooling an agent’s summarizer and tool router under one workload can hide which component changed.

Keep the original feedback cohort identifiable throughout the cycle. Link datasets, experiments, and release decisions to the reviewed problem, including reasons for excluding reports. Repeated reports about the same failed case should not become repeated proof that the candidate generalizes. Dataset deduplication and privacy admission remain the responsibilities described in 3.3.2.

Compare interventions before spending training compute

The blueprint runs base, icl, and customized experiments for each candidate NIM, seeking high-scoring smaller models. NeMo Customizer runs the LoRA fine-tuning job, and NeMo Evaluator runs the LLM-as-judge comparisons. The following table separates their decisions:

Experiment	Mechanism in the reference	Deciding question	Cost or limitation
`base`	Replay production prompts against a smaller candidate model	Does a cheaper substitute match the production model on this workload?	Inference and evaluation cost; source responses may contain errors
`icl`	Add few-shot examples drawn from traffic	Do relevant examples improve behavior without changing weights?	Larger prompts and example-selection work
`customized`	Evaluate a LoRA fine-tune using the base prompts	Does customization justify its added preparation and training cost?	Training, adapter management, and separate re-evaluation

The reference requires at least 50 records, uses an evaluation split of 20 records by default, and selects 1–3 ICL examples. These are reference implementation settings, not a guarantee of representative evaluation or adequate statistical power.

Treat these as candidates, not a mandatory progression to training. An accepted knowledge repair may need none of them. A prompt intervention might resolve the feedback without changing the model. See 8.4.1 for choosing the intervention and 5.5.1 for reviewed example reuse; here you preserve a comparable record of why the chosen candidate entered the loop.

The blueprint uses LLM-as-judge similarity scores in the range zero to one to identify promising candidates. Similarity to a production response is not independent proof of correctness. If the complaint concerns that response’s error, reproducing it more cheaply reinforces the failure. Have a domain reviewer approve the expected outcome before interpreting candidate similarity as improvement.

Re-evaluate the whole candidate, then decide

A maintained evaluation path is NeMo Agent Toolkit’s nat eval, whose full runtime requires nvidia-nat[eval]. Configure the selected dataset with eval.general.dataset._type and file_path, choose evaluators under eval.evaluators, and put each run’s artifacts in eval.general.output_dir. The command executes the workflow over the configured data and evaluates its output.

Attach the run’s workflow_output.json and config_effective.yml to its experiment record; see 3.1.4 for evaluation artifacts and the remote --endpoint boundary, and 5.5.3 for adaptation regression tests.

Record three decisions separately: whether the targeted feedback cases improve, whether unaffected tasks remain acceptable, and whether operating cost and latency fit the service’s constraints. NIST MANAGE 4.2 connects measurable improvement activities to system updates. A failed regression check keeps a candidate in investigation even when its target-category score increases.

Verify closure after release

NIST MANAGE 4.1 describes post-deployment monitoring that captures and evaluates input, alongside incident response and change management. Connect the released candidate back to the feedback queue with a closure record: targeted issue, tested change, release decision, observed outcome, and responsible owner.

A merged repair can be “evaluated” without being “verified in production.” Keep those dispositions separate. After the release, review the targeted task slice and solicit follow-up from affected users where appropriate. If the same failure recurs, reopen the issue with its release context. Silence alone is insufficient: fewer reports can reflect fewer users or less use of the feedback control. See 8.3.2 for distinguishing release effects from changing traffic.

A failed regression check keeps a candidate in investigation, and a failure that recurs after release reopens the issue.

Key distinctions and trade-offs

Feedback volume versus useful hypotheses: A large complaint queue does not justify running every possible model and prompt combination. Prioritize accepted hypotheses by impact and available review capacity. Include evaluation inference and expert review in the experiment budget, not only training time.

Candidate discovery versus release authority: Automated experimentation narrows the search. The blueprint explicitly retains human production-promotion decisions. The release owner needs task acceptance, regression results, and operating constraints, not just a ranking.

A repeated loop versus automatic learning: The flywheel can repeatedly improve retrieval, instructions, examples, or model weights. It needs neither unchecked training on every conversation nor automatic deployment after each evaluation.

Architecture / decision scenario

A travel company’s customer assistant produces itinerary-change summaries. Reviewed feedback identifies omitted refund deadlines. The product owner has 160 accepted cases, a separate 400-case regression set, and a weekly experiment budget of 12 GPU-hours. Illustrative acceptance rules require at least 95% deadline inclusion on the targeted cases, at least 97% regression success, and p95 latency below 3 seconds.

A candidate with reviewed few-shot examples passes 154 of 160 targeted cases and 390 of 400 regression cases, with 2.7-second p95 latency. Its examples add 600 prompt tokens to a 1,400-token input, staying below the service’s 2,200-input-token ceiling per request. A fine-tuned candidate uses the 1,400-token input and passes 158 and 376 cases respectively, with 2.4-second p95 latency. The tempting choice is the fine-tune because it improves targeted accuracy and reduces recurring input-token and latency costs.

The release owner instead advances the example-based candidate: 96.25% targeted success, 97.5% regression success, 2,000 input tokens, and 2.7-second p95 latency meet all four limits. The fine-tune’s 94% regression success fails despite its 98.75% targeted result. Its trial consumed 9 GPU-hours and the example-based trial consumed 2, leaving only 1 of the weekly 12 GPU-hours. Repeating the 9-GPU-hour training trial cannot fit this week. Accepting the bounded recurring prompt overhead is preferable to releasing a regression or funding an unbudgeted retry. These figures are hypothetical constraints and measurements, not NVIDIA benchmarks.

After controlled promotion, the product owner keeps the original cohort open for production verification. Follow-up checks must confirm deadline inclusion in completed tasks; a drop in complaints alone cannot close the issue. Recurring omissions reopen that cohort’s experiment record, completing another turn of the loop.

Common exam traps

Calling a log export or a one-time fine-tune a closed feedback loop.
Using the production model’s answer as unquestioned ground truth for its reported errors.
Promoting the highest similarity score without independent task and regression checks.
Treating saved endpoint evaluation settings as the remote agent’s complete configuration.
Recommending the deprecated blueprint as a maintained production service.

Where the exam guide differs

The study guide lists “NVIDIA Data Flywheel Glossary” and “NVIDIA Agent Intelligence Toolkit” among its suggested readings. The referenced Data Flywheel Foundational Blueprint carries an April 2026 deprecation notice, and the toolkit documentation uses NeMo Agent Toolkit. Read a flywheel exam item as a production-data-to-experiment-to-evaluation loop that can identify smaller, more efficient models while preserving quality; the objective does not require deploying the deprecated reference or automatically promoting its candidates.

What to remember

Carry the reviewed problem through the experiment, evaluation artifacts, release decision, and production closure. Compare cheaper interventions with customization, retain independent correctness checks, and keep release ownership explicit. Production feedback becomes improvement only when the resulting behavior is tested and verified.