NVIDIANVIDIA-Certified Professional: Agentic AI2026-105.5 Adapt reasoning strategies based on prior experiences and feedback.
Test adaptations for regressions, reward defects, and overfitting
Completed by the knowledge check
Learning objective
Verify adaptation against regressions, unreliable rewards, and overfitting.
Core idea
Adaptation makes an agent better at what its learning signal rewards. Verification asks whether that improvement transfers to independent task outcomes without breaking behavior the system already needs. A rising reward curve alone cannot answer either question.
For in-context adaptation, the candidate includes its examples, retrieval selection, and reflected instructions. For DPO or GRPO, it also includes the trained policy and training-data provenance. Compare the complete candidate against the accepted baseline. See 3.2.1 for controlled comparisons and 3.2.2 for slices and run-to-run variance; the additional problem here is that adaptation can contaminate its own tests or learn to exploit their scorer.
Produce an adaptation review record that separates three findings: existing capabilities that regress, defects in the preference or reward signal, and failure to generalize beyond the material used to adapt. Each finding requires a different repair. NIST AI RMF 1.0 provides the documentation basis: MEASURE 2.1 calls for recording test sets, metrics, and testing tools; MEASURE 2.5 calls for documenting limits to generalization. Use the review record to preserve those details for the adaptation being considered.
How it works
Keep evaluation outside the adaptation loop
Reserve cases that cannot enter training pairs, prompt demonstrations, or reflected-instruction examples. Separate them by the entity that makes cases related: a customer account, document family, business process, or time period. A paraphrase of a training case is not necessarily an independent test. Record those relationships in the dataset’s provenance so the release owner can audit the separation.
In the toolkit’s experimental finetuning harness, nat finetune exposes --validation_dataset, --validation_interval (every five epochs by default), and --validation_config_file. ART integration documentation describes running a separate validation dataset and preserving training and validation progression in reward_history.json and reward_plot.png. These mechanisms help reveal generalization differences; their existence does not guarantee the datasets are independent.
Do not retrieve validation answers as demonstrations for the very cases being scored. Freeze the example collection and selection rules for a candidate, or explicitly include changes to them in the adaptation being evaluated. Otherwise a “model improvement” may be an easier prompt that contains the answer. This applies even when no model weights changed.
Validation used to select a checkpoint is part of development. Repeatedly adjusting a policy, example collection, or reward against the same validation failures makes that set less independent. Keep a separate release assessment whose cases and adjudications do not guide each training iteration.
Test the reward as a component
The toolkit’s finetuning.reward_function.name selects an evaluator under eval.evaluators. That evaluator is part of the optimization target. Before trusting its output, assemble a small adjudicated reward-check dataset containing clear successes, clear failures, and plausible shortcuts. Record both the scorer’s result and the domain owner’s expected ordering or outcome.
For a report-writing agent, examples might include a well-written report with incorrect totals and an awkward report with verified totals. If the scorer prefers polish to numerical correctness, DPO pairs derived from its scores and GRPO updates both inherit the defect. Repair the scoring rubric or deterministic check, then regenerate affected labels and comparisons. Training longer is not the remedy.
The DPO builder’s min_score_diff can suppress small score differences, and deduplicate_pairs in the NeMo Customizer trainer can remove exact duplicate comparisons. Neither control verifies the scorer’s criterion. A large margin can still confidently prefer the wrong answer.
Likewise, GRPO’s within-group variation proves only that scores differ. An evaluator returning arbitrary values supplies variation without usefulness. The ART troubleshooting guidance explicitly includes verifying reward calibration when training does not improve. Compare scorer decisions with independently checked task outcomes, including attempts that imitate a successful response while omitting the required action.
Diagnose the improvement pattern
Read training and validation curves together, then inspect the corresponding task failures. Training reward rising while independent validation deteriorates suggests overfitting or a distribution mismatch worth investigating. High reward on both sets with poor adjudicated outcomes points toward a shared reward defect. A gain on the adapted task accompanied by worse performance on a retained capability is a regression, even when the overall average rises.
Run NeMo Agent Toolkit’s nat eval against the candidate and baseline with the full evaluation runtime, nvidia-nat[eval]. Keep workflow_output.json to inspect generated_answer and intermediate_steps, and retain config_effective.yml to identify the configuration actually evaluated. The evaluator-specific output files exist only for evaluators configured under eval.evaluators; do not infer a safety or quality result from a missing file.
For remote evaluation using --endpoint, the saved configuration records evaluation settings, not the server’s workflow configuration. Preserve the remote policy and workflow revision separately. A URL that serves a different checkpoint tomorrow cannot by itself identify the candidate that passed today.
Key distinctions and trade-offs
Use the failure pattern to decide which component to repair:
| Finding | Deciding observation | Inspection mechanism | Repair |
|---|---|---|---|
| Regression | Adapted task improves while a required retained capability declines | Separate evaluator outputs and failed cases in workflow_output.json | Reject the candidate; revise the adaptation and rerun the retained-capability cases |
| Unreliable reward | Reward disagrees with independently adjudicated task success | Reward-check dataset compared with finetuning.reward_function.name output | Repair the evaluator and regenerate affected training signals |
| Overfitting | Training gains fail on independent cases | Training/validation progression in ART integration’s reward_history.json, plus release assessment | Revisit data separation, broaden examples, or choose an earlier candidate before reassessment |
Separate assessment costs extra inference and domain review. It is necessary because the scorer being optimized cannot also be the only authority on whether optimization worked. Keep training reward, task success, and retained-capability results separate in the adaptation review record; averaging them hides which decision failed.
Architecture / decision scenario
A retail analytics team adapts an agent that produces weekly inventory reports. It holds out 400 store-week cases, including 100 requiring an explicit insufficient-data response. The product owner’s release rule allows at most two additional errors on that retained behavior relative to the accepted policy.
The baseline succeeds on 320 of 400 cases and mishandles four of the 100 insufficient-data cases. A candidate succeeds on 344 overall but mishandles 15 insufficient-data cases. Its higher aggregate success rate tempts the team to promote it. The release owner rejects it because 11 additional errors exceed the allowed two; the adaptation improved one behavior by damaging another required behavior.
Meanwhile, training reward rises from 0.62 to 0.91, but held-out reward stays near 0.64. The ML owner inspects reward_history.json and checks for repeated store templates across the split. A domain reviewer also creates 40 reward-check cases, including fabricated but plausible totals. The reward evaluator scores nine fabricated reports as successes.
The team has two repairs, not one: correct and recalibrate the reward, and remove related store templates from the adaptation/assessment overlap. It regenerates affected comparisons and evaluates a revised candidate against both inventory accuracy and insufficient-data behavior. Neither choosing the highest-reward checkpoint nor increasing training epochs addresses the observed defects. All numbers are scenario measurements and release criteria, not toolkit defaults.
Common exam traps
- Treating training reward as an independent measure of production quality.
- Allowing held-out answers to enter retrieved demonstrations.
- Assuming an automatic train/validation split separates related trajectories.
- Using pair deduplication or a score-margin filter as reward calibration.
- Accepting a higher overall average despite a prohibited capability regression.
- Assuming remote evaluation artifacts identify the deployed model and workflow revision.
What to remember
Verify the adapted behavior, the behaviors it must preserve, and the signal that taught it. Independent cases detect failed transfer; adjudicated reward checks expose the wrong target; retained-capability tests catch harmful trade-offs. Repair the component the failure implicates before training or promoting another candidate.
Knowledge check
A contract-summary agent is adapted with new examples and a retrained policy. The held-out set has 500 cases, 80 of which require the agent to decline because the contract is redacted. The release rule allows at most three additional errors on those 80 cases relative to the accepted baseline. The baseline succeeds on 410 cases with five decline errors; the candidate succeeds on 445 with fourteen. What should the release owner do?
During finetuning, training reward rises from 0.55 to 0.88, while reward on the validation dataset stays near 0.57. The split was made by randomly sampling trajectories, and both sets contain reports from the same customer accounts using the same templates. What should the ML owner do next?
min_score_diff so that only stronger pairs are keptA report-writing agent’s reward is high on both the training and the validation sets. A domain reviewer then adjudicates a sample and finds that the evaluator scores reports with fabricated but plausible totals as successes. The team trains with DPO pairs derived from that evaluator’s scores. What is the right repair?
min_score_diff and retrain on the remaining pairsdeduplicate_pairs and retrain on the unique pairsA team evaluates a candidate with nat eval --endpoint against a staging URL and archives the evaluation output directory as release evidence. A week later, the URL serves a different checkpoint, and an auditor asks which policy passed. What should the team have preserved alongside the evaluation output?
workflow_output.json file with each sample’s stepsconfig_effective.yml--endpoint , the saved configuration records the evaluation settings — dataset, evaluators and endpoint URL — but not the workflow configuration running on the remote server. A URL that serves a different checkpoint later cannot identify the candidate that passed, so the policy and workflow revision have to be recorded separately: config_effective.yml , the per-sample steps and the evaluator outputs describe the evaluation without naming what served it.A team evaluates a revised example collection for an agent; no model weights change. Between the baseline run and the candidate run, the store the retriever draws demonstrations from was loaded with the validation cases and their reviewed answers. Validation success rises from 71% to 93%. What should the team conclude?