NVIDIANVIDIA-Certified Professional: Agentic AI2026-105.5 Adapt reasoning strategies based on prior experiences and feedback.
Choose DPO or GRPO from your feedback signal
Completed by the knowledge check
Learning objective
Adapt from preference and reward signals (DPO, GRPO).
Core idea
Preference optimization changes the policy that produces responses. Unlike the examples supplied in 5.5.1, the learning signal enters a training objective and updates model weights. Choose the method according to the comparison you can trust: a preferred and rejected response for the same context, or rewards for several rollouts of the same task.
Direct Preference Optimization (DPO) learns from ranked response pairs relative to a frozen reference policy. Group Relative Policy Optimization (GRPO) samples multiple rollouts, scores them, and learns from their relative performance within a group. Neither method establishes that the preference or reward measures the business outcome correctly. That verification belongs to 5.5.3.
NVIDIA documents both methods in NeMo RL and workflow-level integrations in NeMo Agent Toolkit. The toolkit’s finetuning harness is explicitly experimental and may introduce breaking changes. Treat its component configuration as a versioned integration, rather than assuming that every training backend accepts the same data or controls.
How it works
Use DPO when you can rank alternatives
For a shared context, DPO receives a chosen response and a rejected response. NeMo RL describes increasing the chosen response’s probability and decreasing the rejected response’s probability relative to a frozen reference model. It does not require training a separate reward model. “Preferred” is a relative label: the better of two incomplete answers can still be unsuitable for deployment.
NeMo RL’s documented preference format has context, completions, and task_name. Each completion has a rank and a completion; the lower rank is preferred. Its DPO implementation supports exactly two completions, each containing a single response. A multi-turn context is allowed, but this format is not an arbitrary multi-action trajectory trainer. Keep the unit of comparison aligned with the implementation you select.
The toolkit also documents a NeMo Customizer integration through nvidia-nat-nemo-customizer. It submits jobs to a deployed NeMo Customizer instance on the NeMo Microservices Platform and requires its Entity Store and Datastore. Assign an owner to that platform; this path has service prerequisites beyond NeMo RL’s locally launched training. Under trajectory_builders, _type: dpo_traj_builder collects scored candidates emitted by workflow test-time computation. Its ttc_step_name must match the workflow’s emitted candidate-step name. The builder converts higher-scored and lower-scored alternatives into preference pairs. Human annotations are therefore one source of preferences, not a requirement that every pair be manually written.
With exhaustive_pairs: true, the builder makes all strictly ordered pairs; with false, it chooses best versus worst per turn. min_score_diff filters weak distinctions, while max_pairs_per_turn bounds the number retained. Three candidates scoring 0.9, 0.7, and 0.5 produce three ordered pairs in exhaustive mode and one in best-versus-worst mode. More pairs reuse the same underlying observations; they do not create three independent user judgments.
The documented trainer uses _type: nemo_customizer_trainer, and its adapter uses _type: nemo_customizer_trainer_adapter. The trainer’s deduplicate_pairs removes duplicate prompt/chosen/rejected triples. This reduces repeated records; it does not decide whether two paraphrases leak the same task into training and validation.
Use GRPO when you can score fresh rollouts
GRPO compares multiple responses generated for the same input. Within each group, GRPO learns to increase the probability of high-scoring responses and decrease the probability of low-scoring responses. The toolkit explains that grouping trajectories by example ID enables this comparison without training a separate value function.
In NeMo RL, an Environment runs as a Ray remote actor: it accepts state/action history and returns updated state and rewards. NVIDIA documents Math, Code, and Reward Model environments; the domain owner’s scoring logic belongs in that environment. grpo.num_prompts_per_step sets the prompt batch for each training step.
The toolkit’s OpenPipe ART integration requires nvidia-nat-openpipe-art and a running ART server. It registers openpipe_art_traj_builder, openpipe_art_trainer_adapter, and openpipe_art_trainer under trajectory_builders, trainer_adapters, and trainers. In finetuning, the corresponding trajectory_builder, trainer_adapter, and trainer names select those components. reward_function.name references an evaluator defined under eval.evaluators.
The builder’s num_generations controls rollouts per example. Increasing it gives the optimizer more within-task comparisons, while increasing generation cost. Most GRPO training is on-policy: the toolkit regenerates trajectories at each epoch so that experience reflects the policy being updated. A fixed archive of preference pairs and a stream of fresh policy rollouts have different collection costs.
A group in which every trajectory receives the same reward provides no relative preference signal. The harness documents finetuning.curriculum_learning.min_reward_diff for excluding low-variation groups when curriculum learning is used. If all scores are zero because the evaluator is broken, filtering the groups only hides the symptom. Verify reward calibration; the ART troubleshooting page also recommends checking calibration when rewards fail to improve.
Each epoch regenerates a group of rollouts for the same input, and only a group whose rewards differ gives GRPO a signal to learn from.
Key distinctions and trade-offs
Use the available signal to select the training path:
| Method | Deciding condition | Documented mechanism | Main cost or risk |
|---|---|---|---|
| DPO | You can reliably rank two responses to the same context | NeMo RL preference completions with rank; toolkit dpo_traj_builder | Pair curation, reference-policy computation, and biased or weak preference labels |
| GRPO | You can repeatedly score sampled solutions to the same task | openpipe_art_traj_builder.num_generations and finetuning.reward_function.name | Repeated rollout inference and exploitation of an unreliable reward |
Both require a training budget and evaluation after the update. DPO avoids a separate learned reward-model pipeline, but pair generation can still require several candidate inferences and scoring calls. GRPO avoids a separate learned critic, but does not eliminate policy training or the cost of generating groups. Do not read “no reward model” or “no critic” as “no GPU cost.”
The ML training owner selects the algorithm and backend; the domain owner defines what a preference or reward means. The workflow owner selects the functions being optimized through finetuning.target_functions. Optimizing the final writer and optimizing the planner are different interventions even when both receive the workflow’s final success score.
Architecture / decision scenario
A commercial insurer builds a claims-review assistant that drafts recommendations without authorizing payouts. For explanation quality, reviewers have 6,000 context-matched preferred/rejected pairs. For arithmetic and rule application, a sandbox can score outcomes automatically. The team has a six-hour nightly training window; measured sandbox rollouts take eight seconds each, with four concurrent rollout slots.
The training lead chooses NeMo RL DPO for the reviewed explanation pairs, encoding each shared context with the reviewers’ preferred completion at rank: 0 and the rejected completion at rank: 1. The toolkit’s dpo_traj_builder is the alternative when the workflow emits scored test-time candidates, rather than importing these reviewer-labelled pairs. Replacing these labels with a “longer explanation wins” reward would discard the criterion the reviewers actually assessed. The preference records preserve the same context for both responses and the intended ranking.
For sandbox planning, the lead considers GRPO with four generations for each of 1,000 prompts. That is 4,000 rollouts and approximately 8,000 seconds of collection at the measured four-slot capacity, before training overhead. Eight generations would need approximately 16,000 seconds, leaving much less of the six-hour window for updates and validation. These are scenario measurements, not ART throughput guarantees.
The team chooses four generations initially and checks whether rewards distinguish alternative plans. It does not mix responses from different claims into one group. If the scorer cannot distinguish outcomes, the domain owner fixes the reward before another training run; increasing generations cannot repair a criterion that rewards the wrong behavior.
Common exam traps
- Confusing prompt-example reuse with updating model weights.
- Requiring a separately trained reward model for DPO.
- Grouping GRPO rollouts from unrelated prompts.
- Treating all preference pairs generated from one candidate set as independent feedback.
- Assuming every backend supports the same multi-turn training record.
- Presenting the toolkit finetuning harness as a stable, universal training interface.
What to remember
DPO needs trustworthy within-context rankings; GRPO needs useful reward differences among fresh rollouts of the same task. Match the data format to the backend, identify the workflow functions being trained, and account for collection as well as update cost. Better optimization of a weak signal can produce worse behavior.
Knowledge check
A claims team has 6,000 reviewer-labelled records, each holding a preferred and a rejected explanation for the same claim context. They want to update the model’s weights with NeMo RL on the reviewers’ judgments and to avoid adding a training stage beyond the policy update itself. An engineer proposes training a reward model on the records first. Which approach meets the requirement?
A team runs GRPO through the toolkit’s OpenPipe ART integration with num_generations: 4. To diversify each group, an engineer proposes assembling every group from rollouts of four different claims of similar complexity. For the harness’s on-policy GRPO to learn from relative performance, what should each group contain?
In a GRPO run through the toolkit’s finetuning harness, every group’s rewards are 0.0 after the first epoch. The evaluator is intended to reward correct claim decisions. A domain reviewer reading the trajectories finds that several of them reach the correct claim decision. What should the team do first?
min_reward_diff to exclude the flat groups from trainingnum_generations so that each group holds more rolloutsreward_function.name selectsA workflow emits three scored candidates per turn with distinct scores, and the toolkit’s dpo_traj_builder runs with exhaustive_pairs: true, min_score_diff: 0.0 and max_pairs_per_turn unset. Over 500 turns, a status report says the run produced 1,500 independent user judgments. Which statement describes the data accurately?
deduplicate_pairs makes independentfalse mode, and deduplication removes repeated prompt, chosen and rejected triples without making related pairs independent.Reviewers ranked four candidate responses for each context, and the team will train with NeMo RL’s documented DPO preference format. How must the team shape each training record?
rank valuerank valuerank valuerank is the preferred one; four ranked candidates have to become pairs for this backend. A multi-turn context is allowed, but the format is not an arbitrary multi-action trajectory trainer, and a single score is a reward rather than a preference.