DPO Tic-Tac-Toe: Preference Learning with NeMo Customizer

Why it matters

Preference training is only as good as the comparisons it learns from. This example builds those comparisons from a scorer you can inspect, so you can see where pairs come from before any weights change.

What to practise

Run nat eval first: a test-time compute pipeline generates several candidate moves per turn, scores each with alpha-beta search and records them as intermediate steps. Read the dpo_traj_builder settings — exhaustive_pairs, min_score_diff, max_pairs_per_turn — that turn those candidates into chosen and rejected pairs. Then run nat finetune to submit the job to NeMo Customizer, and compare win, loss and draw rates before and after training. The example assumes a running NeMo Microservices platform — Customizer, Entity Store and Datastore — whose documentation NVIDIA has moved to its archived doc set, so confirm you have that platform before you start.

How it fits

This is the DPO side of 5.5.2’s choice between DPO and GRPO. Compare the scorer-made pairs with 5.5.3’s warning that reward defects survive deduplication and score thresholds.