Choose tuning or training from failures

Learning objective

Distinguish prompt and configuration tuning from model retraining, and define triggers for each.

Core idea

A production failure should trigger a diagnosis and a bounded experiment, rather than an automatic weight update. Changing instructions, decoding parameters, or the workflow’s model selection changes how an existing model is used. Retraining changes learned parameters and produces another model artifact. Both can change behavior, but their data requirements, compute costs, and release responsibilities differ.

NeMo Agent Toolkit implements automated configuration and prompt search through nat optimize. NeMo RL implements supervised fine-tuning through examples/run_sft.py. Run them as separate experiments. Neither a completed search nor a completed training run is sufficient reason to replace production.

Failure attribution is covered in 3.5.1. Here, use that attribution to define which job starts, what it may change, and when the team stops spending money on that path.

How it works

Define an actionable trigger

Write a team-owned trigger record with the affected task slice, baseline release, measurement window, failure count, likely cause, permitted changes, experiment budget, and responsible owner. A falling aggregate score is a symptom; it does not identify a training target. Retain the cases that establish the hypothesis and the evaluation configuration that reproduces them.

For example, repeated omissions of a required response field with correct retrieved facts suggest testing instructions and configuration first. Incorrect answers because an upstream catalog is stale call for repairing the catalog or retrieval path. Training on those answers would preserve the defect in another place. A capability failure that persists with correct context and tested instructions is a stronger reason to prepare supervised examples.

Define a separate efficiency trigger when acceptable quality is too expensive. A model-selection experiment might satisfy that trigger without training. A smaller trained candidate is another option when measured capability remains insufficient. Specify the quality floor before searching, so a cheaper but inadequate candidate cannot win by cost alone.

Bound configuration search

See 3.4.3 for operating the NeMo Agent Toolkit optimizer. Here the trigger record determines which declared OptimizableField settings and SearchSpace ranges enter the experiment. If the failure hypothesis concerns missing instructions, the workflow owner permits prompt changes and holds unrelated model and workflow settings fixed. A search without an actionable trigger is not started.

Budget each enabled search stage separately; numerical trials do not bound prompt search. The documented defaults are optimizer.numeric.n_trials: 20 and optimizer.reps_per_param_set: 3, so a numerical stage using those defaults requires 60 repeated trial evaluations before counting dataset cases and model or judge calls inside each evaluation. Leaving those settings unset still spends that budget.

The team’s experiment pipeline reads the approved trigger record, prepares the restricted configuration, and submits nat optimize --config_file CONFIG_PATH. Here CONFIG_PATH identifies that experiment’s configuration. The pipeline records the submission against the trigger and refuses another search for the same unresolved failure until the owner reviews its results. This is team-owned scheduling, not an optimizer feature.

Escalate to training deliberately

Start a training candidate when the trigger record identifies a repeatable learned behavior to improve, reviewed examples are available, and configuration experiments have failed or have an unacceptable operating cost. Record which experiments established that conclusion. There is no universal failure percentage at which retraining becomes correct.

NeMo RL’s SFT job accepts a YAML experiment configuration through uv run examples/run_sft.py --config CONFIG_PATH. Its dataset mechanisms include ResponseDataset, data.train.data_path, and data.validation.data_path. The training owner must prepare target responses and a separate validation set before scheduling the job. See 8.4.2 for preparing production-derived training candidates; see 5.5.2 for choosing preference or reward-based training.

The team’s pipeline submits the training job only after the owner approves its data snapshots and compute budget. It stops resubmitting when the authorized experiments are exhausted, when a required dataset is unavailable, or when the owner rejects the failure hypothesis. A recurring monitoring alert does not replenish the budget. Keep the release decision distinct: the trigger authorizes compute and evaluation, while the acceptance policy authorizes production use.

A trigger record routes a failure to repair, search or training, and only the acceptance policy authorizes production use.

Key distinctions and trade-offs

Choose the path by the defect it can repair:

Prompt search leaves model weights unchanged but can still damage tool use or safety behavior. Retraining can change many behaviors beyond the target slice. Neither path receives a lighter regression standard merely because the edit looks small.

A trigger also needs an exit decision. If configuration experiments cannot meet the target, the workflow owner either documents a training hypothesis with reviewed targets or closes the experiment as a data or service defect. Repeating the same search after each alert consumes inference and judging capacity without testing another explanation.

Architecture / decision scenario

An insurance employee assistant must correctly produce 97% of policy-comparison summaries and keep p95 latency below four seconds. A reviewed sample of 500 requests has 50 failures: 35 omit an exclusion despite correct retrieved text, and 15 use an outdated policy document. These are team measurements and acceptance targets.

The tempting proposal is a nightly SFT run on all 500 outputs. The workflow lead rejects it: the outputs include known mistakes, and the stale-document failures require a data repair. The content owner corrects the 15 affected source cases. The workflow lead submits a bounded prompt and numerical search for the 35 omission cases, leaving the documented defaults of 20 numerical trials and three repetitions in place.

The selected configuration reaches 98% correctness but takes 4.8 seconds at p95. It remains a candidate because it violates the latency condition. The team compares another configuration on the same cases rather than promoting the highest score automatically. If the constrained search cannot meet both targets and reviewed corrected responses are available, the model owner opens an SFT experiment with explicit training and validation snapshots.

This trigger is specific enough to automate candidate submission while preserving diagnosis. It distinguishes a repair to instructions, a repair to data, and a change to weights rather than treating all three as retraining.

Common exam traps

• Treating nat optimize as a model-weight training command.
• Starting retraining from drift or negative ratings without a diagnosed target behavior.
• Training on raw production outputs that contain the failure being repaired.
• Assuming numeric.n_trials also bounds prompt generations and their populations.
• Promoting a weighted-score winner that violates a mandatory latency or safety condition.

What to remember

Define the trigger, allowed changes, budget, and owner before scheduling an adaptation job. Use prompt and configuration search for defects in how the model is invoked; use training for a verified behavior gap with suitable targets. Both produce candidates that must pass the release process in 8.4.3.

Knowledge check

References

• NVIDIA, NVIDIA NeMo Agent Toolkit Optimizer
• NVIDIA, Supervised Fine-Tuning in NeMo RL