Choose prompting, classification, or safety training

Learning objective

Mitigate bias and toxicity via prompting, safety classifiers, and safety post-training.

Core idea

Choose the intervention at the layer that can change the observed failure. Prompting changes the instructions and examples used during inference. A safety classifier evaluates content against a policy and supports a runtime block. Safety post-training changes the generator’s learned behavior. They consume different budgets and repair different defects.

An agent architect should compare all three against the same reviewed failures. A prompt change can repair a misleading instruction quickly; a classifier can stop a harmful draft from reaching a reader; post-training can target recurring response preferences. None of those choices establishes success without the paired missed-harm and overblocking measurements in 9.3.2.

How it works

Repair explicit instructions first

When the cause is an instruction or demonstration, replace that cause before spending on a model change. Specify relevant decision criteria, avoid stereotyped shortcuts, and include benign examples that should receive useful answers alongside examples that should be refused. A blanket instruction to avoid every sensitive topic can suppress legitimate assistance without repairing the underlying unequal treatment.

For NeMo Agent Toolkit’s _type: react_agent, additional_instructions supplements the base prompt and system_prompt overrides it. The agent requires nvidia-nat[langchain]. Use the supplemental field for a focused policy addition when the base interaction behavior is suitable; review the complete replacement carefully if you override it. See 9.3.1 for identifying a prompt defect rather than inferring one from a detector score.

Prompt tokens are part of every relevant inference request. More examples can increase context and latency, so compare the smallest policy that works against realistic adversarial and benign tests. A carefully worded instruction is still an input to a model, not a measured guarantee that every generated answer complies.

Classify the candidate answer against a policy

See 7.1.3 for detector wiring: NeMo Guardrails activates content safety check output $model=content_safety through rails.output.flows to inspect the candidate answer. Choose a classifier when a documented content-safety category, such as hate speech or violence, captures the failure; an unequal but polite recommendation needs a task-specific bias assessment instead. Test each selected category on quoted, educational, and support contexts, and measure its latency and overblocking cost before deciding whether it improves the application.

Change learned behavior when runtime repairs plateau

Use safety post-training when recurring harmful behavior remains across representative prompts and the organization can curate and validate training data. Start with reviewed examples of helpful safe responses, appropriate refusals, and legitimate sensitive requests. Training only refusals can teach the wrong preference for ordinary tasks. Keep an untouched evaluation set that tests both harm avoidance and useful completion.

NVIDIA’s Safety for Agentic AI blueprint provides a concrete recipe: evaluate safety, security, and accuracy; post-train on curated datasets; deploy the trained model as a NIM; and run it with NeMo Guardrails at inference. It names a four-dataset blend, on-policy prompt generation with the target model, and supervised fine-tuning (SFT) with NeMo Framework RL. Its dataset list includes Nemotron Content Safety Dataset V2 and the Llama Nemotron Post Training Dataset. Use those listed resources to plan safety-specific and useful-task examples together rather than creating a refusal-only training set.

The recipe makes training and runtime classification complementary. SFT changes which answers the generator tends to produce; the deployed Guardrails configuration still checks individual interactions. After training, repeat safety, security, and accuracy evaluation on the candidate model and then evaluate the model-plus-rails combination. Preserve the benign sensitive requests in that comparison so a reduction in toxic drafts cannot conceal an increase in unnecessary refusals.

On-policy prompt generation ties part of the training preparation to the target model. Review those generated cases against the harm rubric and coverage matrix before including them; a generation process does not establish that a dialect or an identity-reference slice is adequately represented. Keep the release holdout separate from the blend and record which target model produced the training cases. This makes the next evaluation a test of transfer to unseen cases rather than a replay of the examples used to shape behavior.

NeMo RL implements SFT through examples/run_sft.py and a YAML configuration such as examples/configs/sft.yaml. Assign the model owner responsibility for the selected blend, training run, and candidate revision. The workflow owner then deploys the candidate as NIM and evaluates it with the same application prompts and runtime policy used for the baseline. Compare the complete delivered task, including classifier cost and benign completion quality, before promoting it.

Direct Preference Optimization (DPO) is an alternative when reviewers can produce response pairs; see 5.5.2 for its mechanics. In safety-specific pairs, prefer a useful safe answer over a harmful answer, and prefer useful permitted assistance over an unnecessary refusal. These comparisons teach different distinctions, so include both in the review plan. The blueprint’s documented post-training recipe uses SFT; selecting DPO is a separate training decision.

Post-training spends curation, training, and regression-evaluation capacity before deployment. Keep the runtime classifier in the release comparison rather than assuming improved model behavior makes its contribution redundant. A trained generator and a detector can still share blind spots in the same language or harm category.

Key distinctions and trade-offs

Compare interventions by their effect and operating cost:

These are complementary choices, not interchangeable scores. Prompting specifies desired behavior, classification checks text, and training changes learned tendencies. The release assessment must still consider helpfulness disparities that a toxic-content policy does not label.

Each intervention acts at a different layer of the path from instructions to the reader.

Architecture / decision scenario

A retail content-generation tool drafts product descriptions under a hypothetical 1.5-second latency target. On reviewed tests, a prompt repair reduces unsupported gender stereotypes from 12% to 4%, but still emits abusive phrasing on 3% of adversarial requests. Adding a tested output classifier reduces delivered abusive responses to 0.5%, while pushing p95 latency from 1.2 to 1.8 seconds and blocking 8% of benign sensitive-product descriptions.

The tempting proposal is to deploy the classifier configuration unchanged because the abusive-response rate falls. The product owner rejects it because latency and legitimate content access fail the release criteria. The workflow owner keeps the prompt repair; the safety owner reviews category definitions and the blocked benign descriptions before another trial. The model owner prepares reviewed SFT examples and DPO preference pairs that preserve useful descriptions without the stereotypes or abuse.

The team compares prompt-only, prompt-plus-classifier, and post-trained candidates on the same untouched set, including benign identity references. Training is a funded candidate repair, not a promise of faster or safer inference. The release owner chooses only after measured harm, overblocking, helpfulness, and latency satisfy the application requirements; runtime escalation policy belongs to 9.4.

Common exam traps

• Treating a stronger refusal instruction as a repair for every disparity.
• Confusing a classifier’s safe verdict with a fairness assessment.
• Choosing a toxicity category to measure unequal recommendation quality.
• Assuming safety post-training removes the need to evaluate runtime rails.
• Treating DPO as a content filter or training only on refusals.

What to remember

Repair instructions where they cause the failure, classify content where runtime policy can catch it, and fund post-training for persistent learned behavior. Name the configuration or training artifact that implements each choice. Retest useful assistance as carefully as harmful output: a system that refuses everything has not solved the task.

Knowledge check

References

• NVIDIA, Configure the ReAct Agent
• NVIDIA, Content Safety
• NVIDIA, Supervised Fine-Tuning in NeMo RL
• NVIDIA, Direct Preference Optimization in NeMo RL
• NVIDIA, Safety for Agentic AI