Skip to lesson

NVIDIANVIDIA-Certified Professional: Agentic AI2026-109.3 Mitigate bias and toxicity in outputs.

Measure missed harms and overblocking separately

Completed by the knowledge check

Learning objective

Measure toxicity and bias with representative tests, tracking missed harms and overblocking.

Core idea

An output-safety evaluation needs two kinds of failure: harmful content that reaches the reader and legitimate assistance that the system blocks. Bias adds a distributional question: which groups, languages, or task types experience those failures? One overall compliance rate can conceal all three problems.

For an evaluation owner, the deliverable is a policy-labeled interaction set and a release comparison with explicit denominators. NeMo Guardrails provides policy-based evaluation through nemoguardrails eval. Its methodology defines compliance rate over interactions for which a policy applies, and explicitly directs you to other tools for precision, recall, or F1. Its separate per-rail moderation evaluation reports harmful-blocked and helpful-allowed percentages; see 7.1.3. Use these results alongside policy compliance to examine which populations experience missed harms and overblocking.

How it works

Build representative policy tests

Define separate policies for abusive generation, legitimate discussion of abuse, task-specific stereotyping, and useful assistance. A request to insult a colleague and a request to summarize a complaint containing that insult may share vocabulary but require different outcomes. Label intended behavior in context, including earlier turns where they change the request’s meaning.

In Guardrails evaluation configuration, policies entries carry an id and description. interactions entries carry an id, inputs, and expected_output; expected outputs identify their governing policy. The documented refusal type expresses an expected refusal. Inputs can include message sequences, so a multi-turn test need not flatten a conversation into one isolated sentence.

Build a coverage matrix across deployment languages, dialects, task types, severity levels, quoted versus directed abuse, and benign identity references. Use real authorized traffic for prevalence and curated challenge cases for rare harms. NVIDIA’s Safety for Agentic AI blueprint names Nemotron Content Safety Dataset V2 for content-safety evaluation and NVIDIA garak for vulnerability scanning. Use the dataset as a labeled content-safety test set and garak as a challenge scanner; inspect their coverage against your deployment before admitting cases to the coverage matrix. Keep their results separate: an adversarial set deliberately enriched for attacks does not estimate everyday incident frequency. NIST MEASURE 2.1 calls for documented test sets, metrics, and tools; MEASURE 2.3 connects evaluation to deployment-like conditions.

Label harms independently of the detector

Have reviewers apply the written rubric to candidate answers and intended user outcomes. Record whether the content is harmful, whether blocking is appropriate, which harm category applies, and which population or task slice the example represents. Your rubric defines these labels; record them beside each interaction. Resolve disagreements with domain specialists rather than letting the tested classifier define its own ground truth.

Guardrails supports models of type: llm-judge and the prompt task llm_judge_check_single_policy_compliance. This can scale policy checks, but a judge is another model whose mistakes need examination. Calibrate its decisions against reviewed cases, especially benign identity discussions, unfamiliar dialects, and subtle stereotypes. Keep disagreements visible so a high judge score cannot erase a known harmful case.

Distinguish harmful candidate outputs from harmful delivered outputs. A classifier may correctly block a toxic draft, while an agent may refuse before a draft exists. If you inspect only delivered answers, blocked drafts disappear from the detector analysis. If you inspect only drafts, you miss the user’s final experience. Maintain separate records for candidate quality, moderation decisions, and delivered task success.

Compare harms across slices

See 7.1.3 for moderation denominators: missed-harm rate is harmful cases allowed divided by all harmful cases, and overblocking rate is its false-positive rate, benign cases blocked divided by all benign cases. On a complete binary-decision set, harmful-case recall equals one minus missed-harm rate; these are complementary views of the same detection behavior.

The per-rail nemoguardrails evaluate moderation workflow takes a split of harmful or helpful and reports percentages of harmful prompts blocked and helpful prompts allowed. With complete binary decisions, missed-harm percentage is 100% minus harmful-blocked percentage, and overblocking percentage is 100% minus helpful-allowed percentage. Keep model errors separate, as the performance documentation does: an error is not automatically an allowed harmful answer or a correct safety block.

For bias analysis, group independently labeled results by language, dialect, identity reference, and task. Report the harmful and benign counts separately for each slice, then compare the corresponding error rates. An identity-reference slice needs both abusive and legitimate mentions; otherwise its block rate says little about unjustified treatment. NIST MEASURE 2.11 provides the assessment artifact for documenting fairness and bias results.

Write a reporting rule before examining the results. For example, your evaluation plan might require at least 100 benign decisions before publishing a slice’s overblocking rate and at least 100 harmful decisions before publishing its missed-harm rate. This is a chosen coverage floor, not a statistical assurance: record uncertainty and seek more cases when a decision is close. Below the floor, show counts and mark coverage insufficient instead of presenting a small sample as a reliable percentage. Do not merge distinct dialects solely to make an underrepresented slice disappear.

Make release criteria apply to each required slice as well as the overall population. Specify the policy, dataset revision, denominator floor, maximum missed harms, maximum overblocking, and reviewer responsible for insufficient coverage. Pair a deployment-weighted summary with these per-slice results. Weighting answers how often a failure occurs in expected traffic; the per-slice requirement prevents a small population’s poor service from being hidden by the majority.

Each slice’s harmful and benign decisions meet their own coverage floor before the matching rate is published.

Each slice's harmful and benign decisions meet their own coverage floor before the matching rate is publishedStarts from independently labeled results grouped by slice, which split into a harmful count and a benign count. Each count meets its own floor of at least 100 decisions before its missed-harm or overblocking rate is published and goes to the release criteria. Below the floor, the slice shows counts and is marked coverage insufficient, which goes to the reviewer responsible for insufficient coverage.yesyesnonoGroupindependentlylabeled resultsHarmful countBenign countAt least 100harmful decisions?At least 100benign decisions?Publishmissed-harm ratePublishoverblocking rateShow counts, markcoverageinsufficientRelease criteriaReviewerresponsible forinsufficientcoverage

Keep stage and label meanings fixed across releases. A refusal that repeats a targeted insult can itself be harmful; counting every refusal as safe mislabels the delivered outcome. For disparities in summary quality or recommendations, add matched-task human review: toxicity classification alone cannot measure omitted viewpoints or unequal useful assistance.

Run comparable configurations

Use nemoguardrails eval run to execute interactions and nemoguardrails eval check-compliance for LLM-as-a-judge policy checks. Review results with nemoguardrails eval ui. Record the agent, detector, policy, prompt, and dataset revisions so you can attribute a change. See 8.3.1 for benchmark maintenance; this lesson adds harm-specific labels and paired error rates.

Compare the Guardrails efficiency measures—latency, LLM-call count, and token consumption—alongside per-slice safety results.

Key distinctions and trade-offs

Compliance rate resembles accuracy for moderation: it includes both correctly blocked harmful interactions and correctly allowed benign ones. A benign-heavy dataset can therefore look excellent while missing most rare severe harms. Inspect each policy and slice rather than accepting the weighted aggregate alone.

Representative sampling estimates expected traffic behavior; challenge testing searches for weaknesses. Matched bias tests probe unjustified differences; natural-traffic slices show how often people encounter them.

Architecture / decision scenario

A municipal meeting companion summarizes public comments, including quoted harassment. In this hypothetical deployment, the product owner requires at most 2% missed harms and 3% benign overblocking overall and for each required dialect slice. The evaluation plan requires 100 labeled decisions of each relevant class before reporting that slice’s rate.

On 1,000 benign cases, the configuration blocks 20: its overall overblocking rate is 2%, within the target. Yet 15 of those blocks come from a 100-case dialect slice: its overblocking rate is 15%, compared with 5/900, about 0.56%, elsewhere. Only 12 harmful cases cover that dialect, so its missed-harm assessment lacks the required coverage even though the aggregate harmful-case result meets 2%.

The tempting proposal is to ship because the overall results pass. The release owner rejects it: one population receives much worse access to legitimate summaries, and its harmful-case coverage remains unresolved. The policy owner examines the blocked quoted testimony, clarifies reporting abuse versus producing it, and adds reviewed contextual cases to interactions. The evaluation owner expands the undercovered harmful slice and reruns an untouched holdout. A separate matched set checks whether speaker identity changes tone or omitted points. Release depends on the per-slice criteria and useful-summary quality, not a passing aggregate.

Common exam traps

  • Dividing missed harms by all requests instead of all harmful cases.
  • Treating every blocked identity reference as a safety success.
  • Reporting attack-set results as production prevalence.
  • Letting the evaluated classifier supply the reference labels.
  • Assuming Guardrails compliance rate automatically supplies precision and recall.

What to remember

Define harm and benign usefulness explicitly, test representative slices, and separate candidate safety from delivered behavior. Pair missed-harm rate with overblocking rate and inspect their denominators. Use Guardrails policy and per-rail moderation evaluation together, keep model errors separate, and require adequate coverage and acceptable error rates for each relevant population.

Knowledge check

References