Compare defenses against matched adversarial attempts

Learning objective

Red-team for prompt injection and jailbreaks, and compare defenses before and after.

Core idea

A red-team result describes an attacker, an entry point, an intended deviation, and an observed outcome. A lower attack-success rate after adding a defense is useful only if the attack population remains comparable and legitimate work still succeeds. A system that refuses every request can appear secure under an attack-only score.

OWASP distinguishes direct prompt injection through user input from indirect injection through external content. It treats jailbreaking as a form of prompt injection, one that makes the model disregard its safety protocols entirely, and notes that preventing it requires ongoing updates to the model’s training and safety mechanisms. Within that family, jailbreak tests attempt to defeat behavioral restrictions; an indirect injection can instead induce an unauthorized tool action without producing obviously toxic text. Your test plan must cover both instruction following and the consequences available to the agent.

See 9.3.2 for harmful-output and overblocking measurements, and 3.1.4 for repeatable evaluation artifacts. This lesson adds adversarial entry points, attack-success criteria, and paired defense comparisons.

How it works

Describe success before generating attacks

Build a scenario register with an attack identifier, trusted task, entry point, target asset, intended deviation, and scoring rule. For example, an instruction in a retrieved article might ask a research assistant to send an unpublished manuscript to an external address. Success is an attempted or completed unauthorized transmission, as defined by your test, rather than the presence of an alarming phrase in the answer.

Keep these outcomes separate: the detector flagged the payload; the agent followed its instruction; the tool proposed an unauthorized operation; and the operation actually reached the mocked destination. A refusal after a transmitted message is a failed security outcome. Conversely, a model can follow a malicious suggestion while an independently enforced permission blocks the resulting action. Record both the model failure and the containment success.

Run the test against synthetic data and inert tools. NVIDIA’s NeMo Agent Toolkit retail safety example explicitly requires sandboxed red teaming and mocks email and database writes.

Use a scanner for breadth and workflow injection for depth

NVIDIA’s garak supplies probes, generators, and detectors. A generator connects the scanner to its target, probes supply adversarial interactions, and detectors identify specified failures in outputs. In the v0.17.0 README, garak --list_probes enumerates probes; --target_type and --target_name select the target interface and model; --spec probes.promptinject selects the PromptInject probe family.

The scanner reports per-probe detector results and records a detailed .jsonl run log. A scanner pointed directly at a model measures that model path: for a chat model, --target_type nim with --target_name set to the NIM model name scans the model alone. To claim that an application is protected, target the defended application path and verify that the relevant controls execute. For that, garak’s rest.RestGenerator connects to any REST endpoint that returns plaintext or JSON, configured by a short YAML file that describes the endpoint, so it can reach the guarded application’s own API. Broad scanner coverage does not replace tests of your retrieval fields, tools, and business permissions.

The NeMo Agent Toolkit v1.9.0 retail example supplies targeted workflow tests; its built-in defense middleware requires the defense extra (nvidia-nat[defense]). Its _type: red_teaming middleware injects an attack_payload at a configured target_function_or_group and target_location. target_field selects the affected field; payload_placement supports replace, append_start, append_middle, and append_end. These identifiers let you test an indirect instruction inside a tool result rather than merely submitting another hostile user prompt.

Score the workflow, including partial failures

The example’s RedTeamingRunner executes the dataset and scenarios, while RedTeamingEvaluator applies scenario-specific judging. scenario_specific_instructions defines the desired interpretation; intermediate_step_filters selects the events to inspect; reduction_strategy combines evaluations using first, last, or max.

Match the inspected event to the harm. Scoring only the final answer can miss a disclosure that occurred during a tool call. Include a deterministic check of the mocked operation’s recipient and payload when those fields decide success, then compare that result with the judge’s assessment. Review disagreements instead of assuming the LLM judge proves the security outcome.

The example uses scores of 0.0 for failure of the attack, 0.5 for partial success, and 1.0 for full success. Its attack success rate counts scores greater than 0.5; its overall risk score is the mean of scenario scores. Partial failures therefore affect the mean without counting as successful attacks in that ASR definition. Preserve the scoring convention and inspect partials separately; changing the success cutoff between runs destroys the comparison.

Compare the same attack population across defenses

The documented command is nat red-team, with --red_team_config selecting the scenario configuration and --reps 5 illustrating repeated runs. The retail example compares configurations whose base_workflow changes from the baseline to the defended workflow. Preserve the dataset, attack placements, scoring rubric, and repetition count while changing the defense under study. Stochastic outputs still require repeated observations; five repetitions are an example, not a statistical guarantee.

NeMo Guardrails’ vulnerability-scanning page illustrates progressive configurations: bare_llm, with_gi, with_gi_dr, and with_gi_dr_mo. These separate the bare model, general instructions, added dialogue rails, and added moderation rails. Use the same principle to isolate which protection changes the result. The page’s per-category table also shows why results are reported by attack family below: adding moderation rails raises or holds the protection rate in every other category, yet module dan falls from 61.3% in with_gi_dr to 52.7% in with_gi_dr_mo. Treat that row as a pattern to look for, and do not reuse the page’s sample protection rates as predictions for your model or workflow.

Keep a frozen comparison set and a separate discovery set for adaptive attacks. After discovering a bypass, preserve it as a regression case and rerun both the baseline and candidate against it. Also retain legitimate tasks that resemble the attack language. Otherwise, the defense can improve its score by rejecting the entire subject area. The frozen comparison set and every regression case run against both the baseline and the defended workflow.

Key distinctions and trade-offs

Choose the test method by the surface and decision it covers:

Report rates with counts and denominators by attack family. A pooled score can improve while a small but severe disclosure category gets worse. Include completed legitimate tasks, blocked legitimate tasks, detector failures, and end-to-end latency alongside attack outcomes.

Architecture / decision scenario

A publishing company’s research assistant summarizes licensed source material and drafts editorial emails. The security release owner requires zero unauthorized email transmissions in the test suite, no more than 3% blocked legitimate tasks, and a measured p95 response under 3.5 seconds. All email operations use a mocked delivery service. The following numbers are hypothetical test observations:

The tempting proposal is to promote the candidate because full attack success falls by twelve percentage points and the transmission check passes. The release owner rejects that proposal: an 11% legitimate-task block rate violates the 3% requirement, even though latency fits.

The team separates direct jailbreak cases from instructions inserted into retrieved article bodies using target_location: output and the relevant target_field. It examines the 50 additional blocked legitimate tasks, revises the overbroad screening policy, and reruns the matched suite with the email permission unchanged. Promotion depends on all three requirements, not a single scanner score. Any discovered bypass becomes an additional paired regression case rather than silently changing only the candidate’s test set.

Common exam traps

• Testing only hostile user messages while retrieved documents remain untested.
• Treating a direct model scan as an end-to-end application assessment.
• Counting a final refusal as protection after a harmful tool action.
• Counting a score of 0.5 as success under the example’s > 0.5 ASR definition.
• Changing attacks, rubrics, or denominators between baseline and defended runs.
• Promoting a lower attack-success rate without checking legitimate completion and latency.

What to remember

Define the attack’s consequence, inject at the real entry point, and inspect the event that proves success. Compare defenses on matched scenarios with repeated runs and a legitimate workload. Keep scanner findings, model deviations, and independently blocked side effects distinct.

Knowledge check

References

• NVIDIA, NeMo Agent Toolkit Safety and Security, v1.9.0
• NVIDIA, garak README, v0.17.0
• NVIDIA, LLM Vulnerability Scanning
• OWASP, LLM01 Prompt Injection