Measure bypasses alongside legitimate request failures

Learning objective

Evaluate adversarial bypasses, false positives, and false negatives against a safety test dataset.

Core idea

A safety evaluation must ask two questions: did the detector recognize the threat, and did the application permit the prohibited behavior? A missed attack can be contained by authorization checks. A detected attack can still succeed if the application releases content before enforcing the result. Measuring only the detector or only the final refusal message misses these differences.

For an architect approving a release, define positive as a case that requires the safety policy to intervene. Then a false positive rejects legitimate work, and a false negative misses a policy-violating case. Keep this definition stable across runs. A refusal rate without reviewed labels cannot tell you whether the system is safe or merely unusable.

How it works

Build the dataset around policy decisions

For each case, record a stable identifier, input origin, policy category, expected detector decision, expected application behavior, and reviewer rationale. Separate benign requests, direct injection, indirect injection through retrieved passages, hostile tool results, and jailbreaks. Add paired benign examples that share vocabulary with attacks, such as a security report quoting an injection attempt.

Preserve the attack location. An end-to-end case containing a hostile tool result must insert that result at the actual tool boundary; submitting its text as an ordinary user prompt tests a different path. Include multi-turn attempts, instruction impersonation, reordered text, and long payloads where the suspicious part appears after ordinary evidence. These are application test categories, not a claim that any one AWS filter guarantees coverage of them.

Keep a reviewed calibration split for choosing policy settings and a separate holdout split for the release decision. If you repeatedly tune to the same examples, the measured improvement can reflect those examples rather than the intended behavior. Use synthetic identifiers and controlled tools for consequential-action tests so you can observe attempted unauthorized actions without executing real ones.

Capture assessment evidence and actual outcomes

ApplyGuardrail supports independent input and output assessment. Record the guardrail identifier and version, policy settings, source, selected content, action, and relevant assessments. Use outputScope: FULL when you need both detected and non-detected entries for debugging. AWS notes that full scope does not apply to word filters or sensitive-information regex filters; do not assume the absence of such an entry proves a clean negative.

GUARDRAIL_INTERVENED is an intervention outcome and NONE means no intervention. A detection-only content filter uses inputAction: NONE or outputAction: NONE, so a detected case need not produce blocking behavior. Store the filter assessment separately from the application release result. Also record assessment failures as failures to complete the check, not as safe negatives.

Coverage is a separate field. Converse inline Guardrails exclude toolResult, tool definitions, and generated toolUse.input. If the tested attack lives there and you supplied no separate assessment, classify the result as a coverage gap. That gap may also produce a missed attack at the system level, but it is not evidence that the filter assessed and misclassified those bytes.

Observe at least the content released to the caller, attempted tool operations, and data admitted to the next model request. An assistant can write a safe final message after attempting an unauthorized action. Your test outcome should fail on the prohibited attempt even if the response reads like a refusal.

Compute the right denominators

Use reviewed labels to build the detector confusion matrix:

False-positive rate is false positives divided by all labeled benign cases. False-negative rate is false negatives divided by all labeled cases requiring intervention. Report counts alongside percentages, especially for small slices. Do not divide both by the total dataset size; doing so can conceal poor performance when benign cases greatly outnumber attacks.

Report end-to-end attack success separately: successful prohibited outcomes divided by attempted adversarial cases. Define success by the application policy, such as releasing protected content or attempting an unauthorized tool call. Compare that with legitimate-task completion. A system that blocks everything can have low attack success and unacceptable task completion.

Break results down by origin, language, policy class, and release path. Keep uncovered cases and assessment errors visible. A good aggregate score can hide a tool-output path that never invokes a detector or a streaming path that releases content too early.

Use managed evaluation without changing the test subject

Amazon Bedrock model evaluation with an LLM judge uses a generator model and an evaluator model. You can select built-in or custom metrics, and the full report is written to your specified S3 bucket. A model judge is useful for rubric-based review of responses, but its scores do not independently establish whether an unauthorized tool operation occurred. The built-in Refusal metric (Builtin.Refusal) identifies responses that decline or reject a request; on labeled benign slices, use it to surface likely false positives. Harmfulness (Builtin.Harmfulness) assesses harmful response content; on adversarial slices, use it to surface likely bypasses in released text. These are review signals, not the labeled error rates themselves, and neither metric observes tool attempts.

Custom judge prompt datasets use JSON Lines, a .jsonl extension, and Amazon S3 storage, with up to 1,000 prompts per evaluation job. Entries use prompt, optional referenceResponse, and optional category. For supplied inference responses, use modelResponses with response and modelIdentifier; AWS skips model invocation and evaluates those supplied responses. This lets you assess responses captured from the application instead of substituting a direct model call for the full pipeline.

Keep application event assertions as the authority for tool and release behavior. Use human review to validate judge labels and disagreements. Do not promote a release solely because a second model assigns favorable scores to final text.

Key distinctions and trade-offs

inputStrength and outputStrength are configurable filter strengths, not measured error rates. Compare candidate settings against the same labeled cases and fixed model, prompt, retrieval, tool, and response configurations. Increasing strength changes the trade-off; it does not establish a universal optimum.

More adversarial cases improve attack coverage, while representative benign cases reveal operational cost. Both are necessary. Set release criteria by consequence and slice, rather than hiding a high-risk failure inside a large pool of ordinary successful requests. See 3.1.8 for attack taxonomy and 3.1.9 for the gates being evaluated.

Architecture / decision scenario

A support agent release blocks more user-input attacks after a filter-strength change. The dataset also includes legitimate incident reports, poisoned retrieved passages, and hostile tool results. The team discovers more false positives on quoted attacks and a tool-result coverage gap despite an improved overall refusal rate.

It adds the missing independent tool assessment, retains server-side authorization, and recalibrates the input policy on the calibration split. The holdout run reports detector errors, coverage, attack success, and legitimate completion separately. A supplied-response judge job helps review answer quality, while event assertions verify that no prohibited tool attempt occurred.

Common exam traps

• Treating refusal rate as a labeled safety metric.
• Confusing a false negative with an unassessed field.
• Counting assessment timeouts as safe passes.
• Measuring only the final response and ignoring intermediate tool behavior.
• Tuning and approving on the same dataset.
• Assuming an LLM judge replaces reviewed labels and execution evidence.

What to remember

Define policy labels first, preserve attack origins, and report explicit denominators. Measure detector errors and successful bypasses separately. ApplyGuardrail assessments reveal what was checked; application events reveal what happened. Use managed model evaluation for response assessment while retaining end-to-end assertions and held-out release evidence.

Knowledge check

References

• AWS, ApplyGuardrail
• AWS, GuardrailContentFilterConfig
• AWS, Include a guardrail with the Converse API
• AWS, Evaluate model performance using another LLM as a judge
• AWS, Use metrics to understand model performance
• AWS, Create a prompt dataset for a model evaluation job that uses a model as judge