Prove guardrails with repeatable adversarial tests

Learning objective

Validate guardrails using synthetic adversarial cases and reproducible regression suites.

Core idea

A guardrail test must establish both that a detector identifies the relevant risk and that the system applies the required consequence. A correct risk annotation with a prohibited payment still executed is a failed guardrail. A blocked legitimate request is also evidence you need, because excessive blocking can make the solution unusable.

Combine generated adversarial exploration with a fixed, labeled regression suite. Generated attacks discover weaknesses; stable cases prove whether a change repairs or reintroduces a known defect. Reproducibility means preserving the inputs, configuration, expected outcomes, and evidence required to compare runs. It does not mean a generative model must return identical text every time.

See 3.2.8 for synthetic dataset construction and 4.1.8 for broader security testing. This lesson validates the controls and failure behavior from 4.2.1 through 4.2.7.

How it works

Define assertions for each guarded consequence

Start with the policy, not the attack generator. Give each test an expected result: permit, reject, require authorized approval, transform, or return an unavailable outcome. These are application test labels, not a universal Foundry evaluator schema. Record the control and intervention point that should enforce the decision.

Pair unsafe cases with legitimate near-neighbors. Test a refund just below the policy threshold, exactly at it, and just above it. Test the same proposed operation with the permitted principal and a different principal. Test unsupported claims with similar statements that the supplied evidence does support. Include schema-valid values that violate business rules, rather than testing only malformed JSON.

Assert the effect as well as the verdict. A denied output must not reach the caller; a denied tool proposal must not execute through the guarded path; a transform must reach the downstream consumer in its transformed form. Check persistence separately when the policy prohibits storing the content. The acceptance conditions in 4.2.5 provide the cases, while 4.2.7 provides fault-mode expectations.

Generate exploratory cases with scoped mechanisms

Microsoft Foundry’s preview synthetic-generation source, azure_ai_synthetic_data_gen_preview, stores generated queries as a reusable project dataset; see 3.2.8 for generation settings. Use generation to explore domain violations and legitimate contrast cases, then review and label the cases against authoritative policy.

Do not regenerate the acceptance suite on every run. Retain useful discoveries as fixed fixtures. A generator’s proposed expected answer is not an independent business-policy oracle: reviewers must establish which rule applies, the trusted facts, and the permitted consequence.

Probe agent-specific attacks

Foundry’s cloud AI Red Teaming Agent scopes prohibited actions, sensitive data leakage, and task adherence to agent targets and cloud red teaming, with text-based scenarios. Its IndirectJailbreak strategy tests instructions carried in tool outputs or returned context. Use that path to probe the admission checks from 4.2.4; see 4.1.8 for red-team configuration and attack strategies. The cloud examples use the preview REST API version 2025-11-15-preview.

Compare Attack Success Rate (ASR) by risk category, target, and strategy. A low aggregate ASR cannot excuse a single prohibited high-impact action, and a clean scan does not prove that the input space has been exhausted. Preserve an attack that exposes a defect as a labeled regression fixture, including the tool result that carried it.

Freeze data and evaluation configuration

Preserve the dataset version with the agent version, model deployment, prompt, tool definitions, guardrail policy, thresholds, and evaluator configuration; see 3.1.2 for dataset schemas and mappings. Distinguish testing stored outputs from invoking the changed agent: rescoring yesterday’s answers cannot show that today’s tool gate executes correctly.

Use stored candidates to isolate a detector or domain-rule change. Use an end-to-end run to verify that the changed workflow invokes the check at the required point and obeys its result.

For exact domain assertions, Foundry supports code-based custom evaluators: a Python grade(sample: dict, item: dict) returns a float from 0.0 to 1.0. They run without network access and have a two-minute grading-call limit, so include the required trusted policy fixture in the test data instead of querying a live account system during grading. pass_threshold and deployment_name are required initialization parameters even though the grader does not call an LLM.

If grade() raises or times out, Foundry records a score of 0.0 and an error. Separate those errors from valid policy rejections in reporting. A failed test harness must not become evidence that the guardrail correctly blocked a candidate.

Inject control failures and observe enforcement

Include detector and resolver failures, late streaming denial, transformed results, and nested-agent activity. Use safe test doubles for consequential tools and inspect their invocation ledger rather than exercising production payments.

Python Agent Hooks is experimental. Its record_sink receives InterceptionRecord evidence including verdict, reason, mode, identity, and sequence. In evaluate_only, the framework records decisions without blocking or transforming; use that mode for comparison, then run enforcement assertions in enforce. The record explains a decision, but the caller, persistence, and tool observations establish whether its consequence occurred.

Use the same fixtures for an evaluate-only run and an enforcement run. In evaluate-only, confirm that the expected deny or transform is recorded while the test doubles observe unchanged execution. In enforce, confirm that the denial prevents the guarded consequence or that downstream consumers receive the transformed candidate. A difference in the verdict itself calls for investigation before comparing enforcement effects.

Define a fault fixture and an observable assertion for each relevant point:

These are fixture expectations for the controls you install, not claims that every point has an identical failure policy. Keep a permitted contrast case for each row so the suite detects controls that block everything. Run tool-stage fixtures through framework-executed tools; provider-hosted execution needs separate downstream evidence, as 4.2.6 explains.

For approval, test a resolver that permits the exact proposal, one that denies it, and one that is unavailable. Change the proposal after approval to prove that your application requires a fresh decision. For persistence, inspect the history store in the configured persistence mode rather than assuming that withheld caller output proves nothing was stored. For concurrent work, inspect every test-tool ledger after cancellation; a discarded result alone cannot establish that no effect occurred.

Key distinctions and trade-offs

Generated scans offer breadth and discovery. Fixed regression fixtures offer stable comparisons and exact policy expectations. Deterministic assertions validate precise rules and side effects; semantic evaluators help assess language risks and need calibration. Use all of them for their respective evidence rather than declaring one aggregate score the release decision.

Record variability and repeat semantic cases when it affects release decisions. Keep exact invariants strict and retain enough evidence to reconstruct each result.

Architecture / decision scenario

A refund workflow changes its final writer and policy interceptor. The team generates synthetic boundary and adversarial queries, runs cloud red teaming for prohibited actions and leakage, and promotes reviewed discoveries into a versioned regression dataset.

The suite includes safe refunds, excessive amounts, unsupported explanations, malformed outputs, and an unavailable policy service. Code-based grading checks recorded proposals against fixed policy facts. A test-tool ledger proves that denied payments did not execute, while caller observations prove that rejected explanations were withheld.

The candidate release reduces aggregate ASR but permits one duplicate refund during a detector timeout. The team rejects the release because the fail-closed invariant failed. It repairs the interceptor, reruns the same dataset and fault fixtures, and retains run identifiers and configuration for the comparison. A fresh attack scan complements that regression evidence rather than replacing it.

Common exam traps

• Counting a correct annotation as proof that an action stopped.
• Regenerating the regression dataset and calling the comparison reproducible.
• Using model-only red teaming to validate agent-only risks.
• Treating evaluator errors as successful guardrail rejections.
• Testing evaluate_only and expecting enforced blocking.
• Accepting a lower aggregate ASR despite a prohibited consequential action.

What to remember

Validate detection, enforcement, and failure behavior separately. Use synthetic attacks to discover cases, then freeze labeled fixtures and configuration for regression. Inspect actual outputs, persistence, and tool effects; a score or verdict alone does not prove that the guarded consequence was prevented.

Knowledge check

References

• Microsoft, Generate synthetic data with the Microsoft Foundry SDK
• Microsoft, Run AI Red Teaming Agent in the cloud
• Microsoft, AI Red Teaming Agent
• Microsoft, Evaluate datasets with the Microsoft Foundry SDK
• Microsoft, Custom Evaluators
• Microsoft, Agent hooks