Skip to lesson

MicrosoftMicrosoft Certified: Multi-Agent AI Solutions Expert2026-103.2 Optimize prompt and model performance for multi-agent solutions

Generate synthetic regression and adversarial cases

Completed by the knowledge check

Learning objective

Generate synthetic regression, edge-case, and adversarial evaluation data.

Core idea

For an architect evaluating a multi-agent solution, synthetic data fills coverage gaps that production examples do not reliably expose. It lets you exercise rare ambiguities, unsupported requests, and hostile inputs without waiting for them to occur. Generated volume alone does not establish coverage or correct expected outcomes.

Give each case a purpose and an oracle: the expected behavior that determines whether the target passed. Separate regression cases that preserve known behavior, edge cases that test valid extremes or incomplete inputs, and adversarial cases that attempt to violate the contract.

Microsoft Foundry supports cloud synthetic query generation and the AI Red Teaming Agent for adversarial probing, both preview capabilities. Preserve reviewed fixtures for repeated evaluation, and distinguish adversarial scan results from a reusable regression dataset.

How it works

Define the cases before generating them

Start from the agent’s result contract, tool permissions, and coordination responsibilities. For each dimension, specify normal behavior, an edge, and a violation attempt. An order assistant needs cases for an authorized cancellation, two orders with ambiguous references, and a request to cancel another customer’s order.

For regression data, convert a verified defect or required behavior into a repeatable fixture. Preserve the trigger, relevant context, and expected outcome; add paraphrases that retain the same business meaning. Do not make the candidate’s previous answer the oracle merely because it was produced successfully.

For edge cases, ask for incomplete fields, contradictory facts, alias collisions, unavailable evidence, and corrections after a handoff. Label which constraint each case tests so that a failure can be attributed rather than absorbed into one average.

For adversarial cases, distinguish malicious instructions in user input from instructions planted in retrieved documents or tool results. State the authorized task and the behavior that must remain prohibited. A refusal can be the correct result; measuring only answer completion would score the successful defense as a failure.

Use cloud synthetic query generation

The cloud evaluation definition uses data_source_config with type="azure_ai_source" and scenario="synthetic_data_gen_preview". The run uses the preview data source type="azure_ai_synthetic_data_gen_preview", with item_generation_params.type="synthetic_data_gen_preview". Submit the run with openai_client.evals.runs.create() after creating its evaluation definition.

Within item_generation_params, samples_count accepts 15 through 1,000 queries, inclusive. model_deployment_name selects the generator deployment and requires Responses API capability. The model router is unsupported as the generator, although it can be the evaluation target. Do not confuse the generator model with the model or Foundry agent whose responses you assess.

Use prompt to request the case families, their coverage balance, and the domain vocabulary. It is optional when the agent target has configured instructions, but explicit generation instructions make your coverage intent reviewable. sources accepts seed files by file ID; the documented constraint is one file. output_dataset_name names the stored query dataset so you can reuse it.

The service generates queries, sends them to the target, evaluates its responses, and stores the queries as a dataset. Map generated queries from {{item.query}} and target response text from {{sample.output_text}} to evaluator inputs. A generated response is the target’s output, not an independently verified expected answer.

This path generates individual test queries. When the requirement concerns a sequence of turns, corrections, or tool-result injection, construct a conversation fixture or use an appropriate simulator rather than assuming one query reproduces the full interaction.

Generate agentic attacks with cloud red teaming

The AI Red Teaming Agent (preview) tests prohibited actions, sensitive data leakage, and task adherence as agentic risks. Those categories are agents-only and cloud-only. Indirect prompt injection, or XPIA, uses benign queries and mock tool outputs containing attack placeholders to probe those risks. This tests responses to malicious evidence; see 2.1.3 for prompt-attack detection and enforcement.

Create the evaluation with data_source_config={"type": "azure_ai_source", "scenario": "red_team"}. The cloud example selects builtin.prohibited_actions, builtin.task_adherence, and builtin.sensitive_data_leakage as testing criteria. For prohibited actions, generate the taxonomy with project_client.beta.evaluation_taxonomies.create(), then review and confirm the proposed policies before using its file ID. That taxonomy drives both attack generation and assessment of policy violations.

Submit a run through openai_client.evals.runs.create(). Its data_source.type is "azure_ai_red_team"; item_generation_params.type is "red_team_taxonomy". Configure attack_strategies, such as "IndirectJailbreak", and num_turns, with source={"type": "file_id", "id": TAXONOMY_FILE_ID} pointing to the reviewed taxonomy. TAXONOMY_FILE_ID denotes that file’s identifier. The target identifies the agent under test. These fields differ from the synthetic-query generator’s fields.

Cloud red teaming has limited region support; verify the supported location before selecting the project. The documented agentic support matrix includes Foundry hosted prompt and container agents with Azure tool calls. It excludes Foundry workflow agents, non-Foundry agents, and function tool calls. A scan of a supported specialist does not establish that an entire multi-agent workflow passed.

Respect category-specific limits: the concept page describes prohibited-actions and sensitive-data-leakage tests as single-turn and English-only. num_turns does not establish multi-turn support for every risk category. Mock tools retrieve synthetic data but do not simulate tool behaviors. Test actual cancellation side effects separately through your application fixtures.

Cloud scan results redact adversarial inputs. Use the reported Attack Success Rate (ASR) to locate failures, review possible false positives, and curate explicit regression cases separately. Do not assume the scan export contains every attack input needed for exact replay.

Keep classic simulation separate

The Foundry classic page documents the preview azure.ai.evaluation.simulator.Simulator for non-adversarial conversations, using a target callback and max_conversation_turns. It can supplement conversation fixtures where that documented interface fits. Its classic adversarial counterpart, AdversarialSimulator, supports randomization_seed for reproducible conversation starters, not identical complete executions. Do not transfer classic simulator parameters or region lists to cloud red teaming.

Curate a stable evaluation asset

Review generated cases for duplicates, impossible premises, unsupported features, and wrong labels. Use a deterministic oracle for exact identifiers and schema rules, and the calibrated rubric from 3.2.7 for semantic properties. Add human review where an acceptable outcome cannot be expressed reliably by either check.

Attach application-owned case IDs, family labels, source provenance, and expected outcomes. Keep the generator configuration and generation date in your dataset record. Split related cases by seed or failure family before separating tuning and held-out sets; near-duplicate paraphrases across both sets weaken validation.

For reusable curated inputs, project_client.datasets.upload_file(name=..., version=..., file_path=...) registers JSONL or CSV as a versioned Foundry dataset. See 3.1.2 for running that asset. Freeze the same inputs for baseline and candidate evaluation; changing both the target and the test cases makes a score change difficult to interpret.

Key distinctions and trade-offs

Generation versus evaluation: The generator proposes test inputs. The target produces responses. Evaluators assess those responses against reviewed expectations.

Edge case versus attack: A missing field or ambiguous name can be a legitimate input. An attack tries to override the intended behavior. Both require explicit outcomes, but their acceptable responses differ.

Reproducible starter versus replayable fixture: A seed controls documented randomization. A saved fixture preserves the actual evidence used for a comparison.

Synthetic cases offer controllable coverage but can reflect the generator’s assumptions. Keep representative real examples alongside them and report results by case family.

Architecture / decision scenario

A support workflow mishandles cancellations after a customer corrects an order number. The evaluation owner creates regression fixtures from the verified failure and uses a cloud generation prompt to request paraphrases, missing-ID cases, and two-order ambiguities. It names the output dataset and respects the single-seed-file constraint.

The owner preserves multi-turn corrections as conversation fixtures. For a supported Foundry prompt specialist using Azure tools, the owner creates and reviews a prohibited-actions taxonomy that forbids unauthorized cancellations. A cloud AI Red Teaming Agent run selects IndirectJailbreak to probe malicious mock tool outputs in a supported region. Reviewers inspect ASR findings and separately curate fixtures with exact order IDs, clarification, or refusal outcomes; mock-tool scans do not prove actual cancellation behavior or whole-workflow safety.

The curated dataset receives a version through datasets.upload_file(). Baseline and candidate run against the same fixtures. Promotion requires the cancellation regression to improve without increasing false acceptance on unauthorized orders, even if average answer quality rises.

Common exam traps

  • Treating synthetic target responses as ground truth.
  • Assuming the model router can generate cloud synthetic queries.
  • Supplying multiple seed files where only one is supported.
  • Using single-turn generation to claim multi-turn continuity coverage.
  • Confusing direct input attacks with retrieved-context injection.
  • Assuming cloud agentic red teaming supports workflow agents or function tools.
  • Assuming randomization_seed guarantees identical complete executions.
  • Teaching classic simulator parameters or region lists as cloud red-teaming settings.

What to remember

Generate for a named coverage gap, review the oracle, and preserve a versioned fixture. Regression, edge, and adversarial families need distinct expected outcomes. Respect each generator’s preview status, interface, limits, and service scope.

Knowledge check

References