Skip to lesson

MicrosoftMicrosoft Certified: Multi-Agent AI Solutions Expert2026-103.1 Design and implement evaluation and validation strategies for multi-agent solutions

Gate response changes with independent checks

Completed by the knowledge check

Learning objective

Evaluate prompt instruction adherence, schema compliance, and response consistency using A/B comparisons and regression baselines.

Core idea

A response can be fluent, structurally valid, and still violate its assigned task. It can follow the task but return JSON that breaks the next agent. It can pass both checks once and change a consequential answer across repeated runs. For an architect, these are three failure classes that need independent acceptance evidence.

Build comparisons around an accepted baseline, a candidate, and fixed test cases. Use Foundry’s behavioral evaluators for instruction adherence, deterministic validation for the output contract, and repeated case-level comparisons for consistency. See 2.1.4 for prompt-version promotion and 3.1.2 for cloud-run configuration; this lesson defines what response evidence should block that promotion.

How it works

Preserve the instructions the judge needs

Foundry’s Task Adherence (preview) evaluator measures whether agent actions respect assigned tasks, rules, procedures, and policy constraints. Its catalog identifier is builtin.task_adherence. Supply the interaction and trusted instructions, not only an isolated answer that lacks the rules being assessed.

For a stored interaction, the messages format can include the system instruction and conversation. For a generated agent target, the cloud evaluation guide maps response to {{sample.output_items}} so the evaluator receives structured interaction data. The agent evaluator documentation also supports separate query and response fields, with a system message in query providing instruction context. Keep the representation consistent between baseline and candidate.

Task Adherence is documented for turn-level evaluation. Do not substitute a conversation-level score from another evaluator and call it equivalent instruction-adherence evidence. Its preview status also belongs in the experiment record; a generic Foundry evaluation label does not remove that qualification.

Choose cases that distinguish following policy from producing a plausible answer: a request that conflicts with the role, retrieved text that tries to override instructions, missing facts that require an uncertainty statement, and a request for an unapproved action. The expected outcome must name the applicable instruction. Review judge reasons on failures to distinguish a real defect from an underspecified rubric or missing context.

Validate the business schema deterministically

Parse the actual emitted response and validate it against the agreed schema. A fenced JSON snippet inside surrounding prose is not the same result as a JSON document when the consumer expects only JSON. Test the same extraction and parsing behavior that the downstream consumer uses rather than repairing malformed responses only for evaluation.

JSON Schema’s properties describes named fields, but fields are optional unless required names them. Additional properties are allowed by default; use additionalProperties: false when the contract prohibits unexpected fields. A type-correct object can still contain a business-invalid value, so supplement schema checks with deterministic domain rules.

Foundry supports custom code-based evaluators for format validation. Their interface is grade(sample: dict, item: dict) -> float, returning a value between 0.0 and 1.0. The documented sandbox includes jsonschema and pydantic, so you can implement structural checks without asking a model judge whether JSON “looks valid.” Register the evaluator with project_client.beta.evaluators.create_version() using definition.code_text, data_schema, and metric metadata. This is the documented beta catalog interface; do not infer a general-availability promise from its availability in the SDK.

For stored-response grading, read item.get("response"). For generated target text, use the documented item.get("sample", {}).get("output_text") access pattern, which Microsoft warns can change in a future API update. Return 1.0 for a fully valid contract and 0.0 for a known validation failure; set the evaluator’s pass_threshold to 1.0. Its documented initialization also requires deployment_name for orchestration, although the grading code does not call an LLM.

Distinguish a rejected response from a broken validator. If grade() raises or times out, Foundry records 0.0 and marks the item as an error. Catch expected parse and validation failures, but preserve unexpected evaluator errors for investigation. The sandbox has no runtime network access and limits a grading call to two minutes. Bring required reference values into the dataset instead of making the validator query a production service.

Measure consistency against meaning and invariants

Consistency does not require identical prose unless identical prose is the contract. Compare stable facts, decision categories, identifiers, required caveats, and output shape across repeated runs of the same case. Keep the input evidence and tool-result fixtures fixed when the experiment is intended to isolate prompt behavior.

Foundry’s builtin.similarity measures semantic similarity against ground_truth. It takes query, response, and ground_truth, or messages with ground_truth, and returns a 1–5 score with default threshold 3. Use it where acceptable paraphrases should retain meaning. It is a reference-relative semantic score, not proof that repeated runs always produce the same business decision.

Use deterministic comparisons for invariants such as an unchanged case identifier or a stable eligibility category. Compare a candidate’s repeated results with the accepted category, not merely with whichever answer the candidate generated first. Otherwise, a consistently wrong answer appears stable. Include approved variations in the fixture where several responses are valid.

Compare A and B against a retained baseline

Run A and B over the same dataset version with the same evaluator definitions, judge deployment, response schema, and inference configuration. Specify the agent versions rather than relying on a name that can resolve to a different version. Change one intended factor, such as the role instruction, and record unavoidable differences separately.

Foundry’s results view supports selecting two or more runs and choosing Compare. Set a baseline run to show deviations from the accepted reference. The comparison uses statistical t-testing; hovering a cell exposes sample size and p-value. A statistically significant improvement in a quality metric does not authorize a candidate that fails a mandatory schema or policy case.

An inconclusive comparison can reflect too few examples or weak evidence of a difference. It does not establish equivalence. Keep per-case results and repeated-run variation alongside aggregate scores. The comparison view is not saved, so retain the run identities and acceptance decision needed to reconstruct the comparison.

Key distinctions and trade-offs

Adherence versus correctness: Following instructions and returning factually correct content are related but separate checks. A correct answer can violate an instruction to escalate.

Schema versus semantics: A valid shape makes data consumable; domain rules determine whether its values are acceptable.

Similarity versus repeatability: Reference similarity tolerates wording variation. Repeated invariant checks detect unstable decisions.

Statistical improvement versus release acceptance: A better average can coexist with a severe regression on a mandatory case. Define the gate before inspecting candidate scores.

Architecture / decision scenario

A specialist agent returns a claims decision with a case identifier, decision category, and explanation. The team compares prompt versions A and B on the same versioned dataset. Cases include insufficient evidence, conflicting retrieved instructions, and a valid request with several acceptable explanation phrasings.

builtin.task_adherence receives the trusted instructions and interaction. A custom grade() validates the JSON schema and the team’s case-identifier and decision rules. builtin.similarity scores explanations against approved references, while repeated runs check whether the decision category stays correct and stable. Judge and validator versions remain fixed.

B produces clearer explanations and improves the comparison’s quality score, but sometimes omits a required field and sometimes decides a case that should be escalated. The team blocks promotion based on those independent failures. It retains A as the baseline and preserves the failing fixtures, rather than weakening the schema or replacing the reference with B’s first answer to obtain a passing score.

Common exam traps

  • Asking an adherence judge to assess rules absent from its input.
  • Treating valid JSON syntax as complete schema compliance.
  • Assuming properties alone makes fields mandatory or disallows extra fields.
  • Using semantic similarity as the only check for identifiers or decision categories.
  • Treating an inconclusive comparison as proof of equivalence.
  • Promoting a candidate because its mean quality improves despite mandatory-case failures.

What to remember

Preserve instruction context, validate emitted results deterministically, and test repeated behavior against accepted invariants. Use fixed A/B conditions and a retained baseline. Keep adherence, schema, and consistency failures visible as separate gates; their scores do not substitute for one another.

Knowledge check

References