Require evaluation evidence before releasing agents

Learning objective

Gate releases on automated agent evaluations and evidence-linked quality criteria.

Core idea

An evaluation produces measurements; a release gate applies an acceptance policy to those measurements for a specific candidate. Keep the two explicit. A completed evaluation is not necessarily a passing evaluation, and a passing evaluation of one agent version is not evidence for another release set.

An evidence-linked criterion states what must pass, how it is measured, and which retained result proves it. For example, a purchase workflow might require every prohibited-operation case to pass an application assertion, plus an acceptable tool-selection pass rate on a versioned dataset. Those are your release requirements, not universal Microsoft thresholds.

See 3.1.2 for cloud evaluation configuration and 3.2.7 for judge calibration. This lesson focuses on consuming that evidence without promoting the wrong candidate, hiding individual failures in an average, or interpreting an incomplete run as success.

How it works

Bind the run to the candidate

Microsoft Foundry cloud evaluation uses AIProjectClient and get_openai_client() to obtain the evaluation client. In Python, openai_client.evals.create() defines the evaluation’s data_source_config and testing_criteria; openai_client.evals.runs.create() submits a run of that definition. Record both the evaluation ID and the run ID in the release evidence.

For generated agent responses, the target workflow uses the data source type azure_ai_target_completions. An agent target uses type: "azure_ai_agent", name, and version. The version field is optional; if omitted, evaluation uses the latest agent version at run time. Pin the intended version so the run evaluates the candidate. The target documentation covers prompt agents and hosted agents; hosted-agent input formatting depends on the supported protocol.

Retain an application release record that joins that target to the code revision, model deployment, prompt, tools, orchestration, and environment actually tested. These are your evidence fields, not a prescribed Foundry release schema. See 4.3.2 for compatible immutable release sets. If the candidate’s tool implementation changes after evaluation, the prior run no longer establishes acceptance of that combination.

Match criteria to observable behavior

A gate should use evidence that can establish its claim. builtin.task_adherence assesses adherence to instructions and is marked preview. Tool Selection and Tool Input Accuracy evaluate different process questions: choosing the right tool does not establish correct arguments. The agent-evaluator documentation marks evaluators including Task Completion, Task Adherence, Output Quality, and Tool Use Quality as preview, while Quality Grader is deprecated and cannot be run.

Configure each testing criterion with the documented evaluator identifier and required mappings. {{item.field_name}} references dataset fields. For generated output, {{sample.output_items}} supplies structured agent output, including tool calls and results; {{sample.output_text}} supplies plain text. A criterion about tool behavior needs the interaction evidence, with tool_definitions where the evaluator requires it. Do not feed only a final sentence to a criterion intended to inspect the tool sequence.

Preview evaluators carry the documented preview restrictions, including no service-level agreement. Keep deterministic assertions for safety-critical application invariants rather than treating a judge’s approval as authorization to perform a side effect. See 4.1.8 for security testing and red teaming and 4.2.3 for tool authorization. Evaluations measure those behaviors; they do not replace their enforcement mechanisms.

Define release policy before examining the candidate’s scores. Record required cases, minimum per-criterion rates, allowable regression from a baseline, and treatment of missing results. Judge thresholds convert scores to pass or fail; the release policy decides whether the resulting distribution is acceptable. Adjusting either after a failed candidate changes the question the test answers.

Wait for a terminal run and retrieve all evidence

Cloud evaluation runs are asynchronous. Poll openai_client.evals.runs.retrieve(run_id=..., eval_id=...). Microsoft’s example stops on completed, failed, or canceled, and raises an error when the terminal status is not completed. A successful submission proves only that the run was accepted, not that scoring finished.

After completion, use openai_client.evals.runs.output_items.list(run_id=..., eval_id=...). Iterating the Python list operation retrieves all pages. Keep run.report_url as a human review link, alongside the retrieved results. A report URL alone is not a machine-readable acceptance decision.

Per-item results include score, label, threshold, and, for LLM judges, a reason. The documented example also includes passed. Aggregate evidence includes result_counts and per_testing_criteria_results, with per-criterion passing and failing counts and pass_rate. Preserve both levels: aggregate rates guide comparison, while individual items show which requirement failed.

Require the expected case count and evaluator coverage before applying thresholds. Treat a failed run, canceled run, exhausted polling deadline, missing item, or missing required criterion as unavailable evidence. Your pipeline should fail the quality decision rather than interpreting absence as zero failures. This is application gate logic built around the documented results API.

A run that is not completed, or completed evidence with a missing item or criterion, fails the quality decision before any threshold is applied.

Connect measurement to promotion

Microsoft also documents the GitHub Action microsoft/ai-agent-evals, with the example pinned to @v3-beta, and the Azure DevOps task AIAgentEvaluation@2. Both documentation pages carry the preview notice. Both accept agent-ids in agent-name:version format, data-path for the dataset file containing evaluators and queries, and deployment-name for the evaluation model deployment. The optional baseline-agent-id selects the comparison baseline; otherwise, the first listed agent is used. Their run summaries report per-metric scores, confidence intervals, and pairwise statistical comparisons when evaluating multiple agents. These pages describe reporting without documenting a release-threshold failure mechanism, so retain an explicit acceptance decision.

Implement a results-processing pipeline job that retrieves the results, applies the recorded policy, writes the decision evidence, and returns failure when acceptance is not established. See 4.3.3 for Invoke REST API and Invoke Azure Function checks that consult candidate-linked evidence before promotion.

Azure Pipelines’ Evaluate artifact check supports container image artifacts and their policy evaluation. It is not a general Foundry agent-quality evaluator. Use a results-processing job or evidence service for behavioral acceptance instead of assuming that a container artifact check reads agent evaluation scores.

Keep human approval tied to the same evidence package. An approver should be able to open the report, identify the failed cases, and verify the candidate and baseline. Environment approvals and identity configuration belong to 4.3.8; traffic promotion belongs to 4.3.3. A quality decision should survive that handoff without being reduced to an unqualified green icon.

Key distinctions and trade-offs

Run completion versus acceptance: status: "completed" establishes that the evaluation finished. Your criteria and policy establish whether the candidate passes.

Judge threshold versus release threshold: An evaluator threshold classifies a score. A release threshold specifies how many classifications, regressions, or critical failures are acceptable.

Aggregate quality versus critical coverage: A high overall rate can coexist with failure on the only case that protects a sensitive operation. Require critical-case outcomes separately.

More criteria increase cost and review effort. Select criteria that change a release decision, preserve a stable baseline, and reserve expensive evaluation for candidates that already pass the fast tests in 4.3.6. Do not lower coverage simply because an evaluator is slow.

Architecture / decision scenario

A claims workflow changes its specialist prompt and reimbursement tool. CI first passes tool and workflow tests, then submits evaluation against the explicit candidate agent name and version. The dataset contains ordinary claims, malformed tool arguments, and prohibited reimbursements.

The gate polls until completed, retrieves every output page, and confirms that all required cases and criteria are present. It compares tool-selection evidence and instruction adherence with the recorded baseline; the latter uses the documented preview evaluator with its status acknowledged. Application assertions independently require zero prohibited reimbursements. That zero is the team’s criterion, not a Foundry default.

The aggregate rate improves, but one prohibited case fails. The gate records the item, its tool interaction, evaluation and run IDs, candidate record, and report URL, then returns failure. Production approval cannot turn an incomplete or failing quality package into a passing measurement. After a fix changes the tool, CI evaluates the changed candidate again rather than reusing the successful rows from the previous run.

Common exam traps

• Promoting after evaluation submission instead of terminal completion.
• Equating completed with all criteria passing.
• Passing text-only output to a tool-process evaluator.
• Ignoring pagination or missing required cases.
• Reusing evidence after a release dependency changes.
• Treating Evaluate artifact as a universal agent-quality gate.

What to remember

Bind evaluation to an explicit candidate, wait for completion, retrieve complete item and aggregate evidence, and apply predeclared acceptance criteria. Keep critical assertions separate from average quality. Link the resulting decision to the run, report, dataset, and compatible release set that actually produced it.

Knowledge check

References

• Microsoft, Introduction to cloud evaluation with Microsoft Foundry SDK
• Microsoft, Evaluate models and agents with the Microsoft Foundry SDK
• Microsoft, Get cloud evaluation results with the Microsoft Foundry SDK
• Microsoft, Agent evaluators for generative AI
• Microsoft, Pipeline deployment approvals
• Microsoft, How to run an evaluation in GitHub Action
• Microsoft, How to run an evaluation in Azure DevOps