Configure cloud evaluation runs and their judges

Learning objective

Configure Foundry cloud evaluations with datasets, evaluators, target definitions, and thresholds; distinguish platform evaluators from SDK-local model judges.

Core idea

A cloud evaluation is a contract between the data being tested, the target that produces output, and the evaluators that score it. For an architect, a completed run is useful only when those contracts match the intended experiment. A correctly executed judge over the wrong response field produces misleading evidence.

Microsoft Foundry cloud evaluations separate an evaluation definition from its runs. The definition declares data shape and testing criteria; a run supplies data and, when required, a model or agent target. Keep these objects distinct from the Python Azure AI Evaluation SDK’s local evaluator orchestration. Both can use a model judge, but a shared metric name does not make their configuration interfaces interchangeable.

How it works

Select stored outputs or generated outputs

If an evaluation invokes a model deployment, assign Foundry User at the Foundry account scope. A project-only role assignment does not authorize model inference.

Use a dataset evaluation when you already have the responses to score. Foundry supports JSONL and CSV datasets, file-backed or inline. For a reusable test suite, project_client.datasets.upload_file() takes name, version, and file_path, returning a dataset whose id can be used by subsequent runs.

Declare the input fields in data_source_config.item_schema. A JSONL dataset run uses data_source.type="jsonl"; CSV uses "csv". A file source uses source.type="file_id" and its id; an inline source uses "file_content". These identify where examples come from, not which model should generate their answers.

Use a target evaluation when you have inputs and want Foundry to generate responses during the run. The data source type is azure_ai_target_completions. The model or agent target receives messages built from input_messages; {{item.query}} substitutes a dataset field into the message template. Do not silently turn a stored-response experiment into a target experiment, because the latter generates different output rather than rescoring the recorded output.

Pin the target independently of the judge

An agent target uses target.type="azure_ai_agent", name, and an optional version. If you omit version, Foundry selects the latest agent version. For a regression experiment, specify the version so a candidate created between runs cannot change what the baseline name resolves to.

Agent-target evaluation supports prompt agents and hosted agents. A hosted agent that uses the responses protocol accepts the structured message template. An invocations-only hosted agent instead needs freeform input_messages matching its /invocations request body. If an agent supports both protocols, the evaluation service defaults to responses. Copying a prompt-agent template into an invocations-only test therefore changes the interface contract, not merely the prompt.

Keep the target deployment separate from the judge deployment. The target generates the behavior under test. An AI-assisted evaluator uses a judge to assess that behavior; selecting a different judge can change scores even when target responses remain fixed. Record both choices alongside the dataset version and evaluation definition.

Map each evaluator to its actual inputs

Create the evaluation with openai_client.evals.create(), passing data_source_config and testing_criteria. A platform built-in criterion identifies type="azure_ai_evaluator", a reporting name, and an evaluator_name such as builtin.coherence or builtin.violence.

data_mapping connects the evaluator’s expected inputs to the test data. For stored outputs, map response to {{item.response}}. For generated outputs, set include_sample_schema=True and map a text response to {{sample.output_text}}. For agent evaluators that need tool calls and interaction structure, use {{sample.output_items}}; their behavioral meaning is covered in 3.1.3 and 3.1.4.

The dataset schema validates the shape of evaluation inputs. It does not prove that an agent’s returned business JSON satisfies its response schema. Keep the deterministic result validator described in 3.1.3 separate from the input-data contract.

Initialization also depends on the evaluator. AI-assisted coherence needs a judge; the cloud dataset example supplies initialization_parameters={"model": MODEL_DEPLOYMENT}. Its violence criterion does not supply a judge deployment. Do not add a model judge to every criterion merely because evaluation is associated with language models.

Microsoft’s cloud workflow examples use initialization_parameters.model, while the evaluator reference examples document deployment_name. Preserve the configuration of the specific documented SDK workflow you implement rather than merging the two examples. This difference is not the local SDK’s model_config interface.

Establish thresholds before reading results

A score and a pass decision are different outputs. Coherence returns a 1–5 score with default pass threshold 3; a score at or above the threshold passes. That default describes evaluator behavior, not your application’s release policy. A regulated workflow can require stronger performance or zero tolerated schema failures even when a general quality metric passes.

Specify supported thresholds through the chosen evaluator’s initialization contract. For example, Foundry’s custom code-based evaluator contract requires pass_threshold and deployment_name, even though the code does not call an LLM. A binary contract validator can return 0.0 or 1.0 and use pass_threshold=1.0. The deployment name is required for service orchestration; it does not turn deterministic grading into a model judgment.

Keep an evaluator threshold separate from an aggregate promotion gate. An application’s rule such as “no contract failures in the mandatory cases” belongs to the release decision over per-item results. A passing average cannot override that rule. Decide how evaluator errors and missing scores affect the gate before the run; they are not positive quality evidence.

Start a run without confusing local orchestration

openai_client.evals.runs.create() binds eval_id to the run’s data_source. Poll with evals.runs.retrieve() and inspect the terminal status and per-item results. Foundry stores cloud evaluation results in the project for SDK and portal review. Starting the SDK call on a laptop does not make this a local evaluation.

Microsoft documents the local SDK workflow under Foundry (classic); its portal guidance applies to that experience. The Python azure.ai.evaluation package exposes evaluator classes such as RelevanceEvaluator(model_config) and the evaluate() function. evaluate() receives callable evaluators and supports evaluator_config.column_mapping with expressions such as ${data.queries}. This is not the cloud API’s testing_criteria catalog or {{item.query}} mapping syntax.

“Local” describes where evaluator orchestration runs, not where every score is computed. AI-assisted quality evaluators such as RelevanceEvaluator call the judge endpoint and deployment identified by azure_endpoint and azure_deployment in model_config. Risk and safety evaluators such as ContentSafetyEvaluator(credential=…, azure_ai_project=…), and GroundednessProEvaluator (preview), instead take azure_ai_project and use the Foundry back-end evaluation service even under local orchestration.

The azure_ai_project argument to evaluate() can also track local evaluation results in Foundry. That result-tracking option is distinct from the project information required by a platform-backed evaluator; neither means that evaluate() becomes a cloud evaluation job.

Key distinctions and trade-offs

Dataset versus target: A dataset experiment scores recorded outputs; a target experiment generates output during the run.

Target versus judge: One produces the candidate behavior; the other scores it. Version both independently.

Execution location versus scoring mechanism: Cloud runs use catalog criteria and mappings; locally orchestrated evaluate() receives Python callables. A local callable can use a judge deployment through model_config or a platform evaluation service through azure_ai_project, so local orchestration alone does not identify the scoring mechanism.

Item threshold versus release gate: A metric’s pass decision is one input to promotion, not the complete business acceptance rule.

A target run adds a generation step that a dataset run lacks, while the judge and the release gate stay separate choices.

Architecture / decision scenario

A support team compares a candidate agent version with its accepted version. It uploads one versioned input dataset and creates an evaluation definition with explicit criteria. Both runs use azure_ai_target_completions, azure_ai_agent, and pinned agent versions. The input template substitutes the same questions for both targets.

The definition maps generated text to {{sample.output_text}} for coherence and maps full structured output for the agent behavior checks. The team fixes its judge deployment and thresholds, then evaluates each version through evals.runs.create(). It preserves per-item failures and blocks promotion when a mandatory contract case fails, even if mean coherence improves.

A developer can also reproduce a narrow quality check with RelevanceEvaluator(model_config) locally. For a locally orchestrated safety check, the developer instead supplies azure_ai_project to ContentSafetyEvaluator, which uses the platform service. The team records the scoring mechanism and mappings for each path before comparing results. It does not paste ${data.*} expressions into a cloud criterion or assume uploading local results reruns them in the cloud.

Common exam traps

• Omitting an agent version while claiming a fixed baseline.
• Mapping a generated response to a dataset column that contains no generated output.
• Giving a tool evaluator only the final text instead of structured interaction data.
• Confusing input dataset schema validation with output contract validation.
• Treating every evaluator as requiring a model judge.
• Equating local orchestration with offline execution or cloud result tracking with cloud execution.

What to remember

Choose stored or generated outputs first, pin the target, and map every evaluator to the correct fields. Use the initialization contract for that interface, preserve the judge choice, and separate item thresholds from promotion rules. Distinguish cloud jobs, locally orchestrated model judges, and platform-backed evaluators invoked from local code; they do not share one configuration contract.

Knowledge check

References

• Microsoft, Introduction to cloud evaluation with Microsoft Foundry SDK
• Microsoft, Evaluate datasets with the Microsoft Foundry SDK
• Microsoft, Evaluate models and agents with the Microsoft Foundry SDK
• Microsoft, General purpose evaluators
• Microsoft, Custom evaluators
• Microsoft, Azure AI Evaluation client library for Python
• Microsoft, Local evaluation with the Azure AI Evaluation SDK (classic)
• Microsoft, Get cloud evaluation results with the Microsoft Foundry SDK
• Microsoft, Set up permissions for Microsoft Foundry evaluation workflows