Build a multi-agent evaluation quality gate

Why it matters

Individual agents can each answer plausibly while the whole journey fails, so a release gate has to measure component, handoff, and system quality and own the decision in code.

What to practise

Complete an evaluator factory with intent-resolution, task-adherence, and response-completeness evaluators plus a journey-coherence judge, all on a Microsoft Foundry deployment with a passwordless credential, and run them through the SDK’s batch evaluate() over a synthetic JSONL dataset. Compute exact agreement between the judge and human labels, then compare the measured metrics with absolute and delta thresholds. Make one candidate response contradict an earlier handoff and confirm the score changes and the gate decides from that measured score, not from a scenario label. The Foundry agent evaluators are preview.

How it fits

This is an SDK-local judge run, the counterpart to the cloud runs in 3.1.2. Classifying each low-scoring row as component, handoff, or system failure is 3.1.7. For 3.2, the calibration gate is 3.2.7 and the canary, regression, and historical-failure cases are 3.2.8.