Automated evaluation with cloud evaluators

Why it matters

Hand scoring does not scale past a few cases; cloud evaluators apply the same criteria to every row and every pull request.

What to practise

Run one script that uploads a JSONL dataset of 89 pre-generated query-response pairs to Microsoft Foundry, defines an evaluation with the built-in intent-resolution, relevance, and groundedness evaluators, starts a cloud run, and polls until it completes. Each evaluator returns a 1–5 score, a pass or fail label against a configurable threshold, and a reason. Review the report in the Foundry portal and look for patterns in the failed items. Then enable the workflow’s pull_request trigger, authenticate it through OIDC federated credentials, and let it post results as a pull-request comment. Optionally, rerun with a stricter threshold.

How it fits

This is the cloud-run side of 3.1.2: a reusable dataset, evaluator mappings, and thresholds. The dataset holds stored responses, so the run scores answers rather than calling the agent as a target. Compare the threshold choice with the regression gate in 3.1.3.