Turn human reviews into actionable evidence

Learning objective

Design human evaluations for Foundry agents with reviewer permissions, response sampling, evaluation templates, feedback, and escalation.

Core idea

Human evaluation answers questions whose meaning depends on business context: whether an answer resolves the actual request, whether an action respects policy, and whether uncertainty warrants escalation. For an architect, the task is to turn those judgments into comparable evidence rather than collect unstructured approval from whoever happens to try the agent.

Microsoft Foundry provides evaluation templates and response-level feedback in the agent builder’s Preview web app. You still design the response sample, rubric, and follow-up ownership. A favorable rating is evidence about a reviewed response; it is not permission for the agent to execute a consequential action. See 1.1.6 for execution checkpoints.

How it works

Separate template management from reviewer access

The human-evaluation workflow requires a Foundry project with an agent and Application Insights configured. Microsoft specifies Foundry Project Manager or higher on the project for creating and managing evaluation templates. Human reviewers need at least Foundry User on the project and Reader on the Foundry account to access the preview web app and submit feedback.

These are distinct scopes and duties. Giving someone a link to Preview does not supply their role assignments. Giving every reviewer template-management permissions increases their authority beyond the review task. Use Access control (IAM) > Check access on each resource to inspect the reviewer’s role assignments and scopes. Before scheduling a review session, verify that the intended reviewer identity can open the agent and save feedback.

Microsoft renamed Azure AI User to Foundry User and Azure AI Project Manager to Foundry Project Manager; the role IDs and core permissions are unchanged. Use the complete role names when interpreting an exam option or documenting access. The Preview button opens the agent’s browser-based chat interface; its label does not establish the release status of evaluation features.

Define a response sample before collecting ratings

Choose the unit of judgment: one response in its conversation context. Assemble representative inputs across agent roles, task types, language requirements, ambiguous requests, refusals, and failure paths. Include a routine-traffic sample to estimate typical behavior and a separately labeled risk-focused sample to expose rare but costly defects.

This is evaluation design, not a claim that Foundry’s human-feedback UI automatically draws a random production sample. In the documented workflow, reviewers enter an input, run the agent, and select a response’s Feedback button. They can navigate to previous responses, skip responses, or submit multiple evaluations for the same response.

Those choices change what a score means. Ten ratings of one response are ten evaluation instances, not ten independently sampled agent responses. Skipped cases can bias results if reviewers avoid long or confusing failures. Keep a separate sample inventory with a case identifier, agent version, task category, and expected review coverage, and reconcile it against the saved results. These are fields in your review process, not reserved Foundry feedback field names.

Use repeated review of a shared subset to calibrate reviewers. If one reviewer rates an unsupported claim as helpful and another rates it as incorrect, clarify the rubric before aggregating their scores. Preserve disagreements that reveal an ambiguous business policy instead of forcing a consensus that hides the policy gap.

Make the template carry the decision

In the agent’s Human Evaluation tab under Evaluation, create a named template with a description. Supported question types are thumbs up/down, slider, multiple choice, and free-form text. Use a separate question for each criterion that can drive a different fix.

For example, a thumbs-down accuracy rating should not also mean that escalation was unnecessary. Use a multiple-choice question for the response outcome, a slider with defined anchors for usefulness, and free-form text for the evidence behind a serious defect. A question such as “Does this require policy-owner review?” makes escalation visible in the feedback without pretending that a rating invokes an approval workflow.

A created template starts Inactive. Select Set as active before collecting feedback. Only one template can be active at a time; activating another deactivates the previous one. Record which template applies to each review batch, and avoid changing its questions halfway through a comparison. Otherwise, differences in scores might reflect different criteria rather than different agent behavior.

Save feedback and route serious findings

After the agent responds in Preview, Feedback opens the active template. Reviewers can answer some or all questions, then choose Save or Cancel. Missing answers therefore need explicit handling in analysis: an unanswered accuracy question is not an accuracy pass.

Builders inspect saved instances in Evaluation Results. The JSON Output includes the timestamp, user prompt, agent response, template questions, and reviewer answers. Download Results exports a CSV with one row per evaluation instance. Use these documented artifacts to support adjudication and trace a finding back to the response that produced it.

Define escalation as a team decision over those artifacts. For example, a reviewer selects a rubric outcome indicating a prohibited action, adds the relevant policy and response excerpt, and the evaluation owner routes that instance to the policy owner. That owner decides whether to correct the prompt, narrow tool exposure, add a regression case, or stop promotion. This routing is your operational process; saving Foundry feedback alone does not establish that remediation occurred.

Evaluation data resides in Application Insights and follows its retention policy. Export evidence when the review’s retention requirement exceeds that policy. Deleting a template prevents retrieval of that template and its associated results from the portal, so deletion is not a cleanup step to perform before preserving review evidence.

Key distinctions and trade-offs

Feedback versus approval: Review feedback measures behavior after a response. Execution approval authorizes a specific action before it occurs.

Representative versus risk-focused sampling: Representative cases support estimates of ordinary quality. Deliberately enriched failure cases reveal weaknesses but do not estimate their production frequency without accounting for the sampling design.

Response versus evaluation instance: Multiple reviewer submissions improve understanding of disagreement, but they do not increase the number of independent responses tested.

Rubric stability versus flexibility: A stable active template supports comparisons. Editing it supports evolving requirements, but requires a separately identified review batch.

Architecture / decision scenario

A claims-support solution uses a triage agent and a policy specialist. Before promoting a candidate, the evaluation owner selects routine claim questions, ambiguous eligibility cases, and cases where the specialist should request human review. The owner records these groups separately and asks two reviewers to score an overlapping calibration subset.

The template manager has Foundry Project Manager on the project. Reviewers receive Foundry User on the project and Reader on the account. The manager activates a template with accuracy, policy compliance, usefulness, and escalation questions; reviewers run selected inputs in Preview and save response feedback.

One response gives a confident eligibility decision despite missing evidence. Its escalation answer and explanatory text identify the defect. The owner exports the corresponding CSV instance, routes it to the policy owner, and adds the adjudicated case to the regression suite. The promotion decision accounts for unanswered questions and duplicate reviews rather than reporting every saved instance as an independently passing response.

Common exam traps

• Treating access to the preview URL as sufficient reviewer authorization.
• Giving reviewers template-management permissions because they need to submit feedback.
• Forgetting that a created template starts inactive or that only one is active.
• Counting duplicate evaluations as additional sampled responses.
• Treating skipped or unanswered questions as passing results.
• Assuming a saved escalation rating approves an action or completes remediation.

What to remember

Assign reviewer access at the documented scopes, activate a stable template, and control which responses enter the sample. Analyze saved instances with their missing answers and repeated reviews intact. Use the JSON and CSV evidence to assign an escalation owner and preserve adjudicated failures for later tests.

Knowledge check

References

• Microsoft, Human evaluation for Microsoft Foundry agents
• Microsoft, Set up permissions for Microsoft Foundry evaluation workflows