Skip to lesson

MicrosoftMicrosoft Certified: Multi-Agent AI Solutions Expert2026-103.2 Optimize prompt and model performance for multi-agent solutions

Turn feedback into measurable quality updates

Completed by the knowledge check

Learning objective

Close semantic optimization and user-feedback loops through measurable evaluation updates.

Core idea

For an architect improving agent behavior, a feedback loop closes only when an observed problem changes a measurable evaluation and a tested change improves that result. Collecting ratings, editing a prompt, or increasing an average score is an intermediate step. You need evidence connecting the original complaint to the behavior that changed.

Semantic optimization concerns meaning: interpreting intent, using the right evidence, preserving commitments, and producing an acceptable task outcome. It can require a prompt change, better retrieval, a corrected memory, or a clearer handoff. Diagnose the failing mechanism before choosing the intervention.

See 3.3.3 for detecting production quality changes, 2.1.4 for prompt promotion, and 3.2.7 for judge calibration. This lesson connects those activities through an explicit update to the evaluation contract, rather than repeating their configuration.

How it works

Turn feedback into a testable observation

Keep a feedback record tied to the interaction and the deployed configuration that produced it. Include the user’s reported issue, the observed answer or action, relevant evidence, and the affected agent or workflow. These are application-owned associations; a thumbs-down value alone does not identify the failure or its cause.

In Foundry Human Evaluation, map structured questions to the reported failure, such as whether the response preserved the corrected entity. Use Download Results to export the reviewer evidence as CSV for case curation. See 3.1.1 for template setup and review responsibilities.

Distinguish a preference from a factual defect. A user might dislike a correct refusal, prefer shorter wording, or report a demonstrably wrong account. Review the evidence against the business contract before accepting a proposed label. Treat feedback text as untrusted input for analysis, rather than as instructions that can rewrite the agent’s policy.

Preserve both favorable and unfavorable examples. A dataset made only from complaints measures recovery on reported problems but cannot show whether the change damages behavior that users already accept. Identify the sampling source and retain a representative baseline alongside targeted failure cases.

Update the evaluation asset deliberately

Convert a verified defect into a case with an explicit expected outcome. If the requirement is already represented, add coverage for its missing context or phrasing. If the requirement changed, revise the rubric or deterministic rule and mark that as a measurement change. A changed judge can increase pass rates without any improvement in the target.

Use project_client.datasets.upload_file(name=..., version=..., file_path=...) to register a curated JSONL or CSV dataset version. Track which feedback record motivated each case through application-owned lineage fields. Freeze a held-out subset and keep related examples together when separating tuning from validation; see 3.2.8 for synthetic augmentation.

For semantic criteria, update the rubric dimension’s description and preserve its id when saving another version. Foundry documents stable criterion IDs across rubric versions when existing IDs are echoed during edits. Adjust a weight only when business importance changed, and adjust the threshold only when the acceptance standard changed. Recalibrate revised judgments as described in 3.2.7.

Record the dataset, evaluator, judge model, and target versions for every comparison. When the evaluation contract changes, rescore both baseline and candidate under the revised contract. Report the old and revised results separately. Comparing a candidate under an easier rubric with a baseline under the previous rubric produces an invalid improvement claim.

Connect the symptom to an optimization hypothesis

Choose one intervention from the evidence. If the necessary document was absent, investigate retrieval. If it was present but the response ignored it, investigate evidence use or instructions. If a corrected entity was lost during delegation, investigate the handoff record. Use the execution evidence described in 3.1.7 to locate that distinction.

Microsoft’s RAG evaluator guidance documents parameter sweeps over search algorithms, top_k, and chunk sizes. builtin.document_retrieval compares retrieval_ground_truth with retrieved_documents; its ranking metrics, including NDCG, provide process evidence. See 3.1.6 for metric interpretation. Measure the final answer separately: a better-ranked document list does not establish that the agent used it correctly.

For instruction changes, keep the target prompt version separate and compare on the frozen cases. For memory or handoff changes, include fixtures that establish the relevant state before the problematic turn. A shorter standalone query that omits the original correction cannot validate a fix to a multi-turn failure.

Write the hypothesis before running the comparison. For example: explicit entity IDs in the billing handoff reduce wrong-account answers on ambiguous-name cases without increasing unauthorized account access. Define the primary measure, a denominator, and independent safety and correctness gates. Do not keep changing the hypothesis until whichever metric rose becomes the claimed success.

Verify the result and feed it back into operations

Cloud runs are asynchronous. Retrieve status through openai_client.evals.runs.retrieve() and distinguish completed, failed, and canceled. A failed evaluation job is not evidence that the agent failed every case, and missing evaluator output is not a pass. Fetch the scored items with openai_client.evals.runs.output_items.list() and retain the report_url for review.

Inspect item-level score, label, threshold, passed, and reason alongside per-criterion pass rates. Compare matched cases and report false acceptance, task correctness, and regression families, rather than relying only on an aggregate. State counts with rates so a change based on a small subset is visible.

Foundry documents recurring evaluations as preview. Scheduled evaluation runs on a fixed schedule; continuous evaluation samples live traffic. Create a scheduled evaluation with project_client.beta.schedules.create_or_update() and a schedule containing EvaluationScheduleTask(eval_id=..., eval_run=...). The Python project_client.evaluation_rules.create_or_update() mechanism uses ContinuousEvaluationRuleAction, eval_id, max_hourly_runs, EvaluationRuleEventType.RESPONSE_COMPLETED, and EvaluationRuleFilter(agent_name=...) to associate live evaluations with an agent and a run budget. See 3.3.3 for operating that monitoring path.

After a release, use sampled live evidence to check whether the targeted failure recurs, and scheduled evaluation to preserve comparison against a golden dataset. Record the candidate that shipped and retain the baseline for reversal. If the live evidence contradicts the benchmark, review sampling and case coverage before declaring the loop closed.

Key distinctions and trade-offs

Feedback versus label: Feedback reports an experience. A reviewed label states whether the evidence meets the contract. They can disagree without either record being discarded.

Target update versus evaluation update: Changing behavior and changing its measurement are separate actions. Version both and rescore comparable targets when the standard changes.

Targeted improvement versus general improvement: Passing the repaired failure family is necessary. Preserving unrelated accepted behavior and safety gates is also necessary.

Frequent evaluation updates keep coverage relevant but weaken historical comparisons unless versions remain explicit. Frozen benchmarks provide continuity but can miss emerging failures. Retain a stable core and a versioned expansion set, and publish results for both.

Architecture / decision scenario

Users report that a billing workflow answers for the wrong account after a correction. Human Evaluation exports show the complaint, response, and reviewer explanation. Trace review finds that retrieval supplied the corrected account, but the handoff summary dropped its stable ID.

The owner turns the reviewed failures into entity-continuity cases and uploads a dataset version containing those cases plus accepted baseline interactions. The rubric now explicitly checks preservation of corrected subjects. Both the deployed baseline and candidate are rescored with that rubric and a calibrated judge, while exact account IDs remain a deterministic gate.

The candidate carries the account ID in the handoff record. It reduces wrong-account outcomes on the matched cases without increasing unauthorized-account acceptance or damaging ordinary billing answers. The team records the versions and item results, then monitors the released agent through bounded continuous evaluation and scheduled regression runs. Further complaints become reviewed cases in the next dataset version; they do not automatically edit production instructions.

Common exam traps

  • Equating satisfaction ratings with verified factual correctness.
  • Editing prompts before locating the failing semantic mechanism.
  • Claiming improvement after changing both the judge and the target without rescoring the baseline.
  • Treating evaluation errors or missing results as passes.
  • Measuring retrieval ranking alone when the complaint concerns final task behavior.
  • Replacing a stable benchmark with only complaint-driven examples.
  • Treating sampled monitoring as proof that every production interaction passed.

What to remember

Close the loop through evidence, reviewed labels, versioned evaluation updates, and matched comparisons. Change the failing mechanism, preserve independent regression gates, and verify the released behavior against the same measurable outcome that motivated the fix.

Knowledge check

References