Promote prompt versions from comparative evidence

Learning objective

Version and compare prompts using A/B evidence, regression tests, and promotion gates.

Core idea

A prompt revision is a behavioral change. Version the configuration that produces a response, compare it with an identified baseline, and promote it only when it meets declared quality and safety criteria. An impressive sample answer is evidence about one case, not a release decision.

A/B evidence requires comparable conditions. Changing the prompt, model, evidence, and evaluator together tests a release bundle, not the effect of prompt wording.

See 2.1.1 through 2.1.3 for the role contract, runtime assembly, and defensive policy. This lesson owns the change process that preserves those contracts when instructions evolve.

How it works

Identify exactly what changed

Foundry’s prompt-agent quickstart uses project.agents.create_version() with a PromptAgentDefinition. The returned agent includes name, id, and version. Record the exact baseline and candidate version rather than relying on an unqualified agent name to identify behavior.

Also record the model deployment, tool and parameter descriptions, output schema, runtime template version, and retrieval configuration when they affect the test. A prompt-agent version identifies the agent configuration; it does not automatically identify every external data source or application validator. Keep those dependencies in your release evidence.

Record the dataset identifier, expected outcomes, evaluator configuration, and evaluation model alongside each version. A changed rubric can otherwise look like a prompt improvement.

Compare baseline A and candidate B fairly

Use a representative held-out dataset for the release comparison. Include ordinary successes, missing evidence, contradictions, malformed inputs, and policy-override attempts. Preserve cases that previously caused defects as a regression set. Separate cases used to tune the wording from cases used to approve the release; repeatedly optimizing against the approval set weakens its independence.

For a prompt-only comparison, keep model deployment, inference settings, tools, evidence, and evaluators fixed. Run A and B on the same input records and inspect paired outcomes. If stochastic responses affect the decision, repeat enough cases to understand variation rather than treating one favorable sample as conclusive. These are experimental-design choices owned by your application.

For online A/B testing, Azure App Configuration variant feature flags assign variants through Allocation to users, groups, or percentile buckets, such as 10% candidate and 90% baseline. Map each variant to an identified prompt configuration and retain version attribution in your application’s outcome records. DefaultWhenEnabled handles unassigned percentiles; setting Enabled to false sends users to DefaultWhenDisabled, the kill-switch variant. This allocation mechanism does not establish metric analysis. The Foundry optimizer compares candidates offline on a dataset; selecting one does not split production traffic.

Use the Foundry comparison mechanism

Foundry’s agent optimizer evaluates a baseline and generates alternative configurations. The Agent Service overview labels the optimizer preview; the dedicated optimizer overview and prompt-agent optimization quickstart omit that label. Treat the feature as preview for this lesson rather than inferring general availability from an omitted warning.

For prompt agents, use the portal optimization wizard and select the baseline agent version, dataset, evaluators, optimization model, and evaluation model. The prompt-agent optimization quickstart documents registered datasets, uploaded JSONL, or data generated from agent traces. Its dataset must use the evaluator-required column names because the wizard does not support column mapping.

The optimizer scores the baseline and candidates against the same dataset. Under Optimization runs, compare the before-and-after instructions and per-evaluator scores. If Compare across models is enabled, inspect each model’s results separately. A cross-model result is evidence about a prompt/model combination, not a controlled prompt-only comparison.

Prompt-agent function-calling tools execute on the client side. During tool-description optimization, the optimizer can improve tool and parameter descriptions but cannot execute or evaluate those tools. Therefore, a better tool-description score does not establish that the integration makes the correct external change. Test tool execution and permission checks in your application before release.

Enforce regression and promotion gates

Use deterministic checks for exact contract requirements: required output fields, allowed decisions, source membership, and prohibited action proposals. Use task-quality evaluators for qualities that require judgment. Microsoft’s agent-evaluator reference includes tool_input_accuracy, and labels task_adherence and task_completion as preview. The latter measure adherence to instructions and completion of the requested task, respectively.

Keep system instructions and tool definitions in the evaluation data when the chosen evaluator needs them. An evaluator asked to judge policy adherence without the policy receives an incomplete test. Preserve evaluator explanations and failing examples, not only the final pass rate, so a reviewer can see which behavior changed.

The optimizer’s composite score rescales evaluator scores to 0–1, reverses lower-is-better scores, and averages across evaluators and tasks. This average can hide a severe failure in a small but important slice. Microsoft’s interpretation bands are: a delta below 0.03 is noise, 0.03–0.10 is moderate, 0.10–0.20 is significant, and above 0.20 is major, often indicating a poor baseline. These bands do not replace application statistical analysis or per-slice safety gates.

Define gates before looking at the winning candidate. For example, require no regression on the known injection cases, full schema validation, acceptable task accuracy, and an agreed latency and token budget. These thresholds are application policy, not built-in universal Foundry promotion requirements. Evaluate each gate separately; a quality improvement cannot compensate for a prohibited tool action.

Use the completed run’s Token usage view to inspect measured input, output, and total tokens by phase and model. It reports token counts, not a final currency charge. A missing value means usage was not attributed or measured, not that the work was free. Longer instructions can increase cost even when they improve quality.

Separate promotion from activation

The prompt-agent optimization quickstart documents Promote candidate and Promote to agent version. Promotion creates a new prompt-agent version with the selected configuration. If the agent is pinned to a specific version, that candidate receives no traffic until you set it as the active version in the agent’s details.

Preserve the approved baseline and version attribution when activating the candidate. Reverting traffic to a known version should be an explicit deployment decision, not an attempt to reconstruct its wording from memory. Verify behavior after activation using the same contract checks and monitored outcomes used for approval.

Prompt flow variants are a separate classic-portal mechanism. Its variants apply to LLM tool nodes, and Microsoft schedules Prompt flow retirement for April 20, 2027, recommending migration to Microsoft Agent Framework. Do not design a Foundry prompt-agent release process around a classic variant as though it were an agent version.

Key distinctions and trade-offs

The change artifacts answer different questions:

Larger datasets improve coverage at greater evaluation cost. Changing several configuration dimensions validates a release bundle while reducing causal clarity about prompt wording.

Architecture / decision scenario

A claims team proposes a reviewer prompt that explains missing evidence more clearly. Baseline A and candidate B use the same model deployment, template, evidence records, and schema. The approval set includes ordinary claims, contradictory invoices, missing amounts, and previously observed instruction overrides.

The optimizer reports a composite-score increase of 0.06 for B, within Microsoft’s moderate-improvement band. The team inspects per-evaluator outcomes and runs application regression checks. B improves explanations but proposes approval on one injected document case. That case fails the declared no-policy-regression gate, so the team retains A even though B’s average is higher.

After revising the candidate, the team reruns the comparison on held-out cases and verifies tool execution separately because optimizer tool-description evaluation does not execute client-side tools. The passing candidate is promoted to a new version. The pinned agent continues using A until the approved deployment activates the candidate; result records retain the selected version for monitoring and rollback decisions. For an online prompt comparison, App Configuration allocates 10% of users to the candidate and 90% to A. The application maps each variant to its version and uses A as DefaultWhenDisabled if the flag is disabled.

Common exam traps

• Calling a simultaneous prompt/model change a prompt-only A/B test.
• Using tuning examples as the only release regression dataset.
• Treating a higher average score as proof that every policy gate passed.
• Assuming tool-description optimization tests real tool execution.
• Confusing promotion with automatic traffic activation for a pinned agent.
• Treating classic Prompt flow variants as Foundry prompt-agent versions.

What to remember

Compare matched inputs and inspect critical failures independently of averages. Preserve regression evidence and the known baseline through promotion and activation.

Knowledge check

References

• Microsoft, Quickstart: Create a prompt agent
• Microsoft, What is Microsoft Foundry Agent Service?
• Microsoft, Agent optimizer in Foundry Agent Service overview
• Microsoft, Quickstart: Optimize a prompt agent
• Microsoft, Agent Evaluators for Generative AI
• Microsoft, Tune prompts using variants (classic)
• Microsoft, Use variant feature flags