Stage releases and roll back regressions

Learning objective

Gate releases on evaluation pass rates, roll out in stages, and roll back changes that regress.

Core idea

An evaluation result authorizes a next step, not unlimited production exposure. Define what must pass before deployment, how much traffic the candidate can receive, which signals permit expansion, and which conditions restore baseline. Name the people and application controls that make those decisions executable.

Anthropic distinguishes capability evaluations from regression evaluations. Capability tests ask what the agent can do; regression tests ask whether it still handles tasks it previously handled. A release process needs the latter even when the candidate’s headline capability score improves.

How it works

Turn evaluation criteria into a release decision

The release record binds an immutable candidate bundle to dataset and grader revisions, trial policy, result artifacts, thresholds, and approvers. Include the prompt revision, Messages API model, request settings, tools, and harness revision from 4.3.1. Approving a score without its bundle leaves the deployment free to serve something else.

Gate on the regression suite: Anthropic expects a nearly 100% pass rate, and a decline signals something broken. Block promotion while that decline is investigated. Capability evaluations deliberately start at low pass rates on difficult tasks; their scores measure improvement rather than supply a regression pass floor. Separately, a team might require 95% passes on a defined application acceptance suite, alongside stricter critical-case checks. That percentage is an application threshold, not Anthropic’s regression expectation.

Specify the denominator and trial aggregation. A task-level pass rate and an all-trials-success rate answer different questions; see 4.3.3. The release pipeline must evaluate the same statistic the owner approved. A candidate cannot satisfy a one-attempt service requirement by substituting its best-of-five result.

Automate the decision in the team’s deployment pipeline: evaluate the declared thresholds, retain the result, and stop promotion if they fail. Distinguish a grader failure from an agent failure without turning missing results into passes. If the evaluation infrastructure is broken, repair or rerun it under the approved policy before authorizing exposure.

Inspect critical slices independently of the average. A model can improve common tasks enough to lift its overall score while breaking a rare action whose consequences dominate the deployment decision. A critical failure threshold is a veto, not a weight that abundant easy successes can cancel.

Choose stages by the uncertainty they resolve

An offline regression suite checks known cases without affecting users. Passing it makes the candidate eligible for a controlled production stage. It does not establish actual user benefit or guarantee coverage of every production condition.

A shadow stage runs the candidate on live inputs while baseline remains responsible for the user-visible result. It can reveal traffic-mix, resource, and grading problems before users receive candidate answers. Budget the duplicate inference work, and prevent shadow tool calls from performing live writes. A shadow workflow that issues a real refund has production impact despite hiding its final response.

A canary exposes a limited population to the candidate for a defined period. Google’s SRE guidance defines canarying as a partial, time-limited deployment evaluated against a control. Select its size and duration from traffic diversity, load patterns, and the time signals need to develop. A handful of successful requests during an idle hour does not test peak-load behavior.

An online A/B experiment measures the intended benefit when sufficient traffic and a suitable outcome exist; see 4.3.2. A canary and an A/B experiment can share routing infrastructure, but their decision questions differ. Detecting an operational regression at small exposure does not imply enough statistical information to establish a modest completion-rate improvement.

Stage selection is deliberate. A low-risk prompt adjustment might need a smaller operational sequence than an agent-harness change that alters external actions. Record the uncertainty each stage resolves and the conditions for moving forward rather than treating every percentage as a mandatory ritual.

Evaluate the candidate separately from the fleet

Keep measurements attributable to the candidate bundle and concurrent control. Google demonstrates that a small failing canary can be nearly invisible in an aggregate service error rate. Break down quality, errors, latency, and resource use by served variant before expanding exposure.

Predeclare the observation window, required volume, promotion rule, and rollback thresholds. For example, a team might use 5%, 25%, and full exposure, requiring a peak-load period and sufficient completed tasks at each intermediate stage. Those percentages are illustrative rollout choices, not vendor defaults.

Preserve a known baseline through the expansion. A changed traffic mix, shared dependency outage, or overlapping release can contaminate the comparison. Google’s guidance warns about shared failure domains and overlapping canaries. Pause expansion while attribution is uncertain instead of assuming every difference is either the model’s fault or irrelevant noise.

The release owner approves benefit; the operational owner can halt expansion or restore baseline under the approved rules. A risk owner reviews critical quality failures. These responsibilities belong in the release record, including who can act when the experiment owner is unavailable.

Restore the executable baseline

Implement routing through an application-controlled mapping from assigned population to immutable bundle. The rollback action restores the validated baseline’s prompt, model, configuration, and compatible harness together. Moving only the prompt back can leave an untested combination live.

Verify rollback before broad exposure: the baseline remains deployable, the routing change takes effect, served-bundle telemetry confirms the change, and in-flight work has a defined completion or cancellation policy. A pinned model ID preserves model identity; it does not guarantee availability indefinitely, since Anthropic gives each ID its own retirement schedule.

Rollback changes future execution. It does not undo an external write already completed by the candidate. Include incident containment and any necessary reconciliation in the operating procedure. Preserve candidate transcripts and outcomes for diagnosis, then turn confirmed failure cases into regression tests before trying the repaired release. Anthropic also describes high-scoring capability evaluations graduating into continuously run regression suites: demonstrated capability becomes behavior the release must preserve.

Each stage either expands exposure or restores baseline under its rollback thresholds, and confirmed failure cases are turned into regression tests.

Key distinctions and trade-offs

Eligibility versus promotion: Offline passing authorizes the next stage. Live observations determine whether expansion remains acceptable.

Canary versus benefit experiment: Both compare exposed populations, but a short canary principally limits operational risk. Establishing benefit requires the sample and inference plan appropriate to its effect.

Rollback versus remediation: Restoring baseline limits further candidate execution. Repairing an incorrect refund, message, or record requires a separate operational action.

Staging adds time and duplicate evaluation cost. It buys earlier detection at limited exposure. The architect defends that cost by tying each stage to a specific failure that offline tests cannot adequately reveal.

Architecture / decision scenario

A healthcare scheduling provider prepares a prompt-and-harness release. Its offline policy requires 96% passes on a defined application acceptance suite, no regression-suite decline, and zero wrong-patient actions in a critical suite. Candidate scores 97% on acceptance, preserves the regression results, and has no critical failures, so product proposes immediate full deployment.

The architect instead follows the approved stages. Shadow execution uses isolated tools and hidden responses to inspect production inputs. A 5% canary then receives real scheduling requests, retaining the baseline as control. The service owner watches a seven-second p95 ceiling and a maximum 1% failed-booking rate by served bundle.

During peak load, candidate failures reach 4% while baseline remains at 0.4%. Fleet averages obscure the small arm. The operational owner restores the entire baseline mapping and verifies served-bundle telemetry; the 97% offline score cannot override this regression. A candidate booking already written remains an incident-reconciliation task, and the failed workflow becomes a regression case before another release attempt.

Common exam traps

• Treating a passing offline score as permission for immediate full exposure.
• Letting aggregate pass rates erase critical-case failures.
• Hiding candidate failures inside fleet-wide averages.
• Allowing shadow execution to perform real external actions.
• Restoring only one component of the validated baseline bundle.
• Assuming rollback reverses completed business actions.

What to remember

Bind release thresholds to an exact candidate and evaluation procedure. Advance exposure only when the stage’s quality and operational checks hold. Preserve an executable baseline, assign rollback authority, and verify what actually serves after restoration.

Knowledge check

References

• Anthropic, Demystifying evals for AI agents
• Anthropic, Define success criteria and build evaluations
• Anthropic, Create a Message
• Anthropic, Model IDs and versioning
• Anthropic, Optimizing for cost and intelligence
• Google, Canarying releases