AnthropicClaude Certified Architect – Professional2026-104.3 Conduct A/B testing and iterative improvements
Assign comparable traffic before measuring lift
Completed by the knowledge check
Learning objective
Design online A/B tests with hypotheses, randomization, success and guardrail metrics, and controls for selection and temporal confounding.
Core idea
An online A/B test compares deployed variants on randomly assigned, concurrent traffic. It asks whether a change improves actual user outcomes, rather than whether a grader prefers its outputs. Anthropic describes this as a complement to offline evaluation: online tests measure outcomes such as task completion and retention but require enough traffic and can take days or weeks.
The architect owns the assignment mechanism and the decision plan. A dashboard showing two averages does not establish a causal comparison if different populations, time periods, or measurement rules produced them.
How it works
Write the experiment plan first
Start with a falsifiable hypothesis: a specific change should improve a defined outcome for an eligible population. For example, revising clarification instructions should increase completed returns among customers who start the returns assistant. Keep the changed prompt and its baseline immutable, following 4.3.1.
The experiment plan identifies the owner, population, exclusions, assignment unit, traffic allocation, primary metric, guardrails, sample requirement, duration, and stopping rule. It also names the baseline and candidate bundles, including the Messages API model value and prompt revision. The application implements this plan; these experiment labels are team-defined artifacts.
Choose exclusions before assignment. If a language is unsupported by the product, exclude it consistently from both arms. If candidate execution fails, retain that assigned user’s outcome in the analysis rather than excluding the failure as unusable data. Otherwise the apparently successful candidate can win by losing its difficult cases.
Randomize where the experience stays coherent
NIST defines completely randomized designs by random assignment of factor levels to experimental units. Apply that principle to the unit that can receive one coherent product experience: a user, account, session, or independent task. Request-level assignment is appropriate only where requests are independent and switching cannot contaminate the outcome.
For a multi-turn assistant, use stable assignment across the conversation. If the experience changes halfway through, neither arm represents the conversation that produced the result. When colleagues share habits or workflows, assign the team or account instead of individual requests. The analysis must respect that grouping; thousands of messages from a few teams do not create thousands of independent experimental units.
One implementation is a tested assignment function that hashes an opaque account identifier with an experiment-specific salt and maps the result to configured buckets. Persist the assignment or make it reproducible. Record the experiment identifier, assigned arm, actual served bundle, and exposure timestamp in the application event stream. None of those names is a Claude API routing parameter.
Check the assignment path before interpreting outcomes. Compare eligible and assigned counts, planned and observed allocation, and assignment versus actual exposure. A fallback that sends candidate-assigned users to baseline needs explicit accounting; silently relabeling them after seeing the outcome defeats randomization.
The application event stream records the assigned arm and the actual served bundle, so a fallback that sends candidate-assigned users to baseline gets explicit accounting.
Separate success from vetoes
Choose one primary outcome that expresses the hypothesis. Anthropic’s success-criteria guidance includes implicit user feedback such as task completion. Define its denominator and observation window: completed eligible returns within 24 hours of assignment divided by all assigned eligible starters, for example. The same event definitions and maturation window apply to both arms.
Guardrail metrics constrain acceptable harm. A higher completion rate cannot justify exceeding the agreed limits for incorrect refunds, escalation failures, cost per completed return, or p95 end-to-end latency. Record thresholds and response owners before exposure. Product owns the benefit target; operations and the relevant risk owner approve the veto thresholds.
Log each guardrail by arm. Google’s canarying guidance shows why an overall service metric can hide a severe regression in a small candidate population. Aggregate health can look stable while candidate users experience failures.
Separate stopping for harm from claiming benefit. The operational owner can stop an unsafe arm immediately under its guardrail rule. Declaring success follows the planned inference procedure in 4.3.3; a favorable morning is not an independent release criterion.
Control who enters and when
Volunteer-only candidate users create selection bias: they can differ in skill, patience, or motivation from baseline users. Randomly assign the same eligible population. If a limited beta is necessary, describe the conclusion as applying to that beta population rather than the entire service.
Run baseline and candidate concurrently. Comparing candidate during a promotional weekend with baseline during an ordinary week confounds the change with demand, inventory, staffing, and customer mix. Concurrent assignment lets both arms experience shared time effects. It does not repair arm-specific incidents or unequal eligibility rules; examine those separately.
NIST’s randomized-block guidance treats time of day and operator differences as nuisance factors. Where a few known factors dominate the outcome, randomize within declared blocks such as region or account-size band and preserve those factors in analysis. Avoid inventing favorable segments after results arrive.
Cover the business cycles the plan requires and allow outcomes to mature. A novelty effect can produce an early apparent benefit that fades. Record effect estimates by date segment as a diagnostic, but do not let a transient early response redefine the stopping rule.
Key distinctions and trade-offs
Concurrent control versus historical benchmark: Historical data helps estimate baseline rates and plan sample sizes. A concurrent randomized control supports the experiment’s comparison under shared conditions.
Assignment versus exposure: Assignment defines the intended comparison. Exposure records what actually ran. Both are necessary to diagnose routing failures without selectively removing unfavorable outcomes.
User outcome versus output grade: A grader can detect unsupported claims. Only a product outcome records whether a customer completed the return. Use the grader’s verdict as a guardrail, not as the outcome.
Larger candidate allocations accumulate information faster but expose more users. A small canary limits exposure; it may not supply enough independent observations to establish the intended benefit. Stage the experiment according to 4.3.4 rather than weakening the benefit criterion.
Architecture / decision scenario
A retailer wants to raise returns completion from a 60% baseline by at least three percentage points. Its proposed prompt has passed offline checks. Product suggests showing it only to loyalty members for one weekend and comparing them with last month’s customers.
The architect rejects that design: loyalty status selects a different population, and the weekend promotion changes traffic. The experiment owner instead assigns eligible customer accounts concurrently, 50/50, with assignment stable across sessions. Completion is measured within 24 hours, including candidate failures in the assigned denominator.
The plan declares region blocks, the sample and duration calculation, a cost-per-completed-return ceiling 10% above baseline, and an eight-second p95 ceiling. The operations owner can stop the arm on a guardrail breach. Product can approve benefit only after the planned analysis; a 64% candidate completion rate among volunteers would answer a different question.
Common exam traps
- Using opt-in candidate users as though randomization selected them.
- Switching prompts between turns of the same conversation.
- Comparing separate months instead of concurrent traffic.
- Removing failed candidate executions from its denominator.
- Choosing the primary metric or favorable segment after inspecting results.
What to remember
Predeclare a hypothesis, eligible population, stable assignment unit, primary metric, guardrails, and stopping plan. Randomize comparable users concurrently and preserve assignment as well as exposure. Control known nuisance factors and time effects before interpreting lift.
Knowledge check
A B2B procurement assistant holds multi-turn conversations, and users in the same customer account share templates and hand work to each other, so one user’s experience shapes their colleagues’ outcomes. The team will A/B test a revised prompt on task completion. Which assignment unit fits?
In a returns-assistant A/B test, the candidate times out for 6% of the customers assigned to it. None of those customers completes a return within the planned observation window. The dashboard drops them as unusable data and shows the candidate ahead on completion. How should the analysis treat them?
Product wants to know whether a revised prompt raises booking completion for all eligible travelers. It proposes enabling the prompt for travelers who opt in during a holiday-sale week and comparing them with last month’s travelers. Which design answers the question?
During a capacity incident, a fallback served the baseline bundle to 8% of the users assigned to the candidate. The event stream records both the assigned arm and the served bundle. An analyst proposes moving those users into the baseline arm. What should the experiment owner do?
References
- Anthropic, Demystifying evals for AI agents
- Anthropic, Define success criteria and build evaluations
- Anthropic, Create a Message
- NIST, Completely randomized designs
- NIST, Randomized block designs
- Google, Canarying releases