AnthropicClaude Certified Architect – Professional2026-104.3 Conduct A/B testing and iterative improvements
Read uncertain results before declaring improvement
Completed by the knowledge check
Learning objective
Interpret results with sample adequacy, significance testing, and repeated runs that account for nondeterministic outputs.
Core idea
A measured improvement has two uncertainties: which tasks or users you sampled, and how the agent behaves when it attempts them again. More repetitions on one task reveal its execution variability; more representative tasks reveal broader performance. Neither substitutes for the other.
Anthropic names each attempt at an evaluation task a trial and recommends multiple trials because outputs vary. Your decision record must therefore state the task count, trial count, aggregation rule, and uncertainty, rather than reporting one favorable pass rate as a property of the model.
How it works
Decide whether the sample can answer the question
Sample adequacy depends on the effect you need to detect and the observations your design actually makes independent. Before execution, record the baseline rate, minimum detectable effect, significance level, target power, allocation, and analysis unit. NIST’s sample-size guidance relates required observations to baseline proportion, effect size, significance, and power; its one-population formula is not a drop-in calculator for every online experiment.
A significance level such as alpha = 0.05 controls the planned test’s false-positive probability under its assumptions. Power is the probability of detecting the specified effect when it exists. These are design choices, not universal certification thresholds. A small sample with low power can miss a useful improvement.
Count independent units honestly. If assignment occurs by clinic, repeated messages from one clinic share its workflow and assignment. Analyze clinic-level summaries or use a statistical procedure that accounts for that grouping. Do not insert the message count into an independent two-sample formula. The assignment design is covered in 4.3.2; its consequences carry into inference.
Sample coverage matters alongside size. Ten thousand easy cases cannot establish safety on a rare, high-consequence case type absent from the set. Report the sample represented by each result, and keep critical-case evaluation separate from an overall rate. A result can be precise for the wrong population.
Match the test to the observations
Write the null hypothesis and test direction before inspecting results. For an independent online binary outcome, the null might be equal completion proportions between baseline and candidate. NIST documents a normal-approximation test for sufficiently large independent samples and Fisher’s exact test for small independent samples. Neither automatically handles paired offline cases or clustered online assignments.
For offline comparisons, preserve the pairing: each task receives both variants. Select a paired analysis with the evaluation statistician rather than treating those outputs as unrelated groups. Repeated trials nested within each task need the same care. Their shared task difficulty can make an independent-trial analysis overconfident.
A p-value is the probability, assuming the null hypothesis and the test’s assumptions, of a statistic at least as extreme as observed. It is not the probability the candidate is worse, nor the probability that the null is true. Predeclare the rejection threshold; do not choose a looser threshold after seeing an inconvenient result.
Report an effect estimate and a confidence interval with the test result. A 95% confidence procedure produces intervals containing the true parameter in about 95% of repeated samples under its assumptions. It does not assign a 95% probability to a fixed parameter lying within one particular reported interval.
Practical importance requires a separate rule. A statistically detectable improvement can be too small to justify operational cost. Conversely, an attractive point estimate with a wide interval can leave both harm and benefit plausible. Failure to reject equality does not demonstrate equivalence or non-inferiority; those require a prespecified margin and an analysis designed for that claim.
Measure output variability without rewarding retries
Allocate the same number of trials to both offline variants, reset the environment between trials, and retain all outcomes. Anthropic warns that shared files, caches, or resource exhaustion can create correlated failures and artificial gains. Repeating a contaminated trial setup does not fix it.
Choose an aggregation rule matching the service. Anthropic distinguishes pass@k, the likelihood of at least one success in k attempts, from pass^k, the likelihood that all k trials succeed. A single-answer service cannot claim the success rate of selecting the best of several attempts unless production actually performs and validates that selection.
For illustration, if independent attempts each succeed with probability 0.8, three attempts have an at-least-one-success probability of 1 - 0.2^3 = 0.992; all three succeed with probability 0.8^3 = 0.512. This arithmetic assumes independence and the stated per-attempt probability. It shows how a flattering best-of-three score can coexist with inconsistent service.
Record per-task success frequencies and the distribution across repeated evaluation runs. Do not select the best run or discard failures after inspecting the candidate’s result. When candidate and baseline fluctuate similarly, the measured difference needs enough coverage and repetition to distinguish improvement from execution noise.
Preserve the planned stopping rule
A fixed-horizon test has a planned analysis point. Repeatedly testing and stopping at the first favorable p-value changes that procedure: the overall significance level is no longer the planned one, and false-positive risk can increase. Either retain the planned endpoint or have the statistical owner specify a sequential procedure before launch. Operational stopping for a safety breach remains necessary and is a different decision.
Record unplanned subgroup findings as hypotheses for confirmation. Searching many slices and reporting only the favorable one creates another selection problem. Repeated runs should estimate variability under a fixed procedure, rather than supply repeated opportunities to declare victory.
Key distinctions and trade-offs
Insufficient information versus no improvement: A wide interval means the experiment may not distinguish useful benefit from harm. Preserve an inconclusive outcome instead of forcing a winner.
Tasks versus trials: More tasks expand coverage; more trials estimate consistency on those tasks. Both cost inference and grading resources, and their balance follows the source of uncertainty.
Significance versus release approval: Statistical inference describes the measured difference. Release approval also requires practical benefit and intact guardrails, as 4.3.4 explains.
Architecture / decision scenario
An education provider tests a tutoring prompt. The evaluation owner runs 300 fixed tasks with five trials per variant. Candidate performance improves on average, but a stakeholder proposes citing its best run and treating all 1,500 outputs as independent cases.
The architect requires task-aware analysis and all five trials. The prespecified paired analysis reports a two-point gain with an illustrative 95% interval from minus one to plus five points. The business required at least a three-point improvement to justify migration. That interval supports neither a confident win nor a conclusion that both prompts are equivalent.
The owner retains baseline, investigates which task types fluctuate, and plans a confirmatory evaluation with additional representative tasks and a fixed repetition policy. Repeating the same 300 tasks until one run crosses the significance threshold would answer the stakeholder’s preference, not the production question.
Common exam traps
- Counting repeated outputs as independent tasks or users.
- Treating p below 0.05 as the probability the candidate is better.
- Calling a nonsignificant result proof of equivalence.
- Reporting
pass@kfor a service that exposes one attempt. - Selecting the best repeated run or stopping at the first favorable test.
What to remember
Plan sample size for the intended effect and analysis unit. Interpret significance alongside effect size, uncertainty, and practical value. Repeat trials to measure nondeterminism, preserve task structure, and distinguish at-least-one success from consistent success.
Knowledge check
A benefits-claims agent answers each customer request once, with no retry or selection step, and customers expect the same correct handling every time. Offline evaluation runs five trials per task. Which statistic best describes the behavior the service promises?
A clinic-scheduling prompt is A/B tested with assignment by clinic: 40 clinics and about 60,000 messages. The analyst runs a two-proportion test that treats every message as an independent observation and reports p = 0.001. What should the evaluation owner require?
A team will move to a cheaper prompt only if completion is no more than one point worse than baseline. The planned test finds a difference of −0.5 points, p = 0.40, with a 95% confidence interval from −3 to +2 points. The lead says that “no significant difference” justifies the move. What follows?
A fixed-horizon A/B test was planned to run 21 days. The product owner checks the p-value daily and, on day 6, sees p = 0.03 against the planned alpha of 0.05, with no guardrail breach. They want to declare the candidate the winner. What should the evaluation owner do?