Skip to lesson

AnthropicClaude Certified Architect – Professional2026-104.3 Conduct A/B testing and iterative improvements

Isolate the change before comparing variants

Completed by the knowledge check

Learning objective

Compare prompt, model, and configuration variants offline on fixed evaluation sets, changing one variable at a time.

Core idea

A higher score is useful only if you know what changed. Compare a candidate with its baseline on the same cases, with the same grading rules and operating conditions. Change the prompt, the model, or a configuration setting in a comparison; changing all three measures a bundle whose improvement you cannot attribute.

Anthropic distinguishes the agent harness from the evaluation harness: one executes the agent, while the other runs tasks, captures transcripts, grades outcomes, and aggregates results. Version both. An apparent prompt improvement can instead come from a different tool fixture, a more permissive grader, or leftover state from another trial.

How it works

Specify the candidate precisely

Create a variant record before execution. It identifies the prompt revision, the Messages API model value, the request configuration, tool definitions, agent harness revision, dataset revision, and grader revision. These are fields in your team’s evaluation artifact, rather than a Claude API experiment object.

For a prompt comparison, change only the relevant instructions in system or messages, preserving the model and configuration. For a model comparison, preserve the prompt and the compatible configuration while changing model. For a configuration comparison, retain the prompt and model and vary one supported setting. For example, on Claude Sonnet 4.6 through the Claude API, compare top-level output_config.effort: "high" with output_config.effort: "medium". Top-level effort requires no beta header on supported models; this example does not use the separate per-message effort beta. Record explicit settings and relevant omissions; an omitted option is part of what you tested.

Compatibility matters. A model that cannot accept the baseline settings is not a clean substitution under that request contract. Evaluate a compatible common configuration, or label the candidate as a model-plus-configuration bundle and limit the causal claim. Do not silently remove a setting and report that only the model changed.

Use a pinned model ID. Anthropic documents that each model ID identifies a fixed snapshot, including the dateless IDs introduced with the 4.6 generation. Convenience aliases for some earlier models are pointers and do not carry the same guarantee. Do not infer mutability from whether a date appears in the string. Serving infrastructure can still change observable behavior under a fixed ID, so preserve execution dates as well as model identity.

Hold the comparison material constant

Run each variant against the same fixed evaluation set. The dataset and grader design belong to 4.2; this comparison consumes their versioned outputs. Preserve case identifiers so you can inspect which cases improved and which regressed, rather than comparing two unrelated aggregate scores.

Keep tool responses and initial environment state controlled where the task permits. Anthropic recommends starting each trial from a clean environment: files, cached data, and resource exhaustion can create correlated failures or artificial advantages. If one candidate inherits a successful prior agent’s workspace, the experiment measures that inheritance alongside the candidate.

Interleave or randomize execution order within practical constraints. NIST’s experimental-design guidance distinguishes randomized runs from executing every replication of one treatment before the next. If baseline runs in a quiet environment and candidate runs during a capacity incident, configuration and time are confounded. Preserve run order and infrastructure incidents in the result record.

A fixed dataset does not require pretending live dependencies are constant. If a task genuinely depends on changing external data, record the dependency version or captured response. Where you cannot control it, state the resulting limitation rather than claiming exact reproduction.

Compare outcomes, resources, and regressions

Predeclare what wins: for example, higher rubric pass rate subject to unchanged critical-failure limits and an acceptable latency budget. Anthropic’s success-criteria guidance calls for specific, measurable, relevant targets. Translate those targets into case-level grades and resource summaries before looking at candidate outputs.

Use a paired result ledger. Each case has baseline and candidate outcomes, trial identifiers, grades, and operational measurements. Review baseline-pass/candidate-fail cases explicitly. A gain concentrated in easy tasks can hide losses in the high-consequence workflow that justified the system.

Run repeated trials under a fixed repetition policy because model outputs vary. The statistical interpretation belongs to 4.3.3; here the experimental control is that both variants receive the same trial allocation. Do not give the candidate extra attempts until it passes while retaining only one baseline attempt.

After identifying a promising prompt, freeze that result before comparing models. Then test the combined release candidate against the release baseline. Sequential improvements need this final comparison because prompt and model behavior can interact. One-variable comparisons identify local contributions; they do not establish that every combination preserves them.

Key distinctions and trade-offs

Controlled comparison versus broad search: Exploring many candidates helps discover possibilities. Confirming an improvement requires a frozen candidate and evaluation procedure. Reusing the same scores to tune and certify a prompt encourages adaptation to the evaluation set.

Snapshot identity versus repeatable output: A pinned model preserves model identity. It does not make an individual generated answer deterministic or freeze every part of the serving environment.

One-variable comparison versus interaction study: Holding other factors constant makes attribution clear. It can miss interactions; investigate those with an explicitly designed follow-up rather than presenting a bundled change as a single-factor result.

Offline versus online evidence: Offline execution avoids exposing users, but consumes evaluation budget and measures the tasks represented in the set. User behavior requires the online design in 4.3.2. A candidate can be eligible for that experiment without being approved for general release.

Architecture / decision scenario

A financial-services team revises a document-summary prompt and proposes changing the model in the same release to reduce cost. Its internal review policy permits at most one critical omission in a fixed 500-case evaluation, and its service budget requires p95 completion below six seconds. These are scenario thresholds, not Anthropic limits.

The architect rejects a single bundled comparison because a failure would leave the team unable to identify the cause. The evaluation owner first compares prompt revisions with model, tool fixtures, and configuration held constant. The candidate raises ordinary rubric passes from 440 to 465 while retaining one critical omission and a 5.8-second p95.

Next, the team changes only model under a compatible common request configuration. That candidate reaches 468 passes but causes four critical omissions. Its lower measured cost cannot override the omission threshold. The architect records the model result as rejected. A separate configuration comparison then holds the retained prompt and baseline model, Claude Sonnet 4.6, constant while lowering output_config.effort from "high" to "medium". It preserves the omission threshold and reaches a 5.1-second p95. The owner confirms that exact prompt-and-effort combination against the release baseline before submitting it to 4.3.4.

Common exam traps

  • Changing prompt, model, and settings together, then crediting the prompt for the entire gain.
  • Comparing candidates on different cases or grader revisions.
  • Treating a dateless model ID as necessarily an evergreen alias.
  • Giving one candidate more trials or a previously prepared workspace.
  • Accepting an aggregate improvement that violates a critical-case threshold.

What to remember

Freeze cases, graders, and execution conditions; name the exact changed factor. Preserve model, prompt, configuration, and harness provenance in the comparison record. Repeat trials fairly, inspect regressions, and test the final combination before treating separate improvements as a release.

Knowledge check

References