Skip to lesson

AnthropicClaude Certified Architect – Professional2026-106.2 Communicate architectural decisions and trade-offs

State uncertainty before promising production outcomes

Completed by the knowledge check

Learning objective

Explain probabilistic behavior, evaluation limits, and residual risk to set realistic expectations without overpromising.

Core idea

A quality result describes what happened under a particular evaluation, not what every future request will do. Communicate three separate things: behavior can vary, the evaluation covers a bounded population and measurement method, and harmful outcomes can remain after controls. Stakeholders need all three to decide what use they can responsibly authorize.

Your artifact is a limitation statement attached to the deployment recommendation, with remaining harms recorded for an accountable owner. NIST AI RMF MAP 2.2 addresses knowledge limits and oversight; MEASURE 2.5 addresses documented limits on generalization beyond test conditions; MANAGE 1.4 addresses documented negative residual risks. Together they give precise names to what an optimistic accuracy headline leaves out.

How it works

Explain variability without implying chaos

Anthropic’s agent-evaluation guidance distinguishes a task from a trial: a task specifies the input and success criteria, while a trial is one attempt. It recommends multiple trials because model outputs vary between runs. In a stakeholder briefing, translate that into the operational consequence: a successful demonstration does not mean the same case will always succeed, and repeating an attempt can consume more time and money.

A sampling setting is not a reliability promise: temperature is deprecated, and where temperature: 0.0 is still accepted, Anthropic states that results are not fully deterministic (see 0.1).

Do not equate different wording with a business failure. Define the unacceptable variation: an omitted eligibility condition, a fabricated fact, or an action outside authority. Conversely, consistent wording does not establish factual correctness. Anthropic’s hallucination guidance says Claude can generate incorrect or context-inconsistent text, and that mitigations reduce rather than eliminate that risk.

Separate a variable model judgment from a deterministic enforcement rule. A service can enforce a spending limit on every attempted payment while the model still proposes the wrong recipient. The business statement should identify which property the code checks and which judgment remains fallible. See 5.1.1 for control placement; the task here is communicating its consequence accurately.

Qualify the evaluation claim

A useful claim names the task, population, configuration, scoring rule, number of cases, and attempts per case. Say which aggregation the number represents: a first-attempt rate, pass@k, or pass^k (see 4.3.3).

You do not need to teach the evaluation harness to an executive. You do need to say whether the reported success required selecting a successful attempt from several, and who or what performs that selection in production. In the briefing, put that selection step next to the success rate, with its reviewer workload, latency, and cost; stakeholders must understand the operating process behind the number.

Explain what the test does not cover: missing languages, rare categories, adversarial inputs, different document quality, or operational load outside the prototype. NIST MEASURE 1.1 calls for documenting risks that will not or cannot be measured. An untested slice is an unknown, not an assumed extension of the aggregate result.

Measurement can also be wrong. Anthropic’s evaluation guidance describes model-based graders as requiring calibration against human experts. If an automatic judge supplies the quality rate, disclose that dependency and the unresolved disagreements that materially affect acceptance. Do not turn a model’s assessment of another model into an independent guarantee. See 4.2.2 for grader calibration and expert adjudication.

Turn remaining risk into a decision

After describing controls, name the remaining harm, affected party, detection method, response owner, and scope of acceptance. For example: “An incorrect draft may pass the review rubric; the service owner accepts a limited staff-only pilot with sampled expert review, and can suspend it when that review finds a critical error.” The reviewer workload and suspension authority are part of the proposal.

Residual risk is what remains unmitigated, not only what has already occurred. The risk owner chooses the treatment under 5.2.5. The architect explains the remaining exposure; the organization’s authorized risk owner determines whether it is acceptable. A disclaimer added to the interface cannot substitute for that decision.

Keep the quality observation separate from a future service commitment. State a target as a target, describe the checks used to assess it, and identify the conditions for reducing scope or pausing operation. Subsection 6.3 owns the stakeholder commitment and SLA process; this lesson supplies the honest limitation statement it needs.

Key distinctions and trade-offs

Failure rate versus severity: A small aggregate failure rate can conceal a severe category. Show results for the category whose harm determines deployment scope, even when its case count is too small to support a confident estimate.

No observed failure versus no risk: A finite clean test does not demonstrate the absence of future errors. NIST MEASURE 2.5 calls for documenting generalization limits even when every evaluated case passes.

Abstention versus correctness: An answer withheld for review avoids one automated error path but creates work and delay. Include that workload when communicating expected benefit. Anthropic recommends permitting “I don’t know” as a hallucination mitigation; it does not describe abstention as eliminating hallucinations.

A detailed uncertainty explanation can overwhelm a sponsor. Keep the first page to the measured result, its most important limit, the remaining harm, and the requested scope. Preserve supporting details for reviewers rather than replacing the entire explanation with an unexplained confidence adjective.

Architecture / decision scenario

A government benefits office proposes a staff-only assistant that drafts explanations of application status. It has a five-person review team and a requirement that the system never issue binding eligibility decisions. In this hypothetical evaluation, 570 of 600 first-attempt drafts pass the rubric: 95%. In the appeal subset, only 32 of 40 pass: 80%. Staff must substantially revise failed drafts.

The program sponsor proposes announcing “95% accurate advice for all applicants” and launching a public self-service channel. That claim expands the tested audience and task, hides the weaker appeal performance, and removes the staff review that the deployment assumes. The first-attempt result also says nothing about whether every repeated trial succeeds.

Recommend a scoped staff pilot for routine status explanations, with appeals handled manually and no authority to determine eligibility. Present both rates and their denominators, identify untested languages, and state that factual errors can remain even after review. The routine subset passes 538 of 560 cases, about 96.1%. At an illustrative 1,000 routine drafts per day, that subset’s observed failure rate suggests about 39 substantial revisions; label this as a planning estimate conditional on comparable future traffic. Do not promise that the five reviewers can absorb it without measuring review time.

The program director owns scope and risk acceptance; the review lead owns capacity assessment. The limitation statement requests pilot approval conditional on that capacity, records incorrect explanations as a remaining harm, and names the director’s authority to pause use after a critical error. This creates an actionable expectation without converting a test result into an assurance of universal correctness.

Common exam traps

  • Promising identical answers by lowering temperature.
  • Treating a demonstration or several selected successes as a first-attempt reliability result.
  • Reporting aggregate accuracy without a material high-harm subset.
  • Describing hallucination mitigations as eliminating factual errors.
  • Calling an unevaluated population safe because no failures were recorded there.
  • Treating residual-risk disclosure as permission for the architect to accept every risk.
  • Omitting review effort when describing the value of abstention or escalation.

What to remember

State what was measured, under which conditions, and what remains unknown. Explain variable outcomes separately from enforcement controls, disclose material evaluation limits, and name the remaining harm and its owner. A realistic promise includes the operational cost of handling failure and the scope the organization actually authorizes.

Knowledge check

References