AnthropicClaude Certified Architect – Professional2026-106.3 Manage stakeholder feedback loops and expectation alignment (including SLAs)
Require pilot value before funding scale
Completed by the knowledge check
Learning objective
Run pilots with exit criteria that validate value before scaling.
Core idea
A pilot earns a scaling decision when it demonstrates useful outcomes under the conditions the expanded service must face. A successful demonstration, enthusiastic users, or accurate model outputs alone cannot establish that the service creates enough value to operate.
NIST AI RMF MANAGE 1.1 addresses deciding whether an AI system achieves its intended purpose and whether deployment should proceed. MEASURE 2.3 addresses measurement under conditions resembling deployment, and MEASURE 2.5 addresses documented limits on generalization. Turn those principles into a pilot charter, predefined exit criteria, and a signed scale, revise, or stop decision. This adds delivery ownership to the proof-of-concept hypothesis in 1.1.4.
How it works
Freeze the decision before collecting results
The pilot charter names the workflow, participating population, exclusions, baseline, duration, spending limit, risk owner, and decision date. State the business hypothesis as a change in completed work, not a change in model activity. For example, reducing total handling time for accepted returns cases is a value hypothesis; generating more drafts is an activity measure.
For a Claude API pilot, use a dedicated workspace. In the Claude Console, select it on Workspaces, then use Spend limits to cap monthly API spending and configure threshold alerts. On Rate limits, select the workspace and use Edit beside each model to cap requests per minute, input tokens per minute, and output tokens per minute. These caps bound pilot demand within shared organization limits; they do not reserve production capacity. The monthly API cap does not cover reviewer labor or support time, which remain in the cost ledger in 1.6.2.
Record each exit criterion with its metric, threshold, observation window, data source, and accepting owner. Anthropic recommends success criteria that are specific, measurable, achievable, and relevant, and recognizes that most applications need several criteria. Use that guidance to distinguish required constraints from benefits that can trade against one another.
A pilot can therefore require an improvement in handling time while also imposing a maximum consequential-error rate, a cost ceiling, and a support-capacity limit. Do not average a failed safety constraint into a favorable overall score. If a criterion cannot yet be measured, record that as unresolved rather than treating the absence of a failure report as a pass.
Measure the full operating path
Use a comparable baseline that includes the work the service changes: preparation, review, correction, escalation, and completion. The workflow-cost method belongs to 1.6.2. The pilot decision consumes that ledger, including reviewer labor and support demand, rather than comparing only inference expense with an employee’s salary.
Include ordinary users and difficult cases in the intended deployment population. Anthropic’s evaluation guidance calls for task-specific tests reflecting the real distribution and its edge cases. NIST MEASURE 2.1 calls for documentation of test sets, metrics, and evaluation tools. Keep a pilot measurement plan recording which cases entered, which were excluded, and why; otherwise selective enrollment can make an impractical service appear valuable.
Separate adoption from success conditional on use. Record eligible cases, attempted cases, accepted completions, rejected outputs, and manual fallbacks. Low usage can mean that the workflow is inconvenient, even if the cases that reach the model score well. A pilot that lets participants quietly return to the manual process must count the time and outcome of that return.
Compare outcome rates across consequential case classes, not only in aggregate. A faster common path can conceal a slower exception path that consumes scarce specialists. Document sample limitations rather than treating a small pilot with no observed severe errors as proof that severe errors cannot occur.
Control iteration and stop conditions
Keep the evaluated configuration identifiable in the pilot record: application release, model selection, prompt revision, retrieval configuration, and operating instructions. When a substantive change occurs, distinguish the results before and after it. Do not combine several configurations into one apparent success rate without explaining what is being accepted.
Use the feedback cadence in 6.3.1 to resolve issues during the pilot. Preserve an evaluation set for the exit decision rather than tuning only against cases the team repeatedly inspected. A configuration improvement can justify another measured period; it does not justify moving the acceptance threshold after seeing a poor result.
Define a pause or shutdown trigger and who can invoke it. NIST MANAGE 2.4 addresses assigned responsibility for superseding, disengaging, or deactivating systems that behave inconsistently with intended use. The pilot charter implements that responsibility with an owned stop procedure, a manual continuation plan, and a record of affected work. These are delivery artifacts, not an automatic Claude API pilot-control feature.
Decide within the demonstrated boundary
At the exit review, the accountable sponsor chooses scale, revise, or stop from the agreed criteria. Domain and risk owners retain responsibility for accepting their constraints; a sponsor’s desire to expand does not establish that an unresolved risk has been treated.
A scale decision names the population and operating conditions actually supported, the next expansion limit, its monitoring owner, and the rollback trigger. A revise decision names the failed criterion, proposed correction, remaining budget, and one decision date. A stop decision preserves lessons and returns work to the supported process. Repeated extensions without a bounded hypothesis turn a pilot into an unowned production service.
A revise decision loops back to another measured period against the same criteria.
Key distinctions and trade-offs
Proof of capability versus proof of value: A model can complete a task while the surrounding workflow costs more than the manual baseline. The pilot must measure accepted outcomes and the work required to obtain them.
Learning versus acceptance: Iteration helps discover a viable design. Exit criteria protect the funding decision from being rewritten to fit whichever result emerged. Version changes and retest periods make that distinction visible.
Narrow success versus broad rollout: A constrained population can be the right pilot scope. Its success supports expansion within documented conditions, not an assumption that other languages, user groups, or risk classes behave the same way.
Architecture / decision scenario
A retailer pilots Claude-assisted return-case drafting for four weeks with 20 service agents. The sponsor wants to expand to all 300 agents if drafts are well rated. Before enrollment, the operations owner agrees illustrative exit criteria separately for each case class, standard returns and warranty exceptions: reduce mean handling time from eight minutes to at most six, keep total variable cost below $2 per accepted case, observe no unauthorized refunds, and keep specialist escalation at or below 5% of attempted cases. A $12,000 total pilot cap bounds the decision; a separate monthly workspace Spend limits cap controls its API-spend allocation.
For standard returns, 800 observed cases average 5.5 minutes, cost $1.70 each, and escalate at 3%. For 200 warranty exceptions, handling averages nine minutes, cost reaches $3.10, and escalation is 14%. The blended results are 6.2 minutes, $1.98 per case, and 5.2% escalation: the passing aggregate cost hides the failing warranty segment. No unauthorized refunds are observed. A draft-satisfaction survey is favorable, but the exception segment fails the agreed value and capacity criteria. The observation also cannot establish a zero future refund-error rate.
The tempting alternative is to expand every case type because the main cohort is happy. The architect recommends a bounded expansion for standard returns, keeping warranty exceptions on the manual path. The sponsor funds that scope; operations owns escalation capacity, and the risk owner accepts the documented residual uncertainty. A separate, budgeted revision tests the exception workflow before another decision. Scaling follows demonstrated value rather than hiding a failing segment inside a popular pilot.
Common exam traps
- Treating an impressive demonstration or favorable survey as sufficient proof of business value.
- Defining exit criteria after seeing the results.
- Omitting review labor, abandoned cases, or manual fallback from the cost and outcome record.
- Generalizing a narrow cohort’s success to untested operating conditions.
- Extending a pilot indefinitely without a budget, decision date, or stop owner.
What to remember
Predefine the pilot’s population, baseline, value measures, constraints, budget, and accepting owners. Measure the full workflow, preserve failed segments and uncertainty, and make scale, revise, or stop an explicit decision within the conditions actually tested.
Knowledge check
A telecom pilot set exit criteria separately for each case class before enrollment. Billing disputes meet every criterion. Contract cancellations, a fifth of the volume, miss the handling-time and escalation thresholds. The blended results meet every threshold, and the sponsor wants to scale all case types to 400 agents. What should the architect recommend?
A pilot’s exit criteria include a ceiling on total variable cost per accepted case. The sponsor argues that this criterion passed because API spending stayed under the dedicated workspace’s monthly spend limit throughout the pilot. Each draft is reviewed and corrected by a specialist before it is sent. How should the cost criterion be assessed?
In week three of a six-week pilot, the team revises the prompt after early results fall short of an agreed accuracy threshold. The revised prompt performs better on the cases the team inspected while making the change. The sponsor asks for one success rate for the whole pilot at the exit review. How should the results be reported?
In a pilot, agents may handle any eligible case manually. Agents attempt the assistant on 35% of eligible cases, and those attempts score well against the exit criteria. Agents quietly handle the rest the old way. What should the pilot record before the exit review?