AnthropicClaude Certified Architect – Professional2026-104.1 Define evaluation metrics (accuracy, latency, cost, safety, security)
Assign business targets to metric owners
Completed by the knowledge check
Learning objective
Derive metrics, segment-level acceptance thresholds, and metric owners from business success criteria.
Core idea
Segmented accuracy identifies where a system struggles. The architectural decision is what those results authorize: which work can proceed, under which thresholds, and who can defend the residual risk.
Start with the business outcome, then define the measures that could disprove success. Anthropic’s evaluation guidance calls for specific, measurable, achievable, and relevant criteria, and says most applications need several dimensions. An architect turns those criteria into a versioned scorecard whose definitions and owners remain stable when a candidate performs poorly.
How it works
Translate an outcome into a decision
“Reduce handling time” leaves too much undefined. Reducing time by omitting difficult work can improve that number while damaging the service. State the intended outcome together with what must remain protected: reduce staff handling time for eligible cases while preserving decision quality and keeping review demand within staffed capacity.
For each outcome, identify a corresponding failure. Staff time requires a handling-time measure, decision quality requires a task-specific correctness measure, and review capacity requires a count or rate of escalations. The resulting measures answer different questions; none can stand in for the others. Anthropic’s develop-tests guidance explicitly includes task fidelity, privacy preservation, latency, and price among success criteria.
Use a metric specification as the team’s decision artifact. This is an application-owned document, not a Claude API resource. Give each entry the business outcome, unit of evaluation, numerator and denominator, relevant segments, scoring method, threshold, measurement window, and owner. Include the baseline that makes the threshold defensible. A target copied from another application lacks that connection.
Separate indicators, targets, and commitments
Google’s SRE guidance distinguishes a service level indicator (SLI), a service level objective (SLO), and a service level agreement (SLA). An SLI is a defined quantitative measure. An SLO is its target or acceptable range. An SLA includes consequences for failing an agreed service objective. Calling every internal target an SLA misstates the commitment.
For example, “the proportion of eligible applications with a usable recommendation within 30 seconds” is an indicator. “At least 98% over the evaluation window” is an illustrative target. Neither sentence establishes contractual remedies. The service owner must identify whether a customer agreement imposes a separate obligation before using the target in a release decision.
Keep offline acceptance and operating commitments distinguishable. Passing a fixed evaluation set permits a bounded next step; it does not prove that production meets an SLO under its actual workload. Both can share a metric definition, but their populations and observation windows differ.
Derive segment thresholds from error consequences
Do not assign different thresholds merely because one segment is harder. Ask what an error does there. A wrong product recommendation and an incorrect denial of an entitlement have different consequences even if their measured error rates match. Define the segment by the business decision it supports, then identify the costly error and required protection.
Use baseline performance, operational capacity, and risk tolerance to negotiate a feasible threshold. Anthropic’s achievable criterion explicitly points to prior experiments, benchmarks, research, or expert knowledge. Achievable does not mean automatically acceptable: if the measured capability cannot meet the business requirement, the answer can be continued review or narrower scope.
Distinguish a mandatory acceptance threshold from a ranking preference. Anthropic names three task-scoring modes: weighted, where combined grader scores must reach a threshold; binary, where all graders must pass; and hybrid, which combines the two. Apply that distinction to the scorecard: a required privacy condition cannot be offset by better prose, while handling time or cost can rank candidates that satisfy mandatory conditions. A mandatory threshold can be an absolute bar or a no-worse-than-production regression bar; see 4.2.3 and 6.5.3 for the suites. Record the combination rule before comparing candidates, so a favorable result cannot quietly redefine what success means.
Assign definition, measurement, and action ownership
Name the person responsible for the business meaning of each metric. A domain lead decides what counts as a correct recommendation; an engineering lead owns the comparator and its reproducibility. The service owner decides what action a failed target requires. One person may hold multiple roles, but an unlabeled shared dashboard does not allocate responsibility.
Write the scorecard so that a disagreement has an owner who resolves it. If reviewers disagree on a borderline case, the domain owner settles the rubric before that case becomes an acceptance label. If instrumentation drops outcomes, the measurement owner repairs the denominator rather than marking missing cases successful. If a mandatory threshold fails, the release owner applies the recorded hold or scope-reduction decision.
This ownership model is an architectural application of measurable success criteria, not a vendor-prescribed organization chart. Its purpose is to prevent a threshold from existing without anyone authorized to interpret or act on it. Metric definitions for quality, operations, and safety belong to 4.1.2–4.1.4.
Key distinctions and trade-offs
A business outcome explains why the system exists. A model-quality metric measures a contributing capability. An operational metric measures delivery. Improving a contributing metric is useful only when the business outcome and protected conditions still hold.
A production-weighted aggregate supports capacity and financial planning. A segment threshold determines whether a particular class of work qualifies. Both are useful, but their authority differs. Scope can expand when an additional segment meets its own requirements; one strong segment does not lower another’s required bar.
More mandatory thresholds reduce room for optimization and increase evaluation cost. Too few permit a visible improvement to hide an unacceptable failure. Keep a threshold when its owner can explain the decision that changes when it fails. Keep a ranking measure when it distinguishes acceptable candidates. Remove decorative measures that change neither decision.
Architecture / decision scenario
A financial-services team uses Claude to prepare application-review recommendations. The sponsor wants analyst time reduced from 12 to 8 minutes per case. The service has capacity to review 300 flagged cases a day. These are illustrative business constraints, not Claude limits.
The product lead proposes releasing a candidate with 99% aggregate correctness. The architect’s scorecard also requires at least 99.5% correctness for cases involving adverse recommendations and no more than 250 daily escalations, leaving capacity for unexpected cases. Ordinary recommendations require 98.5%. The domain owner approved those thresholds from error consequences and the existing review process before the evaluation.
Candidate A reaches 99% overall and 7 minutes of analyst time, but scores 97% on adverse recommendations and projects 380 escalations daily. Candidate B reaches 98.8% overall, 99.6% on adverse recommendations, and 8 minutes, with 220 escalations. The tempting aggregate winner is A; its rare-segment failures and review demand violate the funded operating model.
The domain lead owns correctness labels, the platform lead owns scoring and workload projections, and the service owner authorizes B’s bounded pilot. The sponsor still checks whether actual analyst time falls without harming decisions. A’s faster headline does not justify changing the thresholds after seeing the results.
Common exam traps
- Choosing thresholds after evaluating candidates.
- Treating an internal SLO as a contractual SLA.
- Lowering a required threshold because a segment is difficult.
- Letting a weighted score compensate for a mandatory safety or quality failure.
- Assigning a dashboard owner without assigning the business definition or response decision.
- Claiming business success from a model score alone.
What to remember
Translate the funded outcome into measurable conditions before evaluating candidates. State which conditions authorize release, which rank acceptable candidates, and which apply separately to important segments. Give each definition, measurement, and response an owner who can defend the decision.
Knowledge check
A lending team uses Claude to draft application-review recommendations. Before any evaluation, the domain owner approved two mandatory thresholds from error consequences: at least 99.5% correctness on adverse-recommendation cases and at least 98.5% on ordinary cases. Candidate X scores 99.1% on the production-weighted aggregate and 97.2% on adverse cases. Candidate Y scores 98.7% overall, 99.6% on adverse cases and 98.6% on ordinary cases. Both meet every other scorecard condition. Which decision does the scorecard support?
A claims service’s metric specification reads: “Indicator: the proportion of eligible claims receiving a usable recommendation within 30 seconds. Target: at least 98% of production claims in each 28-day window.” No customer agreement attaches any consequence to missing the target. What should the release report call the target?
An insurer’s triage scorecard names a domain lead for the meaning of a correct recommendation, an engineering lead for the scoring comparator, and a service owner for the response to a failed target. Two reviewers disagree on whether a borderline recommendation is correct. Who should settle it before the case becomes an acceptance label?