AnthropicClaude Certified Architect – Professional2026-104.1 Define evaluation metrics (accuracy, latency, cost, safety, security)
Measure useful answers and justified abstentions
Completed by the knowledge check
Learning objective
Define task-quality metrics: accuracy, precision and recall, groundedness, calibrated abstention, format compliance, and task completion.
Core idea
Field correctness and semantic validation establish whether individual outputs match their intended meaning. Professional evaluation adds a broader question: does the system complete the right work, with supported claims and an appropriate decision to answer or abstain?
Define a small family of complementary measures. Anthropic’s evaluation guidance names accuracy, precision, recall, and task completion among measurement options. Its hallucination guidance recommends allowing uncertainty and checking claims against sources. Your scorecard must prevent success on one measure from hiding a failure on another.
How it works
Define the decision unit and its errors
Accuracy is correct decisions divided by evaluated decisions under a declared rubric. Define whether a decision is a classification, an answer, or an entire task. Do not silently compare field accuracy with whole-task accuracy. The unit determines what the resulting percentage authorizes.
For a positive class, define true positives (TP), false positives (FP), and false negatives (FN). Precision is TP / (TP + FP); recall is TP / (TP + FN). Precision asks whether flagged cases are real; recall asks how many real cases were found. Report them per relevant class when error consequences differ. If a denominator is zero, report the measure as undefined with its counts rather than inventing a perfect score.
High aggregate accuracy can hide missed rare exceptions; use the business requirement from 4.1.1 to decide whether missed exceptions or false alarms constrain release. These are task-classification measures; retrieval ranking measures belong to 3.5.5.
Measure source support separately from coverage
Define groundedness as supported factual claims divided by evaluated factual claims. The rubric must say what counts as a claim and what source support is sufficient. Split a sentence containing several independently testable facts when one supported fact could otherwise conceal an unsupported one.
Anthropic recommends direct quotations and verification with citations to make factual claims auditable. A matching quotation is useful evidence, but matching text alone does not establish that it supports the answer’s conclusion. A statement can quote a valid policy passage while applying it to the wrong product or effective date. Check the relationship between the claim, source, and task context.
Pair groundedness with coverage of required information. An answer containing one supported sentence can score perfectly on groundedness while omitting four required findings. Define coverage against the information the task requires, not against the claims the model chose to make. Groundedness constrains what is asserted; coverage constrains what is missing.
Calibrate the decision to abstain
A useful abstention prevents an unsupported answer when required information is unavailable. An unnecessary abstention withholds an answer that the authorized sources support. Label answerable and unanswerable cases explicitly, then measure both behaviors.
For an application-defined answerability rubric, report correct abstentions among unanswerable cases and unnecessary abstentions among answerable cases. Also report answer coverage, the proportion of all evaluated tasks receiving a substantive answer, and correctness among answered tasks. These denominators make an answer-everything policy and an abstain-everything policy visibly different.
If the application uses a confidence score to choose abstention, calibrate it against observed correctness on labeled cases. A group scored around 0.9 should achieve roughly that correctness under the declared scoring method before the score is interpreted as a probability. The number is an application estimate, not a verified probability merely because Claude printed it. Compare candidate thresholds on accepted-answer error and coverage together.
For example, a stricter threshold can reduce errors among answered tasks while doubling review demand. The domain owner must decide whether that loss of coverage satisfies the business requirement. Anthropic documents permission to say “I don’t know” as a hallucination-reduction technique; the scorecard measures whether using that permission is appropriate.
Count valid delivery and completed work
Define format compliance as outputs satisfying the application interface divided by outputs expected to satisfy it. The interface can include serialization, required fields, and locally enforced restrictions. Keep this outcome separate from factual quality. On a supported model and integration, output_config.format configures JSON outputs; it does not replace source evaluation. Check that deployment against the structured-outputs support matrix: for example, the page distinguishes support in the legacy Amazon Bedrock integration from its absence in Claude in Amazon Bedrock.
Anthropic documents exceptions that the format measure must retain. stop_reason: "refusal" can produce content outside the schema. stop_reason: "max_tokens" indicates truncation; the documented remedy is retrying with a higher max_tokens. String enum and const capitalization is not guaranteed, so follow the documented case-insensitive comparison guidance and avoid values distinguished only by capitalization. These outcomes must not disappear because a parser never produced a record.
Define task completion using the business terminal condition: the requested result is delivered and accepted under the task rubric. An agent saying “done” or returning a well-formed record is insufficient. For a recommendation workflow, completion might require a supported answer covering every required question; for an action workflow, it requires observing the authorized result. Keep component scores for diagnosis and the whole-task result for acceptance.
Report completion per trial, then use repeated trials to measure reliability: Anthropic defines pass@k as the probability of at least one correct solution in k attempts and pass^k as the probability that all k trials succeed. Choose pass@k when one usable solution suffices; for a customer-facing agent whose acceptance requires consistent success, set a pass^k bar for a declared k rather than relying on a single run. Anthropic illustrates the distinction with a 75% per-trial success rate: the probability of all three trials succeeding is approximately 42%.
Key distinctions and trade-offs
Precision and recall make different errors expensive. Groundedness and coverage prevent opposite forms of incomplete reporting. Answered-task correctness and abstention behavior reveal whether apparent quality came from declining difficult work.
Completion is the intersection your workflow actually requires, not the arithmetic average of component scores. Measure it directly: correlations between missing information, abstention, and format failures make multiplication of separate rates unreliable. A high score for an intermediate output does not establish a successful business result.
More detailed claim scoring increases evaluation effort. Use a fixed rubric and an accountable domain owner, with difficult cases retained rather than excluded. Dataset and grader construction belong to 4.2; this lesson specifies what those graders must measure.
Architecture / decision scenario
A retailer needs policy-grounded answers to return requests, including an exception decision and the next action. The service owner requires at least 95% recall on eligible exceptions and no more than 2% unnecessary abstentions on answerable cases. The following figures are illustrative evaluation results.
Candidate A achieves 98% overall classification accuracy, 99% groundedness, and 100% format compliance. It finds only 70% of eligible exceptions, abstains unnecessarily on 12% of answerable requests, and completes 76% of tasks. Candidate B reaches 97% classification accuracy, 97% exception precision, 96% exception recall, 99% groundedness, 99% format compliance, and 91% completion, with 1% unnecessary abstentions.
The tempting choice is A because its headline accuracy and schema results are higher. The architect chooses B for the bounded pilot: it meets the exception and abstention requirements, and its completed answers include both required parts. The domain lead verifies support and completeness rather than accepting citation presence as proof. Engineering retains B’s format failures in the completion denominator and diagnoses them separately.
Before approval, the owner compares two abstention thresholds for B on a held-out set. The stricter threshold yields 99% answered-task correctness at 75% answer coverage; the selected threshold yields 98% at 90% coverage and correctly abstains on 98 of 100 unanswerable cases. The business requires at least 98% answered-task correctness and 85% coverage, so the stricter setting fails despite its higher conditional accuracy. Raising coverage by guessing unavailable purchase facts would violate the same quality definition that made B preferable.
Common exam traps
- Letting dominant-class accuracy hide poor exception recall.
- Treating a matching quotation as proof that the conclusion follows.
- Reporting correctness only on answered tasks without abstention coverage.
- Treating a model-written confidence value as an established probability.
- Counting schema compliance as completed business work.
- Dropping refusals and truncated outputs because parsing failed.
What to remember
Define quality around the business decision and its required result. Pair precision with recall, groundedness with coverage, and answered-task correctness with appropriate abstention. Measure format compliance separately, then count actual task completion under the complete rubric.
Knowledge check
A retailer’s assistant classifies return requests, and about 5% of them are eligible exceptions. The service owner says the costly error is an eligible exception classified as an ordinary return; a false alarm costs only a quick staff check. A candidate reports 97% overall accuracy. Which measure should gate its release?
An audit assistant must report all five required findings for each control, every finding supported by the supplied evidence. Candidate X scores 100% groundedness, but reviewers notice its summaries often contain only one or two findings. The summary schema checks serialization and field types, not which findings are present. Which measure should the scorecard add?
A policy assistant may abstain when the authorized sources do not answer a question. The business requires at least 98% correctness among answered questions and at least 85% answer coverage across all evaluated questions. On a held-out labeled set, four abstention thresholds give these results:
- Strict: 99.2% correct among answered, 72% coverage
- Moderate: 98.4% correct among answered, 88% coverage
- Lenient: 97.1% correct among answered, 95% coverage
- None (answer everything): 94.0% correct, 100% coverage
Which threshold meets the requirement?
A claims pipeline asks Claude for structured output with output_config.format and reports per-attempt format compliance as parsed records divided by records produced. Responses that end with stop_reason: "refusal" or stop_reason: "max_tokens" fail to parse and never produce a record, so the report shows 100%. How should the format measure change?
end_turnoutput_config.format to keep every response in schemamax_tokens means the output was truncated. Filtering by stop reason or reporting only the retried result makes the same outcomes disappear.A bank’s customer-facing agent handles card disputes, and every run is seen by a customer, so the service owner requires consistent success on each task type. The team reports a pass@3 of 98%, based on a per-trial success rate of about 75%. Which metric should set the acceptance bar?
kkk attempts, which suits a tool where one usable result is enough; it rises as k grows. pass^k is the probability that all k trials succeed, which is what consistency means here: at 75% per trial, all three succeed only about 42% of the time. A single run hides that variation.References
Anthropic, Demystifying evals for AI agents
Anthropic, Reduce hallucinations
Anthropic, Structured outputs