AnthropicClaude Certified Architect – Professional2026-100 Prerequisites
Define what each performance number measures
Completed by the knowledge check
Learning objective
Define core evaluation and operations terms: accuracy, precision and recall, latency percentiles, time to first token, throughput, cost per task, and reliability.
Core idea
A performance number has meaning only with a task, population, measurement boundary, and unit. For an architect, this vocabulary lets you compare proposals without confusing answer quality, responsiveness, capacity, and successful delivery.
Anthropic’s evaluation guidance calls for specific, measurable, relevant criteria and task-specific test cases. The definitions here establish the baseline; 4.1.2 and 4.1.3 own the detailed quality and operational metric designs.
How it works
For binary classification, define the positive class first. A true positive is a correctly identified positive case; a false positive incorrectly labels a negative case positive; a false negative misses a positive; a true negative correctly identifies a negative.
Anthropic lists accuracy, precision, and recall among its generic evaluation metrics. Their classification definitions have different denominators:
| Measure | Definition | Question answered |
|---|---|---|
| Accuracy | Correct classifications divided by all classifications | How often is the classification correct overall? |
| Precision | True positives divided by true positives plus false positives | How many predicted positives are actually positive? |
| Recall | True positives divided by true positives plus false negatives | How many actual positives are found? |
These formulas describe classification. For generated answers, you must define what counts as correct before reporting “accuracy.” High classification accuracy can conceal failure on a rare positive class; precision and recall expose different mistakes.
Latency is elapsed time between defined start and end events. A latency percentile summarizes a distribution: p50 is the median, while p95 describes a value at or below which roughly 95% of observations fall. A p95 is neither a mean nor a maximum. Define the observation window and whether the measurement ends at model completion or completed application work.
Anthropic defines time to first token (TTFT) as the time from sending the prompt until the model generates the response’s first token. It concerns initial responsiveness, especially for streaming. It does not measure delivery of the complete, validated answer. See 3.3.2 for streaming and other latency mechanisms.
Throughput measures work per unit time, such as completed tasks per second. State the unit: output tokens per second and completed business tasks per second measure different work. High throughput alone says nothing about correctness.
Cost per task divides the cost assigned to a workload by its task count. Specify whether the denominator includes all attempted tasks or only successful tasks, and which costs enter the numerator. For architectural comparisons, record whether retries, tool execution, and human review are included; 4.1.3 and 4.5.1 own that accounting.
For this baseline, define reliability through two observable questions in Google’s SRE guidance: is the service usable, and does it return correct results? Track these over a defined measurement window. Availability is often measured as the fraction of well-formed requests that succeed, also called yield. Correctness asks whether the right answer, data, or analysis was returned. An available endpoint that returns unusable answers does not satisfy the business task. See 1.6.3 for user-journey targets and 4.6.3 for operational alerts.
Key distinctions and trade-offs
Precision emphasizes false positives; recall emphasizes missed positives. TTFT emphasizes initial response; completed-task latency includes the remaining work. Throughput measures capacity; cost per successful task measures the expense of acceptable outcomes. Compare configurations on the same definitions rather than selecting the largest or smallest isolated number.
Architecture / decision scenario
A retail support team must identify refund-eligible requests without overwhelming reviewers. Of 100 labeled cases, 20 are eligible. A candidate finds 16 and wrongly flags eight ineligible cases: accuracy is 88%, precision is about 67% (16 of 24 flagged), and recall is 80% (16 of 20 eligible).
You also request p95 completed-task latency, TTFT, completed tasks per second under a stated load, total cost divided by successful tasks, and the successful-request fraction over a defined window. A faster first token cannot offset a slower refund decision. The support owner weighs missed refunds against unnecessary reviews; the platform owner measures dependable completion.
Common exam traps
- Swapping precision’s predicted-positive denominator with recall’s actual-positive denominator.
- Treating p95 as the slowest request or TTFT as total latency.
- Comparing token throughput with task throughput or ignoring failed-task costs.
What to remember
Write the numerator, denominator, unit, and measurement boundary before using a metric to choose an architecture.
Knowledge check
A fraud screen is tested on 200 labeled transactions, 40 of them fraudulent. It flags 50 transactions, and 30 of the flagged ones are fraudulent. What are its precision and recall?
A support platform compares two configurations on the same workload and load. A reports 90 output tokens per second; B reports eight completed support tasks per second. The team needs to compare task throughput. What should it do?
A loan assistant streams its explanation, then the application validates it before an agent sees the decision. The requirement is that 95% of decisions reach agents within six seconds. Which measurement should the team report against it?
Two configurations handle the same workload, and each cost figure includes all model and tool costs. Configuration A costs $0.020 per attempted task and succeeds on 95% of tasks; configuration B costs $0.015 per attempted task and succeeds on 60%. Which configuration costs less per successful task, and at what figure?