AnthropicClaude Certified Architect – Professional2026-104.1 Define evaluation metrics (accuracy, latency, cost, safety, security)
Price successful work on the user's clock
Completed by the knowledge check
Learning objective
Define operational metrics: latency percentiles, time to first token, throughput, and cost per successful task including retries and human review.
Core idea
A fast model response and a cheap API call are intermediate observations. The operational question is how quickly the system delivers acceptable work, at the offered workload, and what each successful task costs after unsuccessful work and review are included.
Define the clock, population, success condition, and accounting boundary before comparing candidates. Anthropic documents time to first token and response usage; Google’s SRE guidance explains latency distributions and throughput. Combine those observations into application-level measures without claiming that an API event alone establishes business completion.
How it works
Measure the waiting party’s interval
Record at least two boundaries: when the application accepts the task and when its result becomes usable. End-to-end latency spans that interval, including queueing, retrieval, model calls, tools, validation, retries, and any review required before delivery. Stage timings explain delays but do not replace this user-facing clock.
Anthropic defines time to first token (TTFT) from sending the prompt until the model generates its first response token. Keep an API TTFT measure distinct from time until the user sees useful text. Retrieval before the request, transport, buffering, and display can widen that interval. Name both measurements explicitly if the interface depends on perceived responsiveness.
For the Messages API, stream: true requests streaming. The documented sequence includes message_start, content blocks and their deltas, message_delta, and message_stop. A message_start carries empty content, so it is not a useful first-answer measurement. For user-visible text, instrument the first nonempty content_block_delta whose delta.type is text_delta, and separately record when that text reaches the client. This is an application definition of visible-text latency.
A stream can contain tool activity before final answer text. Receiving a tool argument or an initial status sentence does not imply the answer is ready. Likewise, message_stop ends one message, not necessarily a multi-call task. End the task clock at the terminal condition defined in 4.1.2, after required validation and review.
Keep the tail and failures visible
Report latency percentiles such as p50, p95, and p99 with the workload and observation window. A p95 value of 20 seconds means approximately 95% of observations finish within that duration under the stated percentile convention. The slow tail matters even when the mean is acceptable; Google SRE explicitly recommends percentiles to expose distribution shape.
Declare how timed-out and failed tasks appear. Successful-task latency can diagnose the completed subset, but excluding failures from the only latency view rewards a system that abandons slow work. Pair success-conditioned percentiles with timeout and failure counts and an all-task measure such as the proportion receiving an acceptable result before the deadline.
Measure delivered throughput at stated load
Throughput is completed work per unit time. Google SRE describes requests per second as a common throughput indicator. For an agent workflow, state whether you count model calls, terminal tasks, or accepted successful tasks. Retries can raise call throughput while leaving business throughput unchanged.
Record offered load, concurrency, and task mix with the result. A measured 40 accepted tasks per minute at one workload does not establish the same capacity for tasks requiring several tools and review. Keep queue depth or age beside delivered throughput when arrivals can exceed completions; a growing backlog means the tested system is not sustaining that arrival rate.
The service owner should read throughput and latency together. Increasing concurrency can raise completion volume while worsening tail latency. A capacity result is useful only with the quality and deadline requirements that its completed work satisfies. Optimization mechanisms belong to 4.5; here you define the comparison they must survive.
Charge the whole workload to its successful outcomes
Anthropic’s “Optimizing for cost and intelligence” recommends comparing models on cost per completed task, including the tokens, retries, and downstream consequences of failed tasks, and measuring the ranking on your own workload. For a declared cohort, define cost per successful task as total cohort cost divided by tasks meeting the success rubric. Include unsuccessful attempts in the numerator. If none succeed, report zero successes and the incurred cost; do not present a finite efficiency figure.
Build a task cost ledger that joins every model attempt to its originating task and records tool charges and review labor. For Claude API token accounting, retain usage.input_tokens, usage.output_tokens, usage.cache_creation_input_tokens, and usage.cache_read_input_tokens. Apply the applicable model and feature rates from Anthropic’s pricing page rather than multiplying every category by one rate. Include separately billed features when used, and retain billed refused attempts according to Anthropic’s refusal-billing rules rather than assuming every refusal is free. Record human minutes at an explicitly chosen labor rate.
Streaming message_delta.usage counts are cumulative. Accumulate the final totals for an attempt rather than summing every update as additional tokens. When a response is lost and usage is unavailable, mark the cost unresolved and reconcile it against billing; missing telemetry does not establish zero spend. Separate estimated run cost from reconciled billed cost in the ledger.
Human review changes both cost and the definition of success. If review makes a result acceptable, include its labor and waiting time in the reviewed workflow. If comparing autonomous completion, count that same task as requiring intervention and report the reviewed outcome separately. Keep the boundary consistent across candidates.
Key distinctions and trade-offs
TTFT measures response onset; end-to-end latency measures delivery. Output tokens per second measures generation speed; accepted tasks per minute measures useful throughput. Cost per call measures an invocation; cost per successful task pays for all the work needed to obtain acceptable outcomes.
A configuration can improve one measure while worsening another. Streaming can reveal text earlier without shortening validation. More retries can recover tasks while increasing spend and tail delay. Greater automation can reduce labor only if quality remains acceptable. Compare candidates using the same success rubric and cohort boundary.
Architecture / decision scenario
A technology company’s internal support assistant must sustain 20 accepted tasks per minute, show useful text within 2 seconds at p95, and deliver an accepted result within 30 seconds at p95. The finance owner caps total cost at $0.50 per successful task. These are illustrative requirements.
Over 1,000 matched tasks, candidate A has p95 API TTFT of 0.7 seconds and visible-text p95 of 1.4 seconds. It delivers 24 successful tasks per minute, but reviewed completion p95 is 90 seconds. Model and tool spend is $120, review labor is $360, and 800 tasks succeed: $480 / 800 = $0.60 each.
Candidate B has p95 API TTFT of 0.9 seconds and visible-text p95 of 1.8 seconds. It delivers 22 successful tasks per minute with completion p95 of 25 seconds. Its model and tool spend is $210, review labor is $90, and 900 tasks succeed: $300 / 900 = about $0.33 each. Both totals include retries and failed attempts.
An engineer favors A’s faster, cheaper calls. The service owner selects B because it meets throughput and both user-facing latency requirements while staying within the outcome-cost ceiling. Finance signs the labor assumptions; platform engineering owns event timing and attempt reconciliation. A’s API advantage cannot pay for review delay or missing successful outcomes.
Common exam traps
- Timing first response headers as first useful text.
- Treating
message_stopas completion of the whole agent task. - Reporting only mean latency or successful-call latency.
- Counting retry traffic as useful throughput.
- Adding cumulative streaming usage updates together.
- Comparing token spend while excluding failed tasks and review labor.
What to remember
Measure useful response onset and completed delivery separately. Report tails, workload, and failed outcomes alongside throughput. Divide the complete workload cost, including retries and human review, by results that actually meet the success definition.
Knowledge check
An internal support assistant streams Claude responses with stream: true to a chat interface. Some responses begin with a tool call before any text. The team wants a server-side timestamp for when the first user-visible text is available, recorded separately from when that text reaches the browser. Which event should set the timestamp?
message_start eventcontent_block_start eventmessage_delta eventtext_deltamessage_start carries an empty content , and a content block can start as a tool call or with no text yet, so neither marks visible text. message_delta arrives near the end of the message. The first content_block_delta whose delta is a nonempty text_delta is the earliest point visible text exists; when it reaches the browser is a separate, later measurement.Over a cohort of 1,000 matched tasks, a candidate spends $180 on model and tool calls for attempts that succeeded, $60 on failed attempts and retries, and $160 on the human review the accepted workflow requires. 800 tasks meet the success rubric. Finance caps cost per successful task at $0.40. Which figure should be compared with the cap?
A finance team reconciles a task cost ledger against Claude API billing. For non-streamed attempts the ledger’s output tokens match; for streamed attempts the ledger is several times higher. The ledger adds the usage.output_tokens of every message_delta event it receives. Which change fixes the ledger?
text_delta text insteadmessage_start eventmessage_delta are cumulative, so adding each update counts the same tokens again. Keeping the final totals of each attempt gives what the attempt actually used. message_start arrives before generation, counting streamed text only approximates tokens, and dropping retried attempts removes spend that was really billed.An HR assistant’s requirement is that 95% of tasks receive an accepted result within 30 seconds. In a 1,000-task load test, the p95 of successful tasks is 22 seconds, while 120 tasks time out at 60 seconds and are left out of the latency view. What should the service owner conclude?
A support assistant must deliver at least 20 accepted tasks per minute at the tested load. Over ten minutes, it receives 240 tasks, starts 220, makes 600 model calls including retries, and delivers 180 tasks that meet the success rubric. Which rate measures delivery against the requirement?
References
Anthropic, Optimizing for cost and intelligence
Anthropic, Refusals and fallback
Anthropic, Reducing latency
Anthropic, Streaming messages
Anthropic, Pricing
Google, Site Reliability Engineering: Service Level Objectives