Measure service health and answer quality

Learning objective

Select signals to monitor at scale: latency, errors, token usage, cost, cache hit rate, stop reasons, and quality indicators.

Core idea

A production dashboard needs several independent views of health. A faster answer can be less accurate, a lower token count can hide omitted cache usage, and a low returned error rate can coexist with expensive recovery. Select signals that expose those trade-offs rather than compressing everything into a single success counter.

For an architect, the deciding artifact is a signal specification: what event is counted, which population supplies the denominator, which dimensions are retained, and which decision the signal supports. Anthropic supplies execution and usage fields. You define the business outcome and the aggregation that makes it meaningful.

How it works

Separate elapsed time from component time

Measure end-to-end latency at the application request boundary. Add model and tool durations from the execution tree in 3.4.2 to localize delays. A model-call timer alone omits retrieval, approval waits, retries outside that call, and downstream work after generation.

See 1.6.3 for service-level indicators (SLIs), objectives (SLOs), and percentile selection. Keep user-visible latency separate from component timing. Claude Code’s beta claude_code.llm_request spans expose ttft_ms for time to first token and, on v2.1.268 or later, first_content_ms for time to the successful attempt’s first content block; the latter is absent after non-streaming fallback. Neither necessarily measures the first output rendered to the user or complete-result latency.

Do not compare a turn-level p95 with a per-model-call p95 as though they were the same population. A request with ten calls contributes once to the first distribution and ten times to the second. Keep the population in the chart description and incident runbook.

Count errors at the boundary where they occur

Record transport or API failures, tool execution failures, and business failures separately. Claude Code emits one claude_code.api_error only after giving up on internal retries; intermediate failures are not separate error events. Its attempt attribute gives the attempt count. Each retry also produces a gen_ai.request.attempt span event on claude_code.llm_request. Refusals arrive on successful responses as claude_code.api_refusal, not api_error. A tool error and an unsupported final claim need different investigation paths even when both affect one request.

Each retry adds a span event, and Claude Code emits one claude_code.api_error only after giving up on internal retries.

Use request-level failed or incomplete outcomes for the customer-facing denominator, with step-level errors for diagnosis. Recovery can make the final request successful without making its intermediate failures irrelevant: those attempts still consume latency and cost. Conversely, successful model calls do not erase a failed downstream action.

Missing usage is another distinct condition. Anthropic documents that failed or aborted requests can lack token counts in exported spans. Treat missing values as unavailable, rather than turning them into confirmed zero-cost requests. Investigate collection gaps before interpreting a sudden usage decline as optimization.

Count all input-token categories

For the Messages API, cached requests report separate usage.input_tokens, usage.cache_creation_input_tokens, and usage.cache_read_input_tokens. Uncached input is not the whole prompt. The documented total is:

total input = input_tokens
            + cache_creation_input_tokens
            + cache_read_input_tokens

Keep usage.output_tokens separately. In Claude Code telemetry, claude_code.token.usage distinguishes type values input, output, cacheRead, and cacheCreation. Its input category also excludes cache reads and writes. Preserve those categories through aggregation instead of charging every token as ordinary input.

Define cache hit rate explicitly. A request-hit measure can count requests with positive cache_read_input_tokens divided by requests eligible for your cache policy. A token-reuse measure can divide cache-read tokens by total input tokens. These are application-defined measures with different meanings: many tiny hits can produce a high request-hit rate while reusing little of the input volume.

Retain cache creation as well as reads. A release that repeatedly creates prefixes might show high cache activity without achieving useful reuse. Zero cache-creation and cache-read counts mean the request was not cached; see 2.5.1 for diagnosis rather than inferring a hit from a configured breakpoint alone.

Reconcile cost at the correct scope

Claude Code exposes claude_code.cost.usage in USD as an approximation, not official billing data. For direct API integrations, retain the usage categories and the applicable pricing schedule used by your application estimate. Do not label a local token calculation the provider’s final invoice or assume token-only estimates cover separately billed tools.

For Claude Console organizations, the Usage and Cost Admin API provides /v1/organizations/usage_report/messages for bucketed usage and /v1/organizations/cost_report for cost reporting. The cost endpoint uses daily buckets and excludes Priority Tier costs; Priority Tier usage is available through the usage endpoint. These reports support aggregate reconciliation, not attribution to an individual prompt revision or user request.

Use local request traces for route and release attribution, then reconcile both Claude Code cost approximations and direct API estimates against official provider billing reports. Claude Enterprise organizations have a separate Analytics API path; the Console Admin API is not a universal reporting interface for every Claude product.

Pair termination with quality indicators

Monitor stop_reason as a distribution, not as a Boolean error flag. Claude Code exposes it on claude_code.llm_request spans and as a string array in gen_ai.response.finish_reasons. tool_use is normal control flow. max_tokens indicates the configured output limit was reached; the documented remedies include raising max_tokens or continuing the response. refusal needs its own category and stop_details inspection rather than being merged into truncation. Track pause_turn separately: a server-tool loop reached its iteration limit, and you send the assistant content back to continue. model_context_window_exceeded means generation filled the model’s context window; treat it as truncated and monitor its share for long-running agents.

end_turn does not establish quality. Add outcome indicators defined against your success criteria: source-supported claims, required-field validity, completeness, or verified action completion. Record rubric and grader revisions for sampled scores. See 3.4.4 for review selection and alert ownership.

Key distinctions and trade-offs

Signal versus judgment: Latency, usage, and termination describe execution. A quality indicator applies a business criterion to the outcome.

Per-request attribution versus billing reconciliation: Traces explain why a route became expensive. Aggregate provider reports reconcile usage and charges at their documented scope.

Request cache hits versus reused tokens: One measures how often reuse occurs; the other measures how much input it serves. State the denominator before comparing them.

Bounded dimensions such as route, model, and release cohort support comparisons. Every further dimension raises storage and analysis cost. Keep detailed request identifiers on diagnostic records and apply the cardinality controls in 3.4.5.

Architecture / decision scenario

A technology company’s support assistant has a two-second median target and a six-second p95 target. After a prompt release, median latency improves from 1.8 to 1.5 seconds, but p95 grows from 5.5 to 9 seconds. A token chart falls by 40%, and management proposes expanding the release.

The architect delays expansion. Cache-aware accounting shows that the chart omitted cache-read tokens; total input did not fall by 40%. Tool spans identify repeated dependency calls in the slow requests. The max_tokens share also rises, and completeness checks fall from 97% to 91%.

The service owner repairs the dependency behavior and checks the output limit against required answer content. Finance reconciles aggregate cost separately. Expansion depends on meeting both the latency target and the completeness criterion, rather than celebrating a faster median and a misleading token chart.

Common exam traps

• Using average model latency as the user-visible latency distribution.
• Counting missing usage as confirmed zero usage.
• Ignoring cache writes and reads when computing input volume.
• Comparing cache-hit rates with different denominators.
• Expecting daily aggregate cost reports to identify one bad prompt execution.
• Treating tool_use as failure or end_turn as a quality score.

What to remember

Specify populations and denominators, retain every token category, and separate provider reconciliation from request attribution. Latency, errors, cost, cache behavior, and stop reasons explain execution; quality indicators determine whether that execution served the user.

Knowledge check

References

• Anthropic, Monitoring usage
• Anthropic, Observability with OpenTelemetry
• Anthropic, Prompt caching
• Anthropic, Usage and Cost API
• Anthropic, Stop reasons and fallback
• Anthropic, Define success criteria and build evaluations
• Google, Site Reliability Engineering: Service Level Objectives