AnthropicClaude Certified Architect – Professional2026-104.6 Monitor system performance using logging and observability tools

Alert when healthy traffic hides deterioration

Completed by the knowledge check

Learning objective

Alert on latency, error, cost, and quality drift — including retrieval freshness and shifts in task mix — against baselines and SLOs.

Core idea

An alert needs a decision attached to it: who acts, under what condition, and with what first response. A dashboard departure is a reason to investigate; it is not automatically a reason to page someone. Google SRE distinguishes the symptom affecting users from the internal cause that helps explain it.

See 3.4.3 for signal selection and 3.4.4 for monitoring ownership. Here you specify alert rules that retain segment denominators, compare like with like, and detect silent deterioration even when calls remain fast and successful.

How it works

Write an alert specification

Use the owned alert record from 3.4.4, then specify its baseline, SLO comparison, and minimum observation count for the affected segment.

A service level indicator (SLI) measures a defined aspect of service; a service level objective (SLO) gives its target. A baseline describes observed behavior. Keep both: a rolling baseline can slowly follow a degraded service downward, while a fixed SLO still exposes the breach. Conversely, an unusual pattern can deserve investigation before it breaches an SLO.

Separate rules by the decision they support:

Signal	Comparison	First operating question
End-to-end latency	Tail latency and fraction within the user target, by task class	Is the caller missing its deadline, and which step adds the delay?
Errors	Failed workflows and provider error categories, with distinct denominators	Is the failure in admission, generation, tool execution, or final validation?
Cost	Spend rate, forecast, and estimated cost per workflow within a task class	Is traffic increasing, or is each comparable task consuming more?
Quality	Fixed acceptance floor and version-matched sampled scores	Is output worse for comparable inputs, and is the grader unchanged?
Retrieval freshness	Source-to-index lag and stale passages returned	Did a valid source update fail to reach the serving path?
Task mix	Shares and counts by intent, channel, and risk class	Did the workload change enough to explain the aggregate?

The numerical thresholds belong to the service agreement. Copying another team’s two-percent error rule or latency percentile does not establish an acceptable risk level for your users.

Preserve the failure’s meaning

Classify admission failures before choosing a response:

Capacity rate limit: HTTP 429 rate_limit_error can indicate exhausted request or token capacity. Check the limit named in the error and honor retry-after when present. Track anthropic-ratelimit-requests-remaining and anthropic-ratelimit-tokens-remaining as leading headroom signals; the token figure is rounded to the nearest thousand.
Tier spend cap: HTTP 429 also occurs at the organization’s monthly tier cap. On the Messages API, error.details.error_code: "enforced_spend_limit_reached" identifies this path, which has no retry-after. Retrying, including SDK automatic retries, fails until access resumes. Send the first check to the billing owner: confirm the cap and arrange a higher limit or wait for the documented reset.
Configured spend limit: A limit the organization or workspace sets returns HTTP 400 invalid_request_error, with a message identifying the specified usage limit. Check that limit before classifying a burst of 400s as malformed requests; the budget owner can raise or remove it. Claude Code workspace limits are checked separately and can instead return 429 with retry-after.

Keep HTTP 529 overloaded_error separate for provider-overload investigation, and retain stream failure because an error can follow HTTP 200.

Generation termination also informs the investigation. A rising share of stop_reason: "max_tokens" identifies output truncation, which calls for examining the configured output budget and task requirements. A rise in tool_use is a behavior change, not a failure by itself. Interpret it with actual tool outcomes and user acceptance. See 4.6.1 for the fields that preserve these distinctions.

Cost alerts need two clocks. Application estimates can expose an immediate spend-rate change; provider reports support delayed reconciliation. Usage and Cost API data typically appears within five minutes of API request completion, sometimes longer, and cost buckets are daily. Do not build an immediate overspend response on the assumption that its cost endpoint supplies minute-level truth.

Normalize release comparisons to a reference mix

Use per-task-class scores to distinguish task-mix shifts from regressions, as 3.4.4 explains. Suppose easy questions pass at 98% and difficult questions at 80%: a 90 reference mix gives 0.9 × 98% + 0.1 × 80% = 96.2%, while a 50 production mix gives 0.5 × 98% + 0.5 × 80% = 89%. Applying the reference weights to both releases keeps their normalized comparison at 96.2%.

A fixed reference mix is useful for release comparisons: apply the same chosen weights to both versions’ class scores. Display actual production performance alongside that normalized comparison. Normalization helps attribute the change; it must not hide that real users are experiencing a lower success rate. Product and capacity owners still need to respond to a shift toward harder work.

Detect stale retrieval and silent monitoring

The ingestion and retrieval mechanisms belong to 3.5.2–3.5.4. Add an operating freshness specification: source revision and update time, index revision and publication time, and the revision of each returned passage. These are application metadata. Measure both update lag and the share of retrievals returning superseded material; a pipeline that publishes promptly can still serve an obsolete index.

Pair scheduled probes for known changed facts with sampled answer checks. A stable latency graph cannot establish that a policy effective yesterday appears in today’s answers. Give the source owner responsibility for publication failures and the retrieval owner responsibility for serving the correct revision.

Treat missing Agent SDK telemetry as a separate alert condition because export failures are silent by default; see 3.4.1 for exporter diagnostics.

Key distinctions and trade-offs

Page for actionable user harm or an imminent breach with a defined response; queue lower-urgency baseline deviations for owned review. Google SRE warns that noisy alerts teach operators to ignore the pager. Minimum counts and persistence windows reduce noise, but extend detection time for rare high-impact failures; supplement those with targeted probes and review.

Quality scores lag when humans adjudicate them. Behavioral signals such as tool-call rate can detect change sooner, but require verification of harm. Maintain the quality floor while investigating the cheaper leading signal.

Architecture / decision scenario

A healthcare scheduling assistant must complete 95% of requests within eight seconds and keep verified scheduling-policy accuracy at or above 95%. A release coincides with complex referrals increasing from 10% to 50% of traffic. Per-class scores remain 98% and 80%, producing the aggregate decline from 96.2% to 89% described earlier. Separately, retrieval probes still return yesterday’s revision after the team’s agreed 15-minute publication window.

The p95 latency rises from 6.5 to 7.4 seconds, remaining inside eight seconds, and provider errors stay at 0.4%. Referral verification also grows from two to three calls per workflow, raising that class’s estimated cost from $0.08 to $0.12 against an internal $0.10 review threshold. These are scenario measurements, not provider prices.

The tempting response is to roll back the prompt because aggregate quality changed at release time. The architect keeps latency and error alerts quiet, tickets the product owner for the within-class cost increase and difficult-task demand, and pages the retrieval owner for the missed freshness target. The quality floor remains breached regardless of attribution, so affected scheduling answers receive the service’s predefined reviewed workflow until recovery.

Per-class counts, release markers, passage revisions, cost estimates, and error categories let each owner verify recovery on the same population.

Common exam traps

Treating a moving baseline as the service’s acceptance target.
Blaming a prompt release for an aggregate change caused by task mix.
Treating tool-call rate as a direct measure of answer correctness.
Measuring index publication but ignoring which revision retrieval serves.
Assuming no telemetry means no failures or no traffic.
Retrying a tier spend-cap 429 or treating every configured-limit 400 as a client bug.
Paging on every anomaly without an owner and an actionable runbook.

What to remember

Compare stable segments against both observed baselines and fixed service targets. Keep error, cost, quality, freshness, and task-mix explanations distinct. An alert earns its interruption when it identifies a decision and gives the responsible owner enough context to act.