AnthropicClaude Certified Architect – Professional2026-103.4 Analyze observability challenges and select monitoring strategies at scale
Assign review methods and alert owners
Completed by the knowledge check
Learning objective
Choose monitoring strategies — sampled human review, automated online evaluation, anomaly alerting — with clear alert ownership.
Core idea
Monitoring needs a decision path after detection. An automated score without a reliable grader can mislead; human review without a sampling plan can miss an entire route; an anomaly alert without an owner can remain unanswered. Design these methods as complementary parts of one operating process.
Anthropic’s evaluation guidance distinguishes code-based, human, and LLM-based grading. Code grading is fast and scalable for explicit rules, humans handle nuanced judgments at higher cost, and LLM grading requires reliability testing before scaling. The architect chooses which criterion each method measures, how quickly it must respond, and who can act on its result.
How it works
Begin with an operational criterion
Write a measurable success criterion before choosing a grader. “Helpful answers” does not tell an on-call engineer what broke. “Answers about reimbursement state only amounts supported by the supplied policy and identify missing policy information” supports both a rubric and a review decision.
Pair each criterion with its population and response expectation. A quality score on reviewed drafts is not automatically the quality rate across all served answers. Google’s SRE guidance separates the measured SLI from its SLO target; apply the same discipline when specifying a quality indicator. Record the coverage of the measurement alongside its value.
Use the production run reference and rubric revision to connect each assessment to what was served. The exact evaluation lifecycle and offline test design belong to domain 4. Here the issue is which production observations trigger review, an investigation, or a service response.
Sample human review deliberately
Human review is appropriate when consequences depend on context that a deterministic check cannot settle or when you need to validate an automated grader. Anthropic’s research account describes human evaluation finding source-selection biases and unusual failures missed by automated evaluations.
Allocate review across business routes, release cohorts, and high-impact outcomes. Keep a random baseline as well as a flagged queue. Reviewing only complaints or low automated scores is useful for discovering defects, but that selected population does not estimate overall quality. Conversely, a small random sample can miss rare consequential failures.
Document the selection reason on each review item. A reviewer should know whether the item was randomly selected, escalated by a user, or flagged by a check. When comparing periods, hold the sampling method stable or report the change. A rise in findings after increasing high-risk sampling does not alone prove a production regression.
The review artifact should capture criterion, assessment, uncertainty, reviewer role, and a run reference. These are application-defined fields. Give ambiguous assessments an adjudication path instead of forcing false agreement between reviewers who interpreted the rubric differently.
Automate checks that can support action
Use code grading for explicit rules such as required output fields, allowed category values, or a required action receipt. Reserve LLM grading for judgments that need interpretation, such as whether an answer’s claims are supported by provided sources. Anthropic recommends detailed rubrics, specific outputs, and reliability testing before scaling LLM graders.
Calibrate the grader against human-assessed examples that include acceptable abstentions, plausible unsupported answers, and difficult edge cases. Track disagreements rather than accepting a single average agreement rate as sufficient for every route. A grader that systematically accepts unsupported financial claims is unsuitable for that criterion even if its aggregate score looks strong.
Automated online evaluation means assessment of production traffic; it need not block the response. An asynchronous grader protects user latency but detects defects after delivery. An inline check can reject an invalid response before delivery but consumes the request’s latency budget. Choose the location from the consequence and time allowed to react.
Keep grader failures separate from answer failures. A timed-out evaluation means unknown quality for that item, not a passing score. Record evaluation queue age and completed-assessment coverage so a stopped grader cannot leave a reassuring but stale dashboard.
An inline check can reject an invalid response before delivery inside the request’s latency budget, while an asynchronous grader detects defects after delivery.
Route anomalies to someone who can respond
An anomaly is a departure from a baseline, not a diagnosis. Segment quality, latency, and usage by route and release cohort before concluding that a prompt regressed. A shift toward difficult requests can change aggregate quality even when every route’s behavior is stable.
Create an alert record with the measured signal, affected population, comparison window, trigger, owner, and first response. Google SRE distinguishes pages, tickets, and email alerts, and separates symptoms from causes. In this design, an urgent user-facing SLO-breach symptom pages the service owner with an actionable first response; quality drift or grader-freshness gaps open tickets for the domain or evaluation owner. Reserve email alerts for nonurgent information that needs no immediate intervention. Numeric thresholds are application choices, not Anthropic defaults.
For telemetry availability, see 3.4.1. The missing-export alert belongs to platform on-call, whose first check is whether incoming collector records arrive. Domain reviewers should not investigate a fabricated quality improvement caused by missing data.
Key distinctions and trade-offs
The methods answer different operational questions:
| Method | Best use | Main limitation | Accountable role |
|---|---|---|---|
| Sampled human review | Nuance, rare failure discovery, grader calibration | Cost and delayed coverage | Domain review lead |
| Automated production evaluation | Repeated checks and scalable rubric scoring | Calibration errors and grader outages | Evaluation owner |
| Anomaly alerting | Detecting changes that merit investigation | A changed mix can resemble a regression | Service or component owner |
Assign ownership by remedy. A source index defect goes to retrieval ownership, a prompt-specific regression to the release owner, and missing telemetry to the platform team. One incident can involve all three; the alert still needs one initial responder.
Architecture / decision scenario
An education provider serves 80,000 tutoring replies per day. Its four subject specialists can review 120 replies daily, and the product target is to investigate harmful teaching regressions within one working day. Reviewing every answer is infeasible; paging engineers on every low grader score would overwhelm the on-call team.
The architect assigns 60 daily reviews to a random route-stratified baseline and 60 to flagged or disputed answers. A calibrated production grader assesses source support asynchronously, while deterministic checks validate required output fields before delivery. The domain lead owns disputed teaching judgments; the evaluation owner owns grader reliability and queue freshness.
A ticket opens for the domain lead when the assessed algebra cohort falls below the team’s agreed quality threshold with sufficient reviewed volume; a grader-freshness gap opens a ticket for the evaluation owner. The release owner compares prompt cohorts and request mix before rollback. An urgent SLO breach pages the service owner, while a separate missing-export alert goes to platform on-call. The design fits reviewer capacity, preserves a baseline, detects production changes, and gives each alert an actionable owner.
Common exam traps
- Choosing an LLM grader without checking it against human judgments.
- Reporting the failure rate in a flagged queue as the overall failure rate.
- Assuming online evaluation must run synchronously before delivery.
- Treating an anomaly as proof of a particular root cause.
- Converting grader timeouts into passing quality assessments.
- Sending alerts without a named responder and first action.
What to remember
Use human review for nuance and calibration, automation for repeatable coverage, and alerts for timely response. Preserve sampling context, grader reliability, and measurement freshness. Every trigger needs an owner who can inspect the relevant runs and apply a concrete remedy.
Knowledge check
A legal-research assistant’s reviewers read only answers that users disputed or that the automated grader scored low. They find problems in 30% of reviewed answers, and the weekly report presents 30% as the production failure rate. What should the review design add?
A legal-research service wants to scale an LLM grader that judges whether each served answer’s claims are supported by the supplied sources. It has not yet compared the grader with human judgments. What should come before scaling?
An asynchronous production grader stalled for six hours. During that time, items it timed out on were recorded as passing, and the quality dashboard looked unusually good. How should a timed-out evaluation be recorded?
A tutoring service’s alert fires because no telemetry has reached the collector for 40 minutes, and the quality dashboard shows an apparent improvement over the same window. Who should receive the alert first?