Skip to lesson

AnthropicClaude Certified Architect – Professional2026-103.4 Analyze observability challenges and select monitoring strategies at scale

Diagnose outcomes beyond successful model calls

Completed by the knowledge check

Learning objective

Identify observability challenges specific to LLM systems: nondeterminism, multi-step traces, multi-agent fan-out, and quality that infrastructure metrics miss.

Core idea

For an architect operating an agent system, a successful request establishes that processing completed, not that the business task succeeded. Coordinator routing and structured failure reports provide useful local visibility. At production scale, you must also explain how many different execution paths produced acceptable outcomes, which paths failed, and which failures the application never reported as errors.

Anthropic’s multi-agent research account describes agents making different decisions even with identical prompts. Its production tracing distinguished bad search queries, poor source selection, and tool failures behind the same complaint. The Professional decision is which observations make those causes distinguishable without requiring a fixed execution script or storing every conversation.

How it works

Preserve the observed run

A successful replay does not disprove a reported failure. The replay is another run, potentially using different retrieval results, prompt revisions, tool responses, and model behavior. Preserve the original run’s identifiers and configuration references so investigation starts from what happened to the affected request.

In the Agent SDK, beta tracing exposes claude_code.interaction, claude_code.llm_request, and claude_code.tool spans. Their structure distinguishes a turn, its individual model requests, and tool execution. Use that distinction when defining your diagnostic questions: did generation stall, did a dependency consume the time, or did the system finish quickly with inadequate support?

Do not make the expected trace an exact sequence of tool names. Anthropic’s research evaluations explicitly allow different valid paths toward the same goal. A path with three searches and another with ten might both succeed. Judge the result and whether the process respected constraints; investigate inefficient paths separately from incorrect ones.

Separate step success from task success

A model request can finish while the application still owes work. Messages API stop_reason: "tool_use" indicates a tool call, not completion of the user’s task. stop_reason: "end_turn" means generation finished naturally; it does not grade factual accuracy or business completeness. See 3.4.3 for monitoring those signals.

A tool can also return usable data that does not answer the question. Transport success cannot distinguish an irrelevant search result from a relevant one. Define an outcome criterion such as coverage of the requested jurisdictions or consistency with the retrieved policy. Anthropic’s evaluation guidance calls for specific, measurable success criteria rather than a general impression of answer quality.

This creates two observation planes. Execution observations explain what the system did: model requests, tool calls, timing, and errors. Outcome observations assess whether the work met the requirement: source support, completeness, or a verified downstream result. Keep both attached to the run so an outcome failure can be investigated through its execution path.

Account for fan-out and hidden branches

A parent duration alone cannot explain a request that delegates to several workers. Parallel branches overlap: adding their durations overstates elapsed latency, while looking only at the fastest branch understates the wait for completion. Examine the branch the parent actually waited for and the work it accepted into synthesis.

The SDK’s beta traces nest subagent model and tool spans under the parent’s launching claude_code.tool span. This makes an expensive or stalled branch visible even when the final response is brief. Instrumentation details belong to 3.4.2; here the architectural requirement is to retain branch attribution through aggregation.

Fan-out also changes the denominator. Four successful workers and one unsuccessful worker do not necessarily mean an 80% successful business request. If the missing worker owned the required exclusion check, the task might be unusable. Conversely, an optional failed branch need not invalidate a result with sufficient independent support. Record required coverage and accepted limitations as application outcomes, rather than averaging worker statuses into task success.

Treat missing telemetry as a separate failure

The Agent SDK exports telemetry through its Claude Code subprocess. Anthropic documents that export errors fail silently by default: an unreachable collector can drop telemetry while the agent continues operating. An empty error dashboard can therefore mean either healthy execution or a broken observation path.

Set CLAUDE_CODE_OTEL_DIAG_STDERR=1 and capture the SDK’s stderr callback in Python or stderr option in TypeScript to surface exporter diagnostics. This diagnostic option requires Claude Code v2.1.179 or later. Verify arrival at the collector as well as successful agent execution; 3.4.4 owns alert routing for that distinction.

Key distinctions and trade-offs

Reproducibility versus diagnosis: Replaying an input tests a further execution. Original-run observations explain the reported execution. Neither substitutes for the other.

Valid variation versus regression: Different tool paths are expected in a nondeterministic system. Missing required coverage, unsupported claims, or an unacceptable cost distribution are actionable outcomes.

Worker health versus user outcome: Branch status helps localize a problem. Task success depends on the business requirement and the coordinator’s use of branch results.

Detailed traces improve attribution but increase collection cost and exposure. Content-free decision patterns can still show repetitive searches, abandoned branches, and excess delegation; Anthropic describes using interaction structures without monitoring individual conversation contents. Choose observations that answer a diagnostic question, then apply the data controls in 3.4.5.

Architecture / decision scenario

A financial-services team generates analyst briefings with five research workers. Its internal target is 95% complete briefings within 60 seconds. In a reviewed sample, 99% of model requests succeed, but only 88% of briefings cover every required market. The engineering lead proposes increasing the API error alert threshold because the service looks stable.

The architect rejects that change as a response to the coverage defect. The architect retains transport alerts and adds a briefing-level coverage assessment tied to each run. Branch observations show that the exclusions worker frequently exhausts its search without supported findings, while synthesis still presents the briefing as complete.

The analyst owner defines acceptable coverage; the platform owner makes branch outcomes and configuration references observable. A replay that succeeds does not close the incident. The team compares the failing runs with acceptable runs and checks whether the repair improves briefing completeness without violating the 60-second target. Counting five worker completions would have concealed the same defect again.

Common exam traps

  • Treating HTTP success or end_turn as proof of a correct business answer.
  • Requiring every valid agent run to follow one fixed tool sequence.
  • Closing a reported incident because a replay succeeds.
  • Averaging worker success rates into request success without checking required coverage.
  • Adding parallel branch durations and calling the sum end-to-end latency.
  • Assuming absent telemetry proves an absence of failures.

What to remember

Observe the original run, its branches, and its business outcome together. Nondeterminism makes paths variable; it does not make correctness unmeasurable. Execution traces localize causes, outcome criteria detect failures that transport metrics miss, and export diagnostics establish whether the observations arrived.

Knowledge check

References