Isolate the layer behind a failure

Learning objective

Triage failures by layer — input, prompt, retrieval, model, tool, orchestration, output handling — using traces, repeated runs, and controlled component substitution.

Core idea

A wrong answer identifies a failed outcome, not its cause. The same symptom can arise because input handling dropped a condition, retrieval supplied the wrong policy, a prompt contradicted itself, or the application displayed only part of a correct response. Upgrading the model changes one component while leaving the other causes intact.

Treat each diagnosis as a testable claim about a layer. Preserve the failed run, replace one suspected component with a known-good counterpart, and compare repeated trials against the same acceptance rule. Anthropic’s evaluation guidance distinguishes the transcript from the outcome: what the agent said happened can differ from the resulting business state. Use both when deciding what failed.

How it works

Reconstruct the request that actually ran

Start from a specific production case with an expected result and an observed violation. Preserve the input before normalization and the request after assembly. For a Messages API integration, the replay record includes the supplied system, messages, tools, model, and max_tokens, plus other settings the integration sent. Capture returned content and stop_reason, not only the text shown to the customer.

Record retrieved passages and tool results as supplied, with their source revisions. A document identifier alone is insufficient if its content changes before the replay. Preserve the orchestration path as well: which agent received which task, which results reached its parent, and which output the application accepted. See 3.4.2 for trace lineage and 3.4.5 for deciding how much sensitive content to retain.

Anthropic defines a transcript as the complete record of a trial, including API calls, responses, tool calls, and intermediate results. Use that record to compare the model’s input with the application’s intended input. An agent’s retrospective explanation is useful context but does not replace the actual messages.

Locate the first violated stage expectation

Inspect stages in execution order. If the question asks about a named subsidiary but the assembled request contains only the parent company, the input stage already failed. A later hallucinated subsidiary policy is a consequence; changing generation may conceal the missing qualifier without restoring it.

Use the following substitutions to test plausible causes. These are diagnostic experiments in your harness, not Anthropic API options:

A substitution narrows the cause; it does not automatically identify the faulty line of code. See 4.4.3 for separating retrieval from request-assembly faults after an applicable-passage substitution.

Repeat the comparison without changing the question

Anthropic recommends multiple trials because outputs vary between runs. Run the baseline and substituted configuration on the same cases, using the same grader and trial count. Report how often the violation occurs in each configuration. A single passing replay cannot establish that an intermittent defect disappeared.

Freeze mutable tool state or use a resettable test environment. Otherwise, the second run might pass because the first changed a record, not because the substitution helped. Read-only recorded results can isolate inference behavior; a separate execution test checks whether the live tool produces those results correctly.

Do not demand one incidental trajectory when several valid paths satisfy the task. Grade required outcomes and inspect the transcript for the suspected defect. A correct final answer reached through a forbidden action remains a failure, while a different permitted tool order can be acceptable.

Separate cause repair from containment

Check stop_reason before interpreting missing text as weak reasoning. A response ending with max_tokens reached the request’s output limit; the documented remedies are raising max_tokens or continuing the response. An application that published it as complete also has an output-handling defect. Both responsibilities can appear in the same incident.

Containment can precede certainty: hold affected drafts for review while diagnosing them. The permanent change should still follow the isolated cause. Validate the repair on failing cases and previously passing cases so that removing one symptom does not introduce a different failure.

Key distinctions and trade-offs

Correlation versus substitution: A regression starting on deployment day suggests where to investigate. A controlled comparison tests whether the changed component contributes when the other inputs remain fixed.

Trace versus outcome: The transcript shows the operation requested and response returned. The resulting business state establishes whether the promised operation actually succeeded.

Local repair versus end-to-end repair: A correct standalone model response does not clear the full system. Reconnect the repaired component and verify the delivered result, including parsing and downstream effects.

Detailed replay records improve diagnosis but carry storage and access costs. Keep the smallest authorized record that preserves the deciding conditions; a redacted record that removes those conditions cannot support the claimed replay.

Architecture / decision scenario

A financial-services firm drafts internal credit-policy answers within a 12-second p95 response target. Measured p95 latency is nine seconds. After a release, 18 of 100 sampled questions fail review. The product lead proposes sending every question to a more capable model, raising estimated p95 latency by four seconds to 13 seconds. These are illustrative workload measurements, not vendor performance figures.

The architect instead reconstructs the seven stages for each failure. Three inputs lost a subsidiary qualifier. Six requests received superseded passages. Four prompts contained a conflicting exception rule. Two tool lookups returned timeouts as empty records. One parent agent omitted a completed specialist result. Two raw answers were correct but the renderer cut their final conditions. No model-only cause is established by that distribution.

The team runs five trials per case with one substitution at a time, keeping a fixed acceptance rubric. Verified original inputs repair the first group; applicable passages repair the second; the previous prompt repairs the third. Approved tool results and a complete handoff isolate the next two groups. Raw-response comparison identifies the renderer defect without another inference call.

The architect rejects the blanket model upgrade because it exceeds the latency target and does not address the observed stage failures. Input, retrieval, prompt, tool, orchestration, and presentation owners receive distinct repairs. The model remains a candidate cause if failures survive those repairs; the end-to-end replay and a passing-case sample determine whether the release can resume.

Common exam traps

• Treating every incorrect answer as a model-capability problem.
• Replaying with live documents while calling the inputs frozen.
• Changing prompt, retrieval, and model together, then claiming one caused the improvement.
• Accepting one successful replay as proof that a variable failure is fixed.
• Reading only the final answer when the failure occurred in a tool or handoff.
• Confusing a truncated raw response with text lost by the application.

What to remember

Reconstruct what ran, identify the first broken stage expectation, and test one layer at a time. Repeated trials distinguish a real improvement from run variability. Close the incident only after the delivered outcome passes with the repaired component reconnected.

Knowledge check

References

• Anthropic, Demystifying evals for AI agents
• Anthropic, Stop reasons and fallback
• Anthropic, Handle tool calls
• Anthropic, Create a Message