Locate the first broken agent transition

Learning objective

Diagnose tool and agent failures: wrong tool selection, malformed inputs, unhandled errors, and loops.

Core idea

Tool clarity, actionable error results, and protocol-based termination are established controls. Production diagnosis adds a different task: locate which transition failed before deciding whether to repair a tool definition, executor, adapter, or agent controller.

Inspect the requested operation, its inputs, the actual execution outcome, the result Claude received, and the next decision as separate observations. Anthropic’s evaluation guidance recommends examining transcripts as well as outcomes. A plausible final answer can conceal a wrong tool, a swallowed failure, or repeated work that happened to end successfully.

How it works

Build a call-level timeline

For each Messages API client-tool call, correlate the tool_use block’s id, name, and input with execution diagnostics and the returned tool_result.tool_use_id. Preserve the exposed tools definitions and the request’s tool-selection settings. The question is what interface and state Claude had when it made the choice, not which definition exists after someone repaired it.

Add the receiving agent and parent handoff to the timeline. A specialist can complete its task while the parent never receives the result. Conversely, a parent can faithfully repeat an incorrect specialist result. Compare the actual handoff content at that boundary before diagnosing the parent’s reasoning.

In the Agent SDK, AssistantMessage contains model content and UserMessage includes tool execution results. The final ResultMessage.subtype identifies how the loop ended. Use those messages for the SDK-managed timeline rather than assuming an application-owned Messages loop and an SDK-managed loop expose identical control points.

Separate selection, shape, and meaning

A wrong operation is not the same as an invalid input. Compare the task with the chosen tool_use.name and the tool’s documented responsibility. If the required operation was absent from the exposed tools, investigate capability delivery. If it was present but indistinguishable from another tool, inspect names and description values. If both are clear, inspect competing instructions or the agent’s supplied task.

For the correct operation, compare input against input_schema, then against the operation’s business semantics. A string can satisfy a schema while naming the wrong customer’s account. A syntactically valid date range can still violate the service’s supported interval. Report structural and semantic input failures separately so that schema changes do not appear to solve wrong-resource selection. On a tool definition, strict: true uses grammar-constrained sampling to guarantee inputs conform to the supported JSON Schema subset, subject to the structured-outputs JSON Schema limitations. It addresses structural input failures, but a schema-valid wrong customer account still needs semantic validation.

Anthropic’s Define tools page documents description and input_schema; its Handle tool calls page recommends more detailed descriptions when missing information produces invalid calls. The diagnostic experiment is to change that information while retaining the same task and backend, then compare actual calls. See 3.1 for tool-interface design; this lesson tests which boundary needs it.

Compare the backend outcome with the delivered result

When inputs are valid, inspect execution before blaming the next model response. A backend timeout converted into an empty successful result teaches the agent the wrong state. A useful error produced by the backend but discarded by an adapter creates a different defect at the delivery boundary.

For Messages API client tools, the documented failure signal is tool_result.is_error: true, with actionable content. Compare that supplied block with the actual dependency outcome. Do not diagnose from a server log alone: the log can contain an accurate error that Claude never received.

A useful backend error reaches Claude only when the adapter delivers it as an error result.

Check history integrity separately from execution. Anthropic requires the result message to immediately follow the corresponding tool-use message, with result blocks before any text in that user message. A 400 reporting “tool_use ids were found without tool_result blocks immediately after” points to assembly or correlation, not a request to make the tool description more persuasive. Repair the message history and correlation first.

Use an approved recorded tool result to isolate downstream behavior. If the agent completes correctly with that result but not the live one, the execution or result path is implicated. If it fails with both, inspect how the task and supplied result constrain the next decision. Replacing the tool and prompt together would lose this distinction.

Diagnose loops by what changes between calls

A high call count alone does not prove a loop. Some tasks legitimately visit many resources. Compare repeated call signatures and whether each result changes the agent’s information or the environment. Repeated identical reads after an unchanged permission denial differ from successive reads of different required records.

Use the timeline to distinguish these patterns:

In the Agent SDK, ResultMessage.subtype values error_max_turns and error_max_budget_usd are containment signals, not explanations of the repeated behavior. Preserve the preceding messages and diagnose why progress stopped.

Key distinctions and trade-offs

Wrong selection versus wrong arguments: Both can lead to the same backend error. The selected responsibility and actual input determine which interface failed.

Execution error versus error-delivery defect: A dependency can fail honestly while the adapter misrepresents it. The latter often produces misleading downstream reasoning.

Long trajectory versus loop: Count progress toward required outcomes, not only calls. A cap contains exposure but cannot identify the root cause.

Recorded results make comparisons repeatable but do not test the live backend. Finish by testing the repaired integration and its final business state. For tools that write, use a resettable environment so diagnostic repetitions do not become repeated production actions.

Architecture / decision scenario

A government case-management team has three engineers and requires every change of case status to remain reviewable. Staff report wrong office lookups, malformed date inputs, invented “no record” answers, and repeated requests. The delivery lead proposes replacing the specialist agents with a larger model and allowing twice as many turns.

The architect reconstructs call-level timelines. Office lookups choose an archive tool whose description overlaps the live register. Date failures occur on the correct tool but pass a display date where the interface expects its documented format. The “no record” answers follow backend timeouts that an adapter replaced with empty success content. Repeated lookups occur because the parent drops the specialist’s completed result before delegating again.

Controlled tests repair one boundary at a time: distinguish exposed descriptions, clarify the date parameter, preserve is_error: true and actionable error content, and deliver the specialist result intact. The team compares both actual calls and the resulting case state on frozen cases, then tests the live integration in a resettable environment.

The architect rejects the blanket model-and-turn increase: it consumes the small team’s operating budget while leaving four observed defects. Tool owners maintain selection and parameter tests; the adapter owner tests failure delivery; the orchestration owner verifies that completed specialist work reaches the parent. SDK limit outcomes remain incomplete runs requiring an owned recovery decision.

Common exam traps

• Treating every backend error as wrong-tool selection.
• Calling schema-valid inputs semantically correct.
• Assuming an error in server logs was visible to the agent.
• Repairing descriptions when message correlation caused a 400.
• Raising turn limits without inspecting repeated calls and missing handoffs.
• Treating a limit subtype as the root cause or as successful completion.

What to remember

Find the first broken transition from selection through execution, result delivery, and the next decision. Use correlated calls and actual outcomes to assign the repair. Limits contain loops; retained transcripts explain them.

Knowledge check

References

• Anthropic, Define tools
• Anthropic, Handle tool calls
• Anthropic, Agent loop
• Anthropic, Demystifying evals for AI agents
• Anthropic, Strict tool use