Recover failed branches without corrupting completion

Learning objective

Contain agent failures with partial-result recovery and escalation so one failed agent neither halts nor silently corrupts the overall task.

Core idea

For an architect operating a multi-agent service, a failure boundary must preserve useful independent work and prevent unsupported dependent work from passing acceptance. Structured failure reports and visible coverage gaps are assumed here. The professional addition is deciding where recovery stops, which results remain valid, and who accepts a narrowed outcome.

Anthropic’s research system combines model adaptation with deterministic retries and regular checkpoints, resuming work instead of restarting everything. Apply that pattern at the chosen topology’s acceptance boundaries. A worker stopping is an execution event; successful completion of the business task requires separately verified coverage and evidence.

How it works

Recognize an interrupted worker

Recall that foreground API interruptions return partial output with an unfinished-task note or an error, while background failures retain the error and last output.

Wire the background-task adapter to SDKTaskNotificationMessage in TypeScript or TaskNotificationMessage in Python: type: "system", subtype: "task_notification", and status: "completed", "failed", or "stopped". Associate task_id with the ledger assignment and retain summary and output_file for result inspection. A "completed" runtime status still needs recipient acceptance; "failed" and "stopped" must not become successful reports because output text exists. Keep foreground interruptions alongside supported findings and unresolved work too.

A SubagentStop hook can record subagent completion in the same ledger, using its session_id and agent_id to identify the worker. Completion events and evidence acceptance remain separate records; neither a hook nor a fluent final paragraph establishes that the task’s obligations were satisfied.

A turn limit is another interruption. AgentDefinition.maxTurns stops the worker; Claude Code v2.1.246 or later marks that output partial. Increase the limit only if the remaining investigation fits the task’s time and spend envelope.

Persist accepted work at branch boundaries

Checkpoint accepted evidence and the assignment it covers, rather than saving only the coordinator’s summary. Anthropic describes regular checkpoints and durable artifacts that persist outside the lead’s context. Your application can maintain a task ledger linking each assigned obligation to its accepted evidence, incomplete operations, and recovery decision.

These records are application-defined. For example, separate worker_state from acceptance_state and associate the result with task_id and attempt_id. A stopped attempt can contain accepted findings without having satisfied the entire assignment. Keep the rejected or unfinished remainder explicit so a later synthesis cannot mistake a saved artifact for full completion.

Acceptance also limits the blast radius of a bad result. If synthesis already used an unsupported branch claim, correcting that branch requires rechecking the affected conclusions. Keeping unrelated evidence does not justify retaining conclusions derived from rejected evidence. Use the evidence relationships from 1.4.3 to identify what must be reconsidered.

Resume the branch, not the entire investigation

For SDK recovery, a resumed subagent retains its conversation history. Capture session_id and the agentId returned in the Agent tool result, then start the next query with resume: sessionId and name the agent ID in its prompt. Resume the same session and pass the same custom definition in agents. The built-in Explore and Plan agents are one-shot and do not return an agentId, so choose a custom agent or general-purpose if this recovery path is required.

Resume after the underlying API error clears, or continue a turn-limited investigation with a precise remaining assignment. If the failed operation cannot succeed unchanged, use the worker’s concrete recovery alternatives to change its scope or source. A fresh worker with the same impossible assignment adds cost without changing the cause.

Do not confuse session history with an accepted checkpoint. Resumption restores previous calls and results; your application still checks which findings are valid and what remains outstanding. Supply the recovery decision so the worker does not repeat already accepted work or broaden its task to compensate for failure.

Bound amplification and assign escalation

A failure can turn a small graph into many replacement workers. Recall that CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS bounds Agent-tool spawns with documented exemptions; resuming a completed subagent takes a fresh slot without checking that limit. Recovery that resumes several workers can therefore push running work beyond the configured bound. Track active recoveries in the application ledger and admit them only within the service’s remaining concurrency allowance, rather than relying on the spawn limit to reject every resume.

For the SDK’s documented depth, concurrency, and spend configuration, verify TypeScript v0.3.219 or Python v0.2.127 onward, bundling Claude Code v2.1.219 or later. The spend stopping threshold is described in 1.4.1.

Each resumed query() needs an explicit remaining-budget calculation. maxBudgetUsd in TypeScript and max_budget_usd in Python count only that call’s own spend; restored session totals do not count against its cap. Set the resumed call’s cap from the business allowance minus the preceding cumulative total_cost_usd, leaving the overrun headroom described in 1.4.1. If no allowance remains, escalate instead of funding another call.

So resuming an interrupted worker passes through a remaining-budget calculation, which escalates instead when no allowance remains.

A resumed call’s total_cost_usd includes earlier session spend. Without conversation resets, read the returned cumulative total or calculate its increase since the previous result; summing the original and resumed totals double-counts earlier work. Session continuity preserves the transcript, not the original call’s budget enforcement.

Assign an escalation recipient before deployment. A source-access defect goes to the owner of that adapter or access configuration; an unresolved evidence requirement goes to the decision owner who can authorize a limited deliverable. Give that recipient the affected obligation, observed failure, accepted artifact references, attempted recovery, and remaining constraint. These application records implement the escalation contract; a worker’s suggestion does not grant authority to weaken acceptance.

Key distinctions and trade-offs

Containment follows the dependency graph:

Recovery and delivery are separate choices. A time-sensitive service can deliver supported portions with precise limitations when its acceptance policy permits them. If the missing evidence is required for the central decision, withhold that decision while preserving the independent work. Neither an unconditional full restart nor an unconditional partial success is a defensible default.

Architecture / decision scenario

A government procurement team needs a comparison of four proposed services by noon. The assistant may deliver a limited evidence briefing, but its contract forbids a complete ranking without every mandatory criterion. One specialist returns supported availability findings; another fails while retrieving a required access-control document.

The coordinator stores the accepted availability evidence and records the access-control assignment as unfinished. The adapter retains the Agent failure signal and partial output. It does not translate the interruption into an empty successful review or infer that the missing document contains no restrictions.

After the source recovers, the service resumes the custom reviewer in the same SDK session with its agent ID, definition, and remaining criterion. If the remaining budget or time prevents completion, it escalates to the procurement decision owner with the accepted artifacts, attempts, and blocked ranking. The noon briefing can contain supported findings and the exact gap, while the ranking remains withheld.

If a branch lead had already summarized the missing criterion as satisfied, the root rejects that branch conclusion and rechecks its dependent comparison. It keeps unrelated availability evidence. In a peer-messaging design, the same rejection must reach any peer whose recommendation consumed that claim; otherwise recovery of the original worker leaves corrupted conclusions elsewhere in the graph.

Common exam traps

• Treating partial Agent output or a stopped worker as accepted completion.
• Restarting successful branches because an independent worker failed.
• Resuming in a different SDK session and expecting the worker transcript to be available.
• Planning resumable recovery around one-shot Explore or Plan agents.
• Assuming the concurrency setting bounds exempt or resumed execution paths.
• Repairing one worker while retaining downstream conclusions based on its rejected evidence.
• Reusing the original spend cap on resume or summing overlapping session cost totals.

What to remember

Checkpoint accepted evidence, preserve interruption signals, and recover only the affected obligation within the remaining resource envelope. Follow dependencies when deciding which conclusions remain valid. Escalate acceptance decisions to a named owner and keep required unresolved conclusions visibly withheld.

Knowledge check

References

• Anthropic, How we built our multi-agent research system
• Anthropic, Subagents in the SDK
• Anthropic, Create custom subagents
• Anthropic, Track cost and usage
• Anthropic, TypeScript SDK reference
• Anthropic, Python SDK reference
• Anthropic, Control execution with hooks