Keep investigation separate from production remediation

Learning objective

Use Claude for incident investigation under read-only access and gate remediation behind human approval and safety hooks.

Core idea

An incident assistant should accelerate questions without acquiring the incident commander’s authority. Connect it to authorized diagnostic data, require a check for each proposed cause, and make remediation a separate operation with explicit approval. A sentence saying “investigate only” is useful task context, but the access boundary must survive a mistaken or injected instruction.

Assume you already know how PreToolUse denies a call and how bounded code exploration works. Here the architect decides how diagnostic credentials, operational tools, human review, and hook failure behavior fit into an incident workflow. See 7.3.3 for isolating the faulty component.

How it works

Make the diagnostic path read-only at the service

Choose credentials whose backend permissions permit only the queries needed for investigation. A monitoring server might expose application-defined mcp__ops_read__latency and mcp__ops_read__errors tools, returning redacted summaries for an incident’s time range. Those names illustrate a team’s MCP integration; they are not Anthropic-provided monitoring operations. For services with established CLIs, a CLI on a read-only profile is a practical alternative: Anthropic describes CLI tools as the most context-efficient access route.

Do not place an operator’s unrestricted production credentials in the same session and rely on the query tool’s description to keep them unused. A shell or alternate connector could provide another route to a write. Review all available paths, not only the intended MCP tools. Backend authorization is the final control for the resources it protects.

Claude Code warns when MCP output exceeds 10,000 tokens, a fixed threshold, and sets a default maximum of 25,000 tokens. MAX_MCP_OUTPUT_TOKENS changes the token limit for tools without anthropic/maxResultSizeChars; tools declaring that annotation use their own text limit, while image results still obey the token setting. For incident queries, return curated summaries rather than raising the cap to accommodate raw log dumps.

Claude Code permissions provide an additional layer. Use narrow permissions.allow entries for the approved query tools and permissions.deny for prohibited tools. Bash rules match command text rather than every possible invocation of a program. A deny for one deployment command does not prove that every equivalent deployment route is blocked. A mod handling tool.check can even approve a denied call unless managed settings or a Team or Enterprise login preserve deny rules by default; organizational policy can change that default. Review mods and policy alongside the rules. See 7.1.3 for permission and sandbox selection.

Operational records are untrusted input. A log entry can contain attacker-controlled text, and Anthropic warns that MCP servers fetching external content can introduce prompt injection. Query scope and redaction reduce irrelevant or sensitive material. Restricted capabilities limit what a malicious instruction can cause; telling Claude to ignore suspicious lines does not replace them.

Ask for discriminating checks

Give Claude the observed symptom, timestamps, reproduction steps where available, affected service, relevant release changes, and the criterion for recovery. Anthropic’s best practices recommend giving Claude a check it can run and asking it to show the output rather than assert success. Apply that to an incident hypothesis.

Ask for a small set of candidate causes, each with a predicted observation and a permitted query or nonproduction test that could refute it. Keep unavailable data explicit. A plausible cause is still unconfirmed when the assistant cannot access the decisive trace. A focused investigation uses fewer queries and exposes what the next human decision depends on.

Claude Code plan mode runs shell commands to explore: when auto mode is available and useAutoModeDuringPlan is on, as it is by default, the classifier reviews planning shell commands other than critical-path removals and approved commands run without prompting. When auto mode is unavailable or that setting is off, commands outside the built-in read-only set prompt; in interactive terminals with bypass permissions available, planning commands bypass both classifier and prompt, subject to the documented exceptions. Keep backend read-only access independent of this workflow choice.

Put human approval on the actual remediation call

The safest separation leaves remediation in the established deployment system: Claude proposes the action and its supporting observations; the authorized operator reviews and executes it. The investigation session remains read-only throughout. Approval of a diagnosis is not approval of a rollback, restart, or customer message.

If the team deliberately exposes a narrow remediation tool, keep its backend credential narrowly scoped. An ordinary permissions.ask: ["mcp__ops_change__rollback_release"] rule requests confirmation, but a PermissionRequest hook can answer the prompt in modes that ask, and a mod’s tool.check can approve an ask-rule call. An ask rule alone therefore does not establish that a person approved it.

For explicit approval on every call, set _meta["anthropic/requiresUserInteraction"]: true on the remediation tool’s tools/list entry. Claude Code prompts even in acceptEdits, auto, and bypassPermissions; allow rules do not skip it, and dontAsk denies the call. In Claude Code v2.1.292 or later, a mod’s tool.check approval cannot skip this prompt either. --permission-prompt-tool cannot approve a flagged call; the Agent SDK’s canUseTool callback can, so the SDK application must actually obtain human approval rather than automatically returning allow.

Before approval, present the service, target revision, expected effect, possible collateral impact, verification query, and rollback alternative. These are application review requirements. Tie the recorded approval to those exact parameters so a different target does not inherit approval from a general “fix it” instruction.

Make the safety check survive its own failure

Use the synchronous PreToolUse denial and onFailure: "block" pattern from 3.2.5 to check the incident and approved target before execution. Rehearse a missing policy script, an unavailable approval lookup, and a target mismatch: each must leave the production operation unexecuted. For protection against a mod overriding a non-managed hook block, put the enforcing hook in managed settings and verify that deployed configuration. A valid policy check must still preserve the tool’s required human approval.

A background hook cannot supply a synchronous safety decision. Do not set async: true on the check, and do not assume a command-hook timeout behaves like an Agent SDK callback timeout. The reference documents different timeout behavior for those families. Verify the deployed family and version before relying on the result.

After the approved action, query actual service health and record whether the recovery criterion passed. A successful tool invocation establishes execution, not restoration of service. If health remains degraded, return to the incident commander with the observed result rather than chaining further changes under the first approval.

A remediation call reaches execution only through a valid check and the required human approval, and execution is not yet restoration of service.

Key distinctions and trade-offs

A separate human execution path reduces the assistant’s mutation surface but adds a handoff. A narrowly exposed remediation tool can shorten execution time, but requires maintained approval controls, backend scope, hook failure tests, and parameter-specific records. Choose from the incident’s operational needs, not from a desire to eliminate a confirmation click.

Preventive controls and audit records also have different jobs. A query log helps reconstruct what was examined. It cannot prevent a production mutation after execution. Put enforcement before the effect, then use the record to explain the decision afterward.

Architecture / decision scenario

A healthcare scheduling service experiences a tenfold rise in failed appointment confirmations. Its incident commander has a fifteen-minute mitigation target and prohibits sending patient identifiers to the assistant. One engineer proposes connecting a production administrator CLI so Claude can diagnose and restart immediately.

The service architect instead provides two redacted query tools on backend read-only credentials. Claude compares dependency error counts and release timestamps, then confirms the implicated release through a nonproduction reproduction. The commander reviews a rollback proposal with its target revision and recovery query; the normal deployment system remains the preferred execution route.

For a team-approved emergency rollback tool, the architect requires anthropic/requiresUserInteraction: true and a synchronous managed PreToolUse check. A rehearsal with the policy script missing must block under onFailure: "block"; an unattended dontAsk session must deny rather than wait for an invisible approval. The commander approves one target, the operator verifies confirmation failures fall from 12% to below the scenario’s 1% recovery threshold, and any further change needs a separate decision.

Common exam traps

• Treating “read-only” in a prompt or plan-mode label as backend authorization.
• Giving the assistant unrestricted credentials for diagnostic convenience.
• Treating general incident approval as approval of every subsequent write.
• Assuming a crashed command hook blocks by default.
• Raising MCP output limits instead of narrowing a diagnostic query.
• Treating an ask rule as proof that a human answered the prompt.
• Equating successful rollback execution with verified recovery.

What to remember

Enforce diagnostic access at the backend, keep remediation separate, and bind human approval to a specific action. Safety hooks must block reliably when their own checks fail. Verify service recovery after execution and retain the incident commander’s authority over the next step.

Knowledge check

References

• Anthropic, Connect Claude Code to tools via MCP
• Anthropic, Configure permissions
• Anthropic, Choose a permission mode
• Anthropic, Hooks reference
• Anthropic, Best practices for Claude Code