Audit every reachable capability by role

Learning objective

Audit tool and agent configurations for unused, overlapping, ambiguous, or overly powerful capabilities and their cost in selection accuracy, context, and attack surface.

Core idea

A role-specific menu and clear tool descriptions are the starting point. A production audit goes further: it connects each reachable operation to a business need, an owner, observed behavior, context cost, and the damage possible if selection fails.

Capability bloat is excess reachable authority or decision complexity, not a particular tool count. A single administrative wrapper can be more dangerous than twenty distinct read operations. Your deliverable is a disposition for each capability—retain, remove, narrow, merge, or investigate—with enough supporting observations for the service owner to accept the change.

How it works

Inventory the effective configuration

Capture what each role actually receives on each route, including exceptional and recovery routes. For Messages API user-defined tools, inspect the assembled tools entries, including name, description, and input_schema. A registry listing alone misses the application code that selects entries for a particular request.

For Agent SDK specialists, record effective AgentDefinition.tools, including inherited capabilities when the list is omitted, alongside the parent’s configuration. Include the specialists the coordinator can delegate to: removing a direct write tool does not narrow the workflow if another reachable agent still performs that write on request.

Use an audit worksheet with these columns:

The worksheet is a team artifact, not an Anthropic API object. Populate its configuration columns from the deployed request builder or SDK definitions, then attach transcript observations to those entries.

Distinguish four findings

An unused capability has no observed calls in the sampled work. That is a candidate for removal, not sufficient proof. Ask the operation owner about annual processing, incident recovery, and workflows missing from the sample. Record a specific retained use case when keeping an idle capability; “might be useful” is not an operational requirement.

Overlapping capabilities implement substantially the same workflow. Anthropic recommends distinct purposes and selective consolidation around useful workflows. Decide whether the duplicate is an alias to retire or whether its separate resource scope matters. Combining read and write authority into a generic operation to reduce the entry count makes the audit result worse.

An ambiguous pair has a distinction the interface fails to communicate. Compare name, description, and parameter descriptions against actual tool_use.name and tool_use.input in failed cases. Count wrong-operation choices separately from invalid arguments and implementation errors. A frequently selected tool can be popular because it attracts the wrong requests.

An overly powerful capability reaches resources or side effects beyond the role’s assignment. Inspect its implementation as well as its schema: a free-form command or arbitrary destination can hide many operations behind one name. Anthropic’s injection guidance recommends limiting sensitive data and actions through least privilege. Assess what an instruction inside a retrieved document could cause if the agent followed it.

Measure separate costs

Measure selection with labeled requests and raw calls, not only satisfactory final answers. Anthropic’s tool-engineering guidance recommends task accuracy, runtime, call counts, token consumption, and tool errors, plus transcript inspection. Report mistakes by role and capability so a high-volume read workflow cannot conceal a dangerous write-selection problem.

For ordinary client-tool requests, compare the same request with alternative tools arrays using POST /v1/messages/count_tokens. Its input_tokens is an estimate. Hold the model, messages, and system prompt fixed so the difference estimates the configuration’s contribution. The endpoint rejects tool search and other server tools except the advisor tool; use actual Messages response usage for those configurations instead.

Treat context load and billing as separate observations. A smaller bill does not establish fewer reachable actions, and a lower tool count does not establish lower risk. The attack-surface entry records which sensitive data or actions remain reachable and through which paths; it is not a token metric.

Key distinctions and trade-offs

Observed use versus justified need: Traces describe the sampled workload. Owners establish whether legitimate but rare workflows still require a capability. Retain a narrow exception with a named owner rather than restoring a whole server’s operations.

Selection difficulty versus privilege: Descriptions and namespacing improve distinguishability. Removal and narrower implementations reduce available authority. One repair can help both, but neither metric proves the other.

Tool inventory versus agent inventory: A specialist can move risky operations out of the parent’s context while leaving them reachable through delegation. Review the full callable route before claiming removal; see 3.1.2 for the redesign decision.

Architecture / decision scenario

A financial-services dispute assistant must prepare cases within a four-minute service target. Its 44-entry configuration includes duplicate statement retrieval, an annual adjustment operation, broad account administration, and three specialists. In a hypothetical 600-case audit, 24 requests select the wrong retrieval operation; account administration receives no legitimate dispute calls. One specialist can still invoke it.

The platform lead proposes caching the unchanged configuration and adding logs. The architect rejects that as the complete remedy: neither proposal removes excess authority or explains the 24 selection failures. The audit assigns duplicate retrieval to interface consolidation, account administration and its delegation route to removal, and the idle annual operation to its finance owner for confirmation of need.

The service owner accepts the resulting disposition only with role-level selection results, comparable token measurements, and a list of remaining write paths. The annual operation stays exclusively in the year-end role after its owner identifies that workflow. This avoids deleting a rare requirement while leaving ordinary dispute processing narrower. The redesigned set then enters the validation described in 3.1.4.

Common exam traps

• Deleting every zero-call tool without checking seasonal work.
• Treating a frequently called tool as necessary without examining misroutes.
• Auditing declarations while ignoring inherited tools and delegated writes.
• Replacing overlapping reads with a more powerful administrative wrapper.
• Treating reduced token cost as proof of reduced authority.
• Scoring only final answers and missing rejected or redundant calls.

What to remember

Audit the effective configuration per role and route. Connect each capability to a legitimate task, owner, observed calls, context cost, and possible damage. Separate unused, overlapping, ambiguous, and overly powerful findings because each leads to a different change.

Knowledge check

References

• Anthropic, Define tools
• Anthropic, Subagents in the SDK
• Anthropic, Writing effective tools for agents
• Anthropic, Token counting
• Anthropic, Mitigate jailbreaks and prompt injections