Measure discovery before deferring the catalog

Learning objective

Remediate large tool catalogs with on-demand tool search, measuring discovery recall, selection quality, and context savings.

Core idea

After removing unnecessary capabilities, a legitimate role can still need a large catalog. Tool search loads relevant definitions on demand instead of placing the entire catalog in Claude’s context at the start. It adds a discovery decision before the operation-selection decision.

A tool cannot be selected correctly if discovery never makes it available. Evaluate these stages separately: whether search finds the required operations, whether Claude chooses correctly among loaded operations, and whether the completed workflow uses less context at an acceptable latency. Deferral manages context; removal establishes the role boundary in 3.1.2.

How it works

Configure the authorized catalog

On the Claude API, include a search tool in tools and send complete definitions for every discoverable client tool. Set defer_loading: true on definitions that should load only after discovery. Keep the search tool non-deferred; deferring every tool produces a 400 error. Anthropic recommends keeping the three to five most frequently used tools non-deferred to avoid a search on common requests.

Choose between two server-side variants:

Both search tool names, descriptions, argument names, and argument descriptions. The regex variant matches case-insensitively. Clear domain names and descriptions therefore matter to discovery as well as selection. Use service or resource prefixes and business vocabulary.

For MCP-sourced tools, configure defer_loading on the mcp_toolset entry’s default_config, or per tool in configs, rather than on individual client definitions. Retain the enabled allowlist from 3.1.2: deferral changes loading, not privilege. This connector is beta on the Claude API, Claude Platform on AWS, and Microsoft Foundry, requires anthropic-beta: mcp-client-2025-11-20, and is not ZDR-eligible.

Each search returns up to five matching tools by default. Claude can supply a limit from 1 to 10,000, and a request can contain up to 10,000 deferred tools. These limits do not guarantee accurate selection from large result sets.

Respect the response and support boundaries

Claude calls the built-in search through a server_tool_use block. Its tool_search_tool_result contains a tool_search_tool_search_result with tool_references; each tool_reference.tool_name identifies a discovered tool. The API expands those references into definitions before Claude selects an operation. Your application does not expand them itself.

Preserve the assistant content, including those server blocks, unchanged in subsequent history and continue supplying the same authorized tools array. Execute the discovered client tool_use and return its normal tool_result. Do not return a result for the search call’s srvtoolu_... identifier: the server executes that operation, and the API rejects such a client result. Discovered definitions can be reused from history without another search.

The server executes the search and the API expands its references, so your application executes only the discovered client tool_use.

Scope deployment explicitly. The tool-search page lists Claude Sonnet 5.5 (claude-sonnet-5-5) among supported models and excludes Claude Opus 4.1 and earlier models. On Amazon Bedrock, server-side tool search works through InvokeModel, not Converse. This lesson’s measurements assume the Claude API; validate the chosen model and platform before treating a configuration as portable.

When built-in search misses business vocabulary, evaluate custom client-side search, such as embedding-based retrieval. Return a standard tool_result with tool_reference blocks in content; each block names a tool_name whose complete definition remains in top-level tools, normally with defer_loading: true. This trades a different retrieval method for ownership of its implementation and evaluation; use it where that measured benefit justifies the work.

Measure discovery recall before selection quality

Create representative requests with labeled required tool sets. For a request needing three deferred operations, discovering two gives recall of two-thirds. Across the suite, report both per-request recall and the fraction of tasks for which all required deferred tools become available. The latter exposes workflows that fail when even one dependency is missing. These are evaluation definitions you own, not API metrics.

Read tool_references to measure discovery. A required tool kept non-deferred is already available and should not inflate deferred-tool recall. Include tasks whose wording differs from tool names, rare operations, neighboring domains, and multi-operation requests. Inspect empty tool_references as a discovery miss when a relevant authorized operation exists; an empty result is not itself a server error.

Then inspect actual tool_use.name and arguments. Report wrong-operation rate among tasks with successful discovery separately from end-to-end task success. High recall with poor selection suggests ambiguous loaded definitions or too many irrelevant matches. Low recall with good conditional selection suggests a discovery problem. A final answer score alone cannot tell these apart.

For regex misses, Anthropic recommends checking names and descriptions, testing the pattern against searchable text, and adding common keywords. Increasing the search limit can recover omitted matches but also loads more competing definitions. Compare the variants and candidate descriptions on the same labeled suite rather than assuming one search method is universally better.

Measure savings without hiding search overhead

Compare eager loading and deferred loading with the same authorized catalog, model, tasks, and cache conditions. Record Messages usage, total runtime, search calls observed in content, loaded references, client calls, and accepted outcomes. Tool search is not separately metered in usage.server_tool_use; count its server_tool_use blocks instead. Loaded definitions count as input tokens.

Deferred definitions are excluded from the system-prompt prefix; discovery appends inline tool_reference blocks, preserving that prefix for prompt caching. A deferred tool cannot also carry cache_control: that combination returns a 400. Put the cache breakpoint on a non-deferred tool.

Use actual Messages calls for this comparison. The token-counting endpoint rejects tool search, and its estimates cannot substitute for runtime usage. Include every request in a multi-turn workflow, not only the smaller first request. Otherwise, loading many definitions later can make a nominal saving disappear. For total input context, sum usage.input_tokens, usage.cache_creation_input_tokens, and usage.cache_read_input_tokens. Report those components separately so a cache-state change does not masquerade as a deferral improvement.

Anthropic suggests tool search for ten or more tools, definitions above 10,000 tokens, or degraded selection. Small catalogs and requests using every tool can favor eager loading; measure the trade-off.

Key distinctions and trade-offs

Discovery recall versus selection quality: Search exposes candidates; Claude chooses an operation. Diagnose the first failing stage before rewriting everything.

Context saved versus request bytes: Deferred definitions still travel in tools on every request. Deferral changes what loads into model context, not whether the definitions must be supplied.

Recall versus loaded noise: A larger limit can find more required tools and load more irrelevant ones. Accept it only when complete-task results justify the added context and selection burden.

Architecture / decision scenario

A retailer’s returns specialist legitimately needs 72 operations across six product categories. Its business requirement is at least 97% complete-task success with a 12-second response target. The platform lead proposes eagerly loading everything to avoid discovery misses; another engineer proposes deferring every definition, including search.

The architect keeps search and four common operations non-deferred and compares eager loading with BM25 discovery. In an illustrative 200-task evaluation, deferred loading reduces median input consumption by 60%, but only 184 tasks discover every required operation. Selection succeeds on 181 of those. The lower token use cannot justify release against the 97% requirement.

The catalog owner adds customer vocabulary to descriptions and repairs missing category cues. A second run finds all required operations in 198 tasks and completes 196 correctly, with measured latency inside the target. The service owner accepts that candidate subject to the broader denial and recovery checks in 3.1.4. Increasing the result limit remains an alternative to test, rather than a substitute for diagnosing the original misses.

Common exam traps

• Deferring unauthorized tools instead of removing them.
• Deferring the search tool and making the request invalid.
• Counting successful selection only after discovery and hiding missed workflows.
• Returning a client tool_result for a built-in search call.
• Treating default five-result expansion as a maximum catalog size.
• Comparing only first-turn tokens or silently changing cache conditions.

What to remember

Keep an authorized catalog, defer rarely needed definitions, and preserve server search blocks in history. Measure discovery recall, conditional selection quality, complete-task success, and total context consumption separately. Savings matter only when the necessary tools are found and the workflow still meets its target.

Knowledge check

References

• Anthropic, MCP connector
• Anthropic, Tool search tool
• Anthropic, Token counting
• Anthropic, Handle tool calls
• Anthropic, Writing effective tools for agents
• Anthropic, Prompt caching