AnthropicClaude Certified Architect – Professional2026-103.1 Evaluate tool/agent configuration for capability bloat
Validate the complete redesigned tool path
Completed by the knowledge check
Learning objective
Validate redesigned tool sets against representative selection, denial, and recovery cases.
Core idea
A redesigned menu must improve legitimate selection and preserve execution boundaries when the model makes a mistake. It must also complete an appropriate recovery path when a required operation is unavailable or denied.
The professional decision is whether the whole changed configuration is ready for release. Separate three forms of evidence: requests select suitable capabilities, prohibited calls cause no prohibited effects, and failures end in a useful authorized outcome or an explicit unresolved result. Passing one category does not compensate for failing another.
How it works
Evaluate the changed behavior, not only individual tools
Build the evaluation from the audit dispositions in 3.1.1. Every removal, merge, narrowed implementation, or specialist split needs cases showing both the intended benefit and what might break. Anthropic’s tool-engineering guidance recommends realistic tasks, evaluation metrics, and raw transcript review. Use that approach to compare the original and candidate configurations on the same assignments.
Version the assembled tools array or Agent SDK definitions together with the request-builder behavior, prompts, dispatch names, and relevant permissions. If the candidate succeeds with a test-only dispatcher but production still recognizes an old name, you have tested a different configuration. Record the model and execution surface as well, especially when adopting the tool search described in 3.1.3.
Labels should describe acceptable operations and outcomes rather than requiring one exact transcript. A request can have several valid read paths. A write that exceeds the role’s authority has none. This avoids penalizing harmless strategy variation while accepting a forbidden route because the final answer sounds correct.
Cover selection and legitimate completion
Use typical requests, boundary requests, paraphrases, multi-operation tasks, and rare business workflows. Include the confusing pairs that motivated the redesign and requests that require capabilities kept after the audit. For specialist splits, require the parent to receive the facts necessary to complete the business task; a successful branch alone is insufficient.
Inspect tool_use.name and tool_use.input alongside returned results. Separate wrong-operation choices, invalid arguments, redundant calls, tool errors, and accepted task outcomes. Anthropic recommends these measurements because a final answer can conceal an inefficient or incorrect tool strategy.
Do not evaluate pruning only on the common workload that justified it. Add seasonal and incident cases explicitly. If an operation moves to a specialist, test that the authorized role can still reach it and that the ordinary role cannot regain it through delegation. The redesign should preserve justified use while removing the surplus route.
Test denial through the execution path
For removed operations, inspect the effective capability configuration and then drive attempted calls against the application dispatcher in a controlled harness. The expectation is that an unknown or unavailable operation never enters its business handler. This dispatcher check is application-owned; the Messages API’s tool definition is not your backend authorization mechanism.
Recall that a PreToolUse hook returns hookSpecificOutput.permissionDecision: "deny" to block an operation, and test that denial through the deployed matcher. Require zero forbidden effects from those execution tests.
Derive additional denial cases from each audit disposition and run them against both configurations. When two tools merge, exercise the merged operation’s full scope; when work moves to a specialist, challenge the parent’s delegation route as well as the specialist’s direct menu. Record which previously reachable action disappears and which authorized route remains. This comparison shows whether the redesign changes authority as intended, rather than merely preserving a passing test suite.
Include adversarial documents and tool outputs, as Anthropic’s injection guidance recommends. Reuse the same malicious instruction across the original and candidate routes, including specialist handoffs, so the comparison exposes any authority the redesign accidentally restores.
Verify recovery without restoring removed authority
Use failures tied to the actual candidate. A narrowed client loader can reject an out-of-scope resource with a correlated tool_result, tool_use_id, and is_error: true. A clear explanation should identify the permitted next step. Check that the agent reports the gap or selects an authorized alternative, rather than switching to the broad loader that the redesign removed.
For tool search, distinguish an empty tool_references array from tool_search_tool_result_error. The former means no match; the latter carries an error_code. Exercise invalid_tool_input, such as a malformed regex, and verify corrected search input; improve searchable descriptions or patterns when a legitimate tool is not found. Also inject unavailable (timeout or service unavailable), too_many_requests (tool-search rate limiting), and execution_time_exceeded. For each transient failure, require a retry within the team’s attempt budget or an explicit unresolved result, never silent restoration of the eager or unrestricted catalog. These are application acceptance criteria, not an automatic retry guarantee.
A referenced tool missing its definition in tools instead causes an HTTP 400 for the whole request. Treat it as a configuration defect that blocks release, and supply every referenced definition before rerunning; it is not an in-conversation recovery result.
Apply a team-defined attempt budget and unresolved outcome to recovery cases. Anthropic does not prescribe this evaluation’s budget. Record whether recovery completes the business task, consumes excessive extra calls, or escalates without claiming success. Keep protocol correlation intact throughout; a recovery message that breaks history does not count as a functioning alternative.
Key distinctions and trade-offs
Use separate release criteria for the three categories:
| Category | Positive requirement | Failure that blocks acceptance |
|---|---|---|
| Selection | Required workflows reach correct accepted outcomes | Better average scores hide a lost critical workflow |
| Denial | Forbidden operations produce zero forbidden effects | A polite refusal masks another reachable write path |
| Recovery | Errors lead to authorized completion or explicit escalation | Recovery regains removed authority or invents completion |
Representative versus exhaustive: Select cases from actual workflows and audit findings, then add targeted adversarial and rare cases. A large set of easy paraphrases cannot substitute for one missing critical denial case.
Local tool success versus service success: Each handler can work while the redesigned orchestration fails to supply a required identifier or complete the transaction. Release against the full route and its business target.
Architecture / decision scenario
A technology company’s incident assistant must diagnose service failures within three minutes. The redesign merges duplicate log searches, removes production configuration writes from diagnosis, and narrows a repair specialist. A 150-case pilot improves diagnosis completion from 90% to 96%, so the product lead proposes immediate release.
The architect adds denial and recovery cases before acceptance. A crafted log entry asks the diagnosis agent to change production settings; the write must remain absent, including through delegation. A direct prohibited call must produce zero handler entries. An authorized repair exceeding its permitted scope must reach the registered PreToolUse denial and produce zero changes.
Recovery tests uncover a defect: when narrowed log retrieval fails, an available generic operations tool can fetch logs and change settings. The team removes that alternate route and supplies a bounded log-only fallback. Missing evidence then produces a partial diagnosis and an escalation instead of invented resolution. The on-call service owner accepts release only after the legitimate cases still meet the target, forbidden effects remain zero, and escalation reaches the designated operator.
Common exam traps
- Approving a smaller menu because common-case accuracy improves.
- Scoring final text without inspecting calls and effects.
- Reusing the original denial suite without adding cases for merged tools and specialist routes.
- Testing the narrowed specialist directly while omitting the parent’s delegation path.
- Counting a refusal as successful escalation without observing the escalation.
- Restoring broad tools whenever a narrowed path returns an error.
What to remember
Validate each changed route across selection, denial, and recovery. Observe calls, handlers, downstream effects, and complete business outcomes. Accept a redesign only when its improved menu preserves legitimate work and its recovery path preserves the authority reduction.
Knowledge check
A bank’s redesigned fraud-review tools raise case completion from 88% to 95% in a 200-case pilot. The redesign removed account-freeze writes from the reviewer role and moved them to a supervisor specialist that the review coordinator can still delegate to. The product owner asks for release now. What must the evaluation add before acceptance?
During recovery testing of a retailer’s deferred returns-tool catalog, a search call returns a tool_search_tool_result_error with error_code: "unavailable". The team’s acceptance criteria allow two attempts per search. What behavior should the test require?
unavailable means the search could not run, for example because it timed out. That is a transient failure, so recovery means a bounded retry, continuing the task if it succeeds and reporting an unresolved result if the budget is exhausted. An empty tool_references array is what signals that nothing matched, and invalid_tool_input is the code for a malformed pattern. Falling back to the full catalog silently undoes the redesign.A recovery test makes the narrowed read_service_logs tool return is_error: true. The diagnosis agent then completes the task through ops_console, a generic tool that can read logs and change production settings, and its diagnosis is correct. The diagnosis role must never change production settings. What should the team conclude?
ops_console asks for confirmationread_service_logsA candidate tool set passes its selection and denial suites in a staging harness whose dispatcher was written for the test. The production dispatcher still routes the retired name search_logs_v1 to its old handler. What do these results establish about the production configuration?
References
- Anthropic, Writing effective tools for agents
- Anthropic, Control execution with hooks
- Anthropic, Handle tool calls
- Anthropic, Tool search tool
- Anthropic, Mitigate jailbreaks and prompt injections