AnthropicClaude Certified Architect – Professional2026-107.3 Support debugging and operational issue resolution
Separate broken integrations from behavior regressions
Completed by the knowledge check
Learning objective
Isolate integration or tool failures from prompt or model-behavior regressions using correlated prompts, model versions, and tool traces.
Core idea
A worse final answer is a symptom, not a component diagnosis. The model may have received different instructions, missing retrieval results, a changed tool schema, or a failed dependency disguised as empty data. Changing the prompt before locating that difference can hide the fault and introduce another variable.
Use the correlated records described in 3.4.2 and 4.6.1 as investigation input. The addition here is a controlled comparison: identify the first point where equivalent requests diverge, then test the implicated component while holding the others fixed. Error categorization alone cannot establish which release caused the regression.
How it works
Reconstruct the actual failing exchange
Begin with a failed business outcome and its application request identifier. Join it to the Claude API request-id, the assembled system and messages, the requested model, and the tool definitions supplied for that request. Keep prompt revision and tool implementation revision in your application records. Those revision names are application metadata, not automatic Anthropic API fields.
For a client tool, follow the assistant’s tool_use.id to the returned tool_result.tool_use_id, then to the dependency operation and its result. If execution failed, inspect is_error: true and the returned explanation. A model choosing a valid lookup followed by a dependency denial is not the same defect as the model choosing an irrelevant lookup. A successful HTTP inference response does not settle whether the downstream business task succeeded.
Preserve enough authorized content to reproduce the branch, but use sanitized fixtures for repeated testing. A hash identifies an unchanged artifact; it does not let an investigator inspect an artifact that was never retained. When sensitive content cannot be retained, record the authorized retrieval procedure and state the resulting diagnostic limitation.
Find the first observable divergence
Compare one affected case with a known-good case from the same task category. Avoid comparing a simple reference lookup before release with a complex multi-tool case afterward. Use the following sequence to localize what changed:
| First difference | Candidate component | Discriminating check |
|---|---|---|
| Request rejected before inference | Request adapter | Reproduce with the assembled sanitized request |
| Expected tool result missing or failed | Tool or dependency | Execute the same authorized operation outside the agent in a test environment |
| Same tool call, changed result interpretation | Prompt, result contract, or behavior | Replay a fixed tool-result fixture with controlled prompt/model combinations |
| Different tool selection with equivalent context | Prompt, tool description, or behavior | Hold tool definitions and results fixed while comparing revisions |
| Same intermediate result, changed final application verdict | Validator or downstream consumer | Run the recorded result through both validator revisions |
These are hypotheses, not verdicts from a single trace. A tool returning empty data can be correct, a permissions defect, or a parser silently discarding a valid response. Check the backend result and adapter separately. The model’s explanation of what it thought happened cannot replace the recorded exchange.
Compare prompt and model revisions independently
Build a small diagnostic matrix using the same cases, tool definitions, and fixed dependency fixtures: baseline prompt with baseline model, changed prompt with baseline model, baseline prompt with changed model, and both changes together. If only the changed prompt fails across both models, investigate the prompt delta first. If both prompt versions fail only with live dependency calls, a model rollback is poorly supported.
Define success with task-specific measurable criteria, as Anthropic’s evaluation guidance recommends. Grade whether the correct authorized operation was selected, whether required facts survived interpretation, and whether the accepted result met the business rule. Compare rates over a representative set; a single alternate answer can reflect ordinary output variability rather than a release regression.
Use an actual model ID for the comparison. Anthropic documents model IDs as pinned versions, including dateless IDs for the 4.6 generation and later. Do not assume an ID is a moving alias merely because it lacks a date. Some earlier Claude API convenience aliases do move between dated snapshots; resolve which kind the integration uses before claiming identical models.
Pinning model weights is also not complete environmental reproduction. Anthropic distinguishes the pinned model from serving infrastructure such as request routing, safety classifiers, and sampling logic, which can change. If controlled cases deteriorate under the same ID and unchanged application artifacts, preserve request identifiers and observed results for escalation rather than asserting that the weights changed.
Assign the remedy to the demonstrated cause
For a broken adapter, repair serialization or parsing and add a deterministic reproduction. For a tool failure, restore the dependency or its authorized credentials and verify the result before retesting the agent. For a prompt or model regression, retain the failing cases as behavior evaluations and compare the proposed repair against the baseline.
If rollback restores service, record that it implicates the reverted release bundle. It does not identify which component within that bundle caused the failure. See 4.6.4 for the broader incident-to-release process and 7.3.5 for maintaining the diagnostic procedure that operators repeat.
Key distinctions and trade-offs
Fixed tool fixtures isolate behavior but conceal live dependency faults. Live end-to-end checks expose integration failures but introduce changing data and availability. Use both for different questions instead of treating either as a complete diagnosis.
Detailed prompt records improve reproducibility while increasing sensitive-data handling requirements. Store the minimum authorized material needed for the failing branch, connect it to immutable revisions, and restrict access. An aggregate accuracy chart cannot reconstruct a missing tool response.
Architecture / decision scenario
A financial-services document assistant releases a prompt revision and a tool adapter together. Its unsupported-answer rate rises from 2% to 11% on 200 reviewed cases. The model owner proposes an immediate model downgrade, but the model ID did not change and the service cannot send customer records to an unapproved deployment route.
The integration owner correlates prompts and tool traces for failed cases. Claude requested the correct document IDs, but the changed adapter returned empty tool_result content after misreading the dependency’s response. With fixed valid tool results, both prompt revisions pass 96 of 100 diagnostic cases; live calls through the changed adapter pass only 71.
The service owner rolls back the adapter, verifies live retrieval, and preserves the prompt comparison as an explicit unresolved quality question rather than inventing a model cause. A separate fixed-fixture case still fails only with the changed prompt and becomes a targeted prompt repair. The decision separates two causes while honoring the approved data route; one blanket model downgrade would explain neither.
Common exam traps
- Inferring model regression from an aggregate quality drop alone.
- Treating empty tool data as proof that the source contains no records.
- Changing prompt, model, and tool implementation in one diagnostic replay.
- Treating every dateless model ID as an alias.
- Treating recovery after rollback as proof of one internal root cause.
What to remember
Follow the failing outcome backward through actual prompts, model identity, tool calls, and returned results. Find the first divergence, test one variable at a time, and distinguish fixed-fixture behavior checks from live integration checks. Repair the component the comparison implicates.
Knowledge check
After a release, an insurance claims assistant’s wrong-answer rate rises. For the failed cases, Claude requested the correct policy lookup with the right policy ID, and the backend log shows the dependency returned the record, but the tool_result Claude received was empty. Which component should the investigation examine first?
tool_result from the responsetool_result : the tool call was right and the backend returned data, so the record was lost in the adapter between them. Prompt, model and validator all act later in the exchange or on a correct earlier step, and investigating them first would change variables downstream of the fault.A team changed its prompt and its model in one release. Over the same cases and fixed tool-result fixtures, it runs a diagnostic matrix: baseline prompt with baseline model passes, baseline prompt with new model passes, new prompt with baseline model fails, and new prompt with new model fails. What does the matrix support?
Over repeated runs of a representative evaluation set, quality has declined although the integration still sends the model ID claude-sonnet-4-6 and no application artifact changed. Which statement about the model is supported?
A release bundle containing a prompt revision, a tool adapter and a result validator is rolled back, and service recovers. The incident record must state what is known about the cause. What does the recovery establish?
References
- Anthropic, Claude API errors
- Anthropic, Handle tool calls
- Anthropic, Model IDs and versioning
- Anthropic, Define success criteria and build evaluations