Skip to lesson

AnthropicClaude Certified Architect – Professional2026-104.4 Diagnose system issues (prompt failure, hallucinations, model mismatch)

Prove when the model should change

Completed by the knowledge check

Learning objective

Diagnose model mismatch where a task exceeds a model’s capability or a cheaper model would suffice.

Core idea

Model mismatch has two directions. A model can fall short on a difficult task, or an expensive configuration can do work a cheaper one completes acceptably. Neither diagnosis follows from the model’s reputation or a single impressive answer. Both require workload-specific comparisons against the same business requirements.

Anthropic’s model-selection guidance supports efficiency-first and capability-first starting points, then testing actual prompts and data. Treat the chosen model as a hypothesis about sufficient capability at an acceptable operating cost. The decision is whether to retain, tune, upgrade, downgrade, or selectively route work—not whether one model is universally better.

How it works

Clear configuration defects before declaring a ceiling

Use 4.4.1 to exclude missing input, bad retrieval, tool failures, and output handling. A stronger model cannot establish a policy fact that the request never supplied. A cheaper model should not be rejected because its test environment accidentally withheld the required source.

Record the actual Messages API model, max_tokens, thinking configuration, and output_config.effort where supported. Also preserve the prompt and tool definitions. A comparison of model names without their settings is incomplete: the team may be measuring different inference budgets or incompatible request features.

Check the models’ documented request compatibility before running a comparison. Anthropic’s prompting guidance notes model-specific differences in thinking defaults and supported settings. For example, setting budget_tokens on Claude 4.7 and later models returns a 400 error; this request-compatibility failure is not evidence of insufficient capability. Prompt overtriggering is another migration lookalike; see 4.4.2. The same guidance also warns that responses can contain thinking blocks and recommends selecting content by type, with room in max_tokens for thinking. An extractor that assumes the first block is the answer can create a migration regression unrelated to reasoning quality.

Inspect stop_reason alongside grades. max_tokens identifies a response cut at the configured output limit; raise the allowance or use the documented continuation path before labeling omitted reasoning as incapacity. Record that configuration repair as its own comparison so the team can distinguish truncation from a genuine capability difference.

Compare capability on the slice that fails

Freeze the same task inputs, applicable passages, and tool results. Compare the candidate configurations with repeated trials and the same rubric. Group results by relevant task difficulty: simple extraction, exception-heavy interpretation, or multi-step planning, for example. A mean quality score can hide a critical shortfall on a small but consequential slice.

A capability-gap hypothesis gains support when failures survive coherent instructions and usable context, then improve under a more capable configuration on the same cases. Inspect how the failure changes. A stronger model that produces more plausible unsupported prose has not solved the problem; it must meet the original acceptance criterion.

Anthropic’s model-selection guidance identifies effort tuning as often preferable to switching models; where supported, test it before declaring a model-capability ceiling. See 2.1.3 for choosing supported reasoning settings. The Claude API field is output_config.effort; the Effort page documents request-level control without a beta header on supported models. It separately labels per-message effort as beta. This lesson concerns request-level comparisons, not changing effort mid-conversation. Consult that page’s model support list before choosing a sweep.

Keep the interpretation narrow. If higher effort repairs the hard slice, the evidence supports a configuration mismatch. If a different model repairs it at acceptable latency and cost, that supports a model change. If neither helps, revisit the task design or required knowledge instead of repeatedly buying more inference.

Test whether a cheaper configuration suffices

Use the same discipline in the opposite direction. Compare a lower-cost candidate with the deployed configuration on routine tasks, difficult cases, and cases near the acceptance boundary. A successful routine sample alone does not establish that the entire workload can move.

Measure the cost of accepted work, not only one successful API call. Include failed attempts, correction calls, escalation, and the human review the workflow needs. A candidate with lower per-call cost can become more expensive if it increases retries or reviewer intervention. Conversely, a small quality difference may be immaterial if both meet the explicitly approved threshold.

Anthropic’s selection guidance recommends comparing accuracy, response quality, edge cases, cost, and speed. Turn those dimensions into a decision record with per-slice acceptance, latency, cost, and an owner. See 2.1.1 for model-choice economics; here the record explains why the model’s capability fits the assigned work.

Identify the mis-assigned slice

Selective routing needs a tested distinction between routine and difficult work; see 2.1.2 for designing and evaluating that route. Here the diagnostic artifact is the per-slice comparison that establishes which work is mis-assigned before routing is proposed.

Record the failed requirement, baseline error rate, response completion state, applicable-context check, and outcome under each substituted configuration. If both models fail the same cases after receiving usable context, the comparison does not yet support a capability explanation. If only the deployed model repeatedly fails exception-heavy cases while both pass routine ones, the evidence supports a capability shortfall confined to that slice. If the cheaper model also passes the routine slice, the same comparison identifies unnecessary spend there.

Retain counterexamples that weaken the diagnosis. A candidate that improves aggregate accuracy but misses a mandatory exception is not sufficient for that slice. Re-run the integrated workflow after frozen-result comparisons to check whether live tool choice or recovery changes the conclusion.

Key distinctions and trade-offs

Model mismatch versus configuration mismatch: Both can lower quality. A model change is justified only after supported settings and response handling are accounted for.

Capability versus reliability: Solving a task once demonstrates possibility. Repeated accepted outcomes under the workload’s constraints support deployment.

Cheaper call versus cheaper accepted outcome: Retries, review, and escalation can reverse the ranking implied by per-call cost.

Global upgrade versus selective routing: A global change avoids a router but charges every request for the hard slice. Routing earns its complexity only when the end-to-end comparison includes mistakes and operating overhead.

Architecture / decision scenario

A retailer classifies 200,000 supplier messages each month. Ninety percent are routine acknowledgments; ten percent contain delivery exceptions requiring several policy conditions. The service needs at least 99% routine accuracy, 97% exception accuracy, and a three-second p95 response target. All figures here are illustrative workload requirements and measurements, not vendor prices or benchmarks.

Frozen-input trials show the deployed configuration reaches 99.5% on routine messages but only 93% on exceptions. A supported higher-effort configuration reaches 95%, while a more capable model reaches 98% on exceptions and 99.6% on routine messages, with measured p95 latency of 2.8 seconds across the workload. Both comparisons exclude truncated replies and use the same usable policy passages. The exception shortfall therefore survives a configuration repair and supports a capability upgrade for that slice.

A cheaper candidate reaches 99.2% on routine messages but 89% on exceptions. Including retries and review, its measured routine cost is $0.002 per accepted item versus $0.006 for the deployed configuration. The stronger model costs $0.012 per accepted routine item and $0.018 per accepted exception item. Moving everything cheaper fails the exception requirement; a global upgrade meets both accuracy targets and the three-second target but costs six times the cheaper candidate on routine work.

The architect chooses selective routing only after testing an intake label that identifies exception messages and sends ambiguous cases to the stronger path. Operations owns routing-error review; procurement owns the acceptance rubric. The release comparison includes missed exceptions, escalation cost, and the three-second target. If the label cannot maintain the exception requirement, the architect uses the global upgrade, accepting its $0.012 routine-item cost as the price of meeting the requirement without routing. Retaining the deployed configuration at 93% exception accuracy is not an acceptable fallback.

Common exam traps

  • Calling a model incapable because max_tokens cut off its response.
  • Comparing model names while changing prompts, context, and effort together.
  • Accepting an average score that conceals failures in a critical slice.
  • Keeping the most expensive configuration without testing cheaper sufficient ones.
  • Estimating savings from per-call prices while ignoring retries and review.
  • Deploying a routing strategy without evaluating its missed hard cases.

What to remember

Prove both insufficient capability and unnecessary spend with controlled, repeated comparisons. Clear request and response-handling defects first. Choose a model or route from per-slice acceptance, latency, and the complete cost of accepted work.

Knowledge check

References