Separate six sources of model failure

Learning objective

Identify model limitations: hallucination, knowledge cutoff, reasoning errors, context degradation, phrasing sensitivity, and nondeterminism.

Core idea

As an architect, you need a catalog of ways an answer can fail even when the request succeeds. Context capacity and input layout are prerequisites; this lesson adds the business consequence, diagnostic distinction, and accountable check for six model limitations.

Do not register a single risk called “the model is wrong.” A missing policy update, an unsupported assertion, and an incorrect calculation require different interventions. A successful HTTP response tells you that generation completed, not that the result is suitable for a customer decision.

How it works

Separate unsupported content from unavailable knowledge

Hallucination is generated content that is factually incorrect or inconsistent with supplied context. Anthropic’s guidance recommends allowing uncertainty, grounding claims in direct quotes, and checking claims against sources. It explicitly warns that these techniques reduce hallucinations without eliminating them. The architecture therefore needs a rule for withholding an unsupported material claim, even when the rest of the answer reads well.

Record the claim, the supporting passage, and the consequence of accepting it. A passage’s existence does not prove that the answer follows from it: a quoted exception can contradict the claimed general rule. The domain owner defines what constitutes sufficient support. See 4.4.3 for diagnosing retrieval and answer-faithfulness failures.

Knowledge cutoff is a different exposure. Anthropic’s model comparison distinguishes a reliable knowledge cutoff from a training-data cutoff. Consult the deployed model’s row rather than assigning one date to every Claude model. Neither date certifies that every earlier fact is known or correct. For facts that must reflect a particular effective date, require an authoritative source at that date instead of treating model recall as the record of truth.

In a risk entry, name the business change that makes training knowledge insufficient: a revised refund policy, a product withdrawal, or a changed eligibility rule. The information owner maintains that source. The model owner cannot repair missing business information by rewriting an instruction to “be accurate.” Retrieval and live-system selection belong in 3.5.1.

Separate valid facts from valid conclusions

Reasoning errors arise when available facts are combined incorrectly. An answer can quote every input faithfully and still apply the wrong condition, reverse an exception, or compute the wrong result. This distinction matters because adding citations addresses attribution, not logical validity.

For an exact financial calculation, the proposed architecture can have application code compute the result and let Claude explain it. For a judgment that cannot be reduced to code, the domain expert defines worked cases and the conditions that require review. These are design choices, not claims that Claude has a built-in correctness checker. Anthropic’s hallucination guidance describes reviewing summarized reasoning with thinking and display: "summarized"; treat that as diagnostic assistance, never as an independent proof of the conclusion.

Context degradation concerns evidence that was supplied but used poorly as context grows. Anthropic calls declining accuracy and recall with increasing token count context rot. A request that fits its context window can therefore omit a decisive exception in its answer. Your risk catalog should distinguish “source absent” from “source present, qualification ignored.” Otherwise the team might expand retrieval and make the latter problem worse.

Measure consequential omissions across representative input lengths and document layouts. Keep the owner of input curation separate from the owner who approves a business interpretation. Increasing window capacity does not settle whether either owner has demonstrated adequate coverage.

Separate prompt dependence from run-to-run variation

Phrasing sensitivity means wording is part of the behavior being released. Anthropic’s prompting guidance emphasizes clear, explicit instructions and model-specific prompting differences. Treat an edit to system, examples, or the question template as a change requiring comparison on the relevant cases, even when developers describe it as copyediting. See 2.5.2 for prompt-component releases and 4.3.1 for controlled comparisons.

Nondeterminism concerns variation when the intended request is held fixed. The Messages reference states that temperature: 0.0 does not make results fully deterministic on models that support it. It also marks temperature deprecated: models released after Claude Opus 4.6 accept 1.0 for backward compatibility and reject other values with HTTP 400. Do not prescribe zero temperature as a universal production fix.

Store the answer actually used, together with the model and prompt version, rather than promising to regenerate the identical answer later. Repeated trials reveal variation; they do not make a repeated wrong answer correct. See 4.3.3 for interpreting stochastic evaluation results.

Key distinctions and trade-offs

Use the following diagnostic split before choosing a mitigation:

These labels can overlap. A reasoning error may produce a hallucinated conclusion; nondeterminism may change whether an exception is noticed. Use the labels to identify the next discriminating check, not to force every incident into one exclusive category.

Architecture / decision scenario

A financial-services team must draft fee-dispute explanations within 12 seconds. Its pilot of 600 cases contains six unsupported policy claims, four failures to use a policy amendment, three incorrect date calculations, and five missed exceptions in long histories. A cosmetic prompt revision also changes which fee details appear; repeated unchanged requests sometimes disagree on eligibility. These are illustrative pilot results.

The tempting proposal is a larger model with temperature: 0, promoted after one successful replay. The architect rejects that as insufficient: it neither supplies the amendment nor independently checks arithmetic, and the selected model’s documented sampling support must be respected.

The policy owner supplies effective-dated policy material. Application code computes the disputed interval. The model team compares prompt versions, repeats consequential cases, and checks claim support and long-history exception coverage. Unsupported or inconsistent eligibility remains unresolved for a domain reviewer. The service owner accepts additional review latency for those cases while retaining the 12-second target for supported drafts. One aggregate accuracy score would hide which of these responsibilities remains unfulfilled.

Common exam traps

• Treating a knowledge cutoff as a guarantee of complete knowledge before that date.
• Assuming citations establish that a conclusion follows from the cited facts.
• Confusing input fit with reliable use of every supplied qualification.
• Releasing a phrasing change without checking affected outcomes.
• Prescribing temperature: 0 without checking model support or promising deterministic output.
• Choosing a larger model as a substitute for missing information or exact computation.

What to remember

Identify what failed before choosing what to change. Separate missing knowledge, unsupported claims, invalid reasoning, degraded context use, prompt dependence, and run-to-run variation. Assign the information, calculation, prompt, and business-decision owners explicitly; no model setting removes the need to validate consequential results.

Knowledge check

References

• Anthropic, Reduce hallucinations
• Anthropic, Models overview
• Anthropic, Create a Message
• Anthropic, Context windows
• Anthropic, Prompting best practices