Separate retrieval faults from unfaithful answers

Learning objective

Diagnose hallucinations by separating missing, stale, or irrelevant retrieved context from unfaithful use of supplied context.

Core idea

Absent source evidence cannot be recovered by repeatedly demanding a complete answer. The additional diagnostic question for a retrieval system is where the usable fact disappeared: the source collection, retrieval, request assembly, or the answer’s use of supplied material.

Separate factual correctness from faithfulness to context. An answer can faithfully repeat an obsolete passage and still be wrong for the user’s effective date. It can also contradict a correct passage that the request supplied. Those failures need different repairs and different owners; scoring both as “hallucination” hides the distinction.

How it works

Follow a claim back to its source

Choose one consequential claim in the answer and identify the passage that would justify it. Compare three artifacts: the authoritative source for the question’s scope and date, the passages the retriever selected, and the content the model actually received in messages or tool_result blocks.

Do not infer that a passage reached Claude because the search log lists its identifier. An assembler can truncate, deduplicate, or omit a selected result. Conversely, the model can receive an obsolete passage even when the document store contains the corrected one. The final request is the boundary for diagnosing how Claude used context.

Anthropic’s Contextual Retrieval article describes a specific retrieval failure: isolated chunks lose the company or period that makes their content meaningful. That supports checking whether a passage is applicable, rather than treating keyword similarity as proof that it can answer the question. See 3.5.3 for chunk design, 3.5.4 for index revision lifecycle, and 3.5.5 for retrieval measurement.

Separate four context conditions

Classify the evidence condition before choosing a generation change:

If the fact is absent from the authorized source collection itself, record an answerability gap. Retrieval cannot find unpublished information. The business owner must decide whether to obtain a source, narrow the service’s scope, or accept an explicit unknown outcome.

A passage can occupy more than one failure category. A superseded policy for the wrong subsidiary is both stale and irrelevant. Preserve the conditions that matter to the claim instead of forcing every incident into one label. The labels exist to identify repairs, not to produce a convenient incident count.

Use an applicable-context substitution

Replace the production results with an expert-selected passage that actually supports the approved answer. Keep the question, prompt, model, and grader fixed, then repeat trials as in 4.4.1. Improvement supports a context-delivery contribution. It does not yet distinguish a retrieval miss from an assembler that dropped the useful result; compare those two boundaries directly.

If unsupported claims persist with usable context, test the instructions. Anthropic recommends allowing uncertainty, restricting answers to provided documents, extracting direct quotes before answering for documents longer than 20,000 tokens, and verifying claims against supporting quotes. Use these as prompt candidates on the same frozen passages rather than changing retrieval and generation together.

For diagnosis, ask for the supporting quote and check it against the supplied text. A verbatim quote establishes textual presence, but not that the conclusion follows. “Coverage is available after approval” does not support “coverage is approved.” Inspect qualifiers, negation, dates, and entity scope as part of the claim check.

For document-based answers, native citations use citations: {"enabled": true} on each document; citations must be enabled on all or none of the documents within a request. The API extracts cited_text and guarantees valid pointers to supplied documents, while a reviewer still judges whether the cited passage supports the claim and its qualifiers; see 2.2.4 for the mechanism. Native citations cannot be combined with output_config.format structured outputs in the same request; the API returns a 400 error.

If a coherent, source-restricted prompt still fails on usable context, compare model configurations through 4.4.4. Record the surviving error category; do not upgrade the whole workload merely because one long, qualified passage is difficult.

The substitution’s result sends the diagnosis either to the context-delivery boundaries or on to the instructions and then to the model.

Measure retrieval and faithfulness separately

Keep separate denominators. Among answerable questions, measure whether the required applicable passage reached the model. Among requests with usable passages, measure whether the answer stays within what those passages support. Separately count appropriate unknown answers for questions the sources cannot resolve.

An improved answer score alone cannot tell you which component improved. A model might memorize a policy well enough to answer correctly despite a retrieval miss, then fail when the policy changes. Conversely, excellent passage delivery can coexist with poor use of exceptions. Preserve both measures when validating the repair.

Anthropic states that hallucination-reduction techniques do not eliminate hallucinations. Critical claims therefore still need validation appropriate to their consequence. Here the concrete validation artifact is a reviewed claim-to-passage record, with source revision and an applicability judgment, rather than a confidence score generated by the answerer.

Key distinctions and trade-offs

Source absence versus retrieval miss: The first needs additional authorized knowledge or a narrower answer. The second needs a better path to knowledge already available.

Faithfulness versus correctness: Repeating stale context can be faithful and incorrect. Correctness by coincidence does not establish reliable grounding.

Quote match versus support: Textual matching catches invented quotations. A reviewer still has to check whether the quotation supports the particular claim and conditions.

Supplying more passages can improve coverage but also introduces stale versions and competing scopes. Fix the identified applicability problem before expanding context indiscriminately. Keep historical documents when the question concerns a historical date; freshness means appropriate to the question, not always the most recently published revision.

Architecture / decision scenario

A healthcare provider drafts administrative coverage answers for staff; no clinical recommendations are in scope. A reviewer must approve every draft within a four-minute handling budget. In a reviewed sample of 60 incorrect claims, the retrieval team proposes doubling the number of returned passages for every request.

The architect separates the failures. Fifteen claims lack a passage that exists in the authorized collection. Twelve quote superseded coverage rules. Nine use rules for a different plan. Twenty-four contradict an applicable exception already present in the final request. Doubling results has no demonstrated remedy for the last group and could introduce more competing versions.

The retrieval owner tests expert-selected passage substitutions for the first three groups, distinguishing search misses from assembly losses. The content owner repairs revision selection for the requested coverage date. For the final group, the prompt owner tests source restriction and quote-supported answers with all passages frozen; a reviewer checks whether each exception actually supports the claim.

The architect accepts targeted retrieval repairs plus the measured generation repair, retaining human approval. Release requires both applicable-passage delivery and faithful claims to improve without raising reviewer handling time beyond the budget. Questions whose authorized sources remain silent produce an explicit unknown outcome, not a guessed coverage decision.

Common exam traps

• Calling a wrong answer proof that retrieval failed.
• Assuming that a document present in the index reached the model.
• Treating a stale but faithfully repeated policy as a generation-only defect.
• Treating a verbatim quotation as proof of the answer’s interpretation.
• Increasing result count before diagnosing relevance and revision selection.
• Rewarding filled answers while penalizing correct unknown outcomes.

What to remember

Check source availability, retrieved applicability, final request delivery, and claim support separately. Freeze context to test faithful use; substitute usable context to test delivery. Assign the repair to the boundary that failed and verify both retrieval and answer behavior.

Knowledge check

References

• Anthropic, Reduce hallucinations
• Anthropic, Introducing Contextual Retrieval
• Anthropic, Handle tool calls
• Anthropic, Demystifying evals for AI agents
• Anthropic, Citations