Choose retrieval, full context, or live tools

Learning objective

Decide when retrieval is warranted versus loading documents into long context or querying live systems through tools.

Core idea

Choose the information path from the question’s completeness, freshness, and access requirements. Retrieval-augmented generation (RAG) selects passages before Claude answers. Long context supplies the complete relevant documents. A live-system tool asks the authoritative service for state at execution time. Each can ground an answer, but each exposes a different failure mode.

For an architect, the decision is not whether a corpus technically fits a context window. It is whether the selected path meets the business requirement at an acceptable cost and latency. A complete contract comparison, a search across manuals, and an account balance lookup should not inherit the same design merely because they share a chat interface.

How it works

Select passages when the corpus exceeds the task

In RAG, the application retrieves a subset of an external corpus and sends that subset with the question. Anthropic’s contextual retrieval article describes embeddings for semantic similarity and BM25 for lexical matching. The retrieval stage becomes part of the answer’s correctness: omitted exceptions cannot reliably influence the answer.

Use retrieval when many documents exist but each question needs a small, identifiable subset. A policy assistant that must search thousands of procedures benefits from selecting relevant sections rather than transmitting the entire collection for every question. The application owns ingestion, authorization, source versions, and selection; Claude does not build or maintain the organization’s index through a Messages API request.

Citations identify supplied material but do not prove that retrieval found every necessary passage; see 2.2.4 for the search_result interface.

Load complete documents when omissions matter

Long context avoids a retrieval selection step for documents you already know the task requires. A review that asks for every inconsistency across three selected agreements is different from a question asking where one clause appears. If selection excludes a relevant section, the review is incomplete before generation begins.

Anthropic’s long-context guidance places long documents near the beginning of the prompt, with the query after them. It recommends explicit document structure using <document>, <document_content>, and <source> tags, and asking for relevant quotes before analysis. Those are prompt conventions, not access controls or a guarantee of exhaustive reading.

Claude context windows reach 1 million tokens depending on the model; other models have a 200,000-token window. The system prompt, messages, documents, tool definitions, and generated output all count. A request can include up to 600 images or PDF pages on 1-million-token models, or 100 on 200,000-token models; request-size limits can bind before token capacity. Budget the whole request, not only the documents; see 2.4.1.

The Contextual Retrieval article suggests loading the whole knowledge base when it is smaller than 200,000 tokens, approximately 500 pages, and pairs that approach with prompt caching. Treat this as its rule of thumb, not a universal routing threshold. For reused agreements, cache the document prefix with cache_control to reduce repeated processing; see 2.5.1. Caching does not enlarge the context window. Even when a request fits, measure cost and latency rather than assuming either full context or retrieval is cheaper.

Query live state through an authorized tool

For account status, stock availability, or a transaction’s present state, ask the system of record. A periodically embedded snapshot may be accurate for its timestamp yet wrong for the user’s decision. The freshness requirement decides whether that lag is acceptable.

Define a client tool in tools with name, description, and input_schema. Claude requests it through a tool_use block; your application executes the lookup and returns a tool_result associated with that call. Keep the interface narrow: a named balance lookup is easier to authorize and validate than arbitrary SQL against the entire finance database.

Apply the authenticated authorization policy from 3.2.2 to the live lookup. Return a timestamp and source reference in the application-defined result when users must distinguish observed state from a guaranteed reservation. A read of inventory does not itself reserve an item.

Key distinctions and trade-offs

The following choices expose different correctness risks:

A tool is an interface, whereas RAG is a selection pattern. A retrieval service can itself be exposed as a tool. Calling it a tool does not turn indexed documents into live authoritative state. Likewise, documents returned by a live tool can enter the same answer as retrieved policy passages.

Hybrid delivery is often appropriate, but each source should answer a distinct part of the question. Retrieve the return policy, query the actual order, and load the selected disputed agreement in full if the task requires comprehensive comparison. Keep source versions and timestamps distinguishable rather than presenting them as one undated body of knowledge.

Architecture / decision scenario

A retail architect must support policy questions across 40,000 documents, compare two supplier agreements in full, and confirm whether an order can still be cancelled. Policy updates tolerate ten minutes of index lag; order state must come from the transaction service during the request. Interactive answers have a four-second service target.

A proposed single vector index would be easy to operate. It fails the cancellation requirement because a ten-minute snapshot can miss a shipment, and it risks omitting an exception during exhaustive agreement comparison. Loading everything into long context also fails the four-second target and ignores the need for a live order check.

The architect assigns policy questions to authorized retrieval, agreement comparisons to complete selected documents with request budgeting (see 2.4.1) and prompt caching for reused agreement prefixes (see 2.5.1), and cancellation eligibility to an authenticated client tool. The commerce service owns the state lookup; the knowledge team owns policy freshness. A failed live lookup produces an unavailable-state response rather than a guess from the policy index. Before rollout, representative requests must meet both their correctness rubric and the latency target.

Common exam traps

• Choosing RAG solely because documents are involved.
• Treating context capacity as proof of affordable, exhaustive analysis.
• Reading operational state from a stale document index when the decision requires a live value.
• Assuming a retrieval tool’s interface makes its contents authoritative or fresh.
• Treating citations as proof that no relevant passage was omitted.

What to remember

Match the information path to the decision: retrieve a relevant subset, load a complete bounded set, or query authoritative live state. Defend omissions, freshness, authorization, and measured request cost separately. The application owns source selection even when Claude chooses when to call the interface.

Knowledge check

References

• Anthropic, Introducing Contextual Retrieval
• Anthropic, Claude prompting best practices: Long context prompting
• Anthropic, Tool use with Claude
• Anthropic, Search results
• Anthropic, Context windows
• Anthropic, Prompt caching