Tune relevance across the retrieval stages

Learning objective

Improve relevance with hybrid search, reranking, query rewriting or decomposition, freshness filters, and tuned retrieval depth.

Core idea

Retrieval quality is a sequence of decisions: which query to issue, which records are eligible, which candidates to collect, how to order them, and how much to pass to Claude. Improving one stage cannot repair every defect. A reranker cannot select a document that never entered its candidate set, and broader retrieval cannot make an obsolete policy applicable.

Treat each improvement as a measured change to an owned retrieval configuration. Method selection in 3.6.2 establishes the route; this lesson improves relevance within that route. Keep corpus construction and contextual chunk enrichment in subsection 3.5, rather than treating runtime tuning as a reason to repeat ingestion design.

How it works

Combine lexical and semantic candidates

Hybrid search combines lexical and embedding-based retrieval. Anthropic’s Contextual Retrieval article describes taking BM25 results and semantic results, combining them through rank fusion, and removing duplicates. Lexical retrieval contributes exact terms and identifiers; semantic retrieval contributes passages expressed through different wording.

Index-side fusion and deduplication preserve stable chunk identity across lexical and semantic results; see 3.5.4. At query time, record each candidate’s source and the retrieval route that found it so you can determine whether a missed case needs lexical coverage, semantic coverage, or a different query constraint. Compare the combined candidate set with single-method baselines on both identifier-bearing and paraphrased questions.

Rerank the candidate set

A reranker scores retrieved candidates against the query and selects the passages most useful for the answer. Anthropic describes retrieving 150 candidates and retaining 20 in its experiment. Those are experimental settings, not API limits or defaults. The article also compares passing 5, 10, and 20 chunks and reports that 20 performed best among those options on its tested data.

Separate candidate depth from final context depth. Candidate depth determines the work available to the reranker; final context depth determines what Claude receives. Raising only final depth can add irrelevant text without improving the candidates. Raising candidate depth while keeping final depth stable can recover useful evidence but adds reranking work.

Lexical and semantic candidates are fused and deduplicated before the reranker, which selects the final context Claude receives.

Measure candidate recall before judging reranking; see 3.5.5 for retrieval-quality metrics. If the required passage is absent from the candidate set, fix retrieval or the query. If it is present but falls below the final cutoff, investigate reranker behavior and context selection. A reranker improves ordering, not identity, authorization, or population completeness.

Account for the whole request. Reranking adds runtime latency and cost, while a smaller final context can reduce downstream model work. Anthropic explicitly identifies that trade-off. Measure retrieval, reranking, and answer generation separately, then evaluate end-to-end latency and accepted-answer cost. Do not accept a faster reranker setting whose misses drive more retries or human review.

Rewrite wording without changing the question

Query rewriting produces an index-facing expression of the user’s request. It can expand a domain abbreviation, make a conversational follow-up self-contained, or express the concept in terminology used by the corpus. Keep the original question alongside the rewrite. The test is whether the transformation improves discovery while preserving the intended subject, constraints, and requested answer.

If Claude generates a search call, implement the rewrite through a user-defined tool’s input_schema, with explicit fields such as query, product_id, and effective_on. These are application-defined fields, not built-in Claude search parameters. Anthropic’s tool-definition guidance documents name, description, and input_schema, and recommends explaining when a tool applies, what it returns, and what its arguments mean.

A rewritten query must preserve exact identifiers and hard constraints. Do not turn “refund policy for plan P-42 effective on 1 September” into “refund policy” and hope ranking restores the missing conditions. Resolve business aliases through an authoritative mapping or ask for clarification; model wording alone cannot prove that an informal name identifies the intended record.

Decomposition splits a compound request into separately retrievable questions. Preserve which original clause each result supports. For independent read-only sub-queries, Claude may return several tool_use blocks in one response, and your executor can run them concurrently for lower latency. Return one tool_result per call, matched by tool_use_id, together in the next user message before any text. When each query must wait for the preceding result, set disable_parallel_tool_use: true inside tool_choice: {"type": "auto"} to allow at most one tool call per response. If one answer supplies an identifier needed by the next query, record that dependency; fixed versus model-directed execution is the choice in 3.6.4. A single strong passage for one clause does not satisfy the entire compound request.

Filter freshness and tune depth together

Validity metadata and index-side eligibility belong to 3.5.4; freshness filtering uses that contract at query time. Resolve the requested as-of date from the question or established conversation context and pass it through the application-defined effective_on field. If the date is ambiguous, ask for clarification rather than silently substituting today’s policy.

Validate the resulting query constraints against the requested subject and date before issuing retrieval. Missing or inconsistent effective-date metadata needs an explicit unresolved outcome; semantic similarity cannot establish validity. Anthropic’s context-engineering article describes timestamps and reference metadata as useful discovery signals, but those signals do not replace the source owner’s applicability fields.

Tune depth after eligibility. If filters eliminate useful candidates, increasing the cutoff does not repair the wrong predicate. Compare configurations using required-source coverage, final relevance, answer correctness, stale-result rate, and latency. Retain the configuration that meets the workload’s acceptance criteria rather than adopting the experimental 150-to-20 recipe as a production constant.

Key distinctions and trade-offs

The stages have different jobs:

Each extra stage consumes operating budget and adds a configuration the team must own. Add stages for demonstrated failure classes and measure their interaction, rather than assuming that combining every technique always improves the service.

Architecture / decision scenario

An education publisher’s staff assistant must return policy explanations within three seconds at the 95th percentile. Its hypothetical evaluation shows 91% accuracy for paraphrases, 70% for course codes, and 64% for questions combining a code with a historical refund condition. A proposal raises final context from 10 to 40 passages for every request.

The retrieval owner chooses hybrid search to recover code-bearing candidates, then reranks a broader candidate set into a smaller final context. The team compares 80-to-12 and 150-to-20 configurations on the same questions. In this hypothetical comparison, overall answer accuracy on the same 300-question set rises from 92% to 93%, but 95th-percentile end-to-end latency increases from 2.7 to 3.5 seconds. The architect selects the 80-to-12 configuration because the 150-to-20 option misses the three-second 95th-percentile target.

The query transformation preserves the course code and expands “the policy then” into the date established in the conversation. Decomposition retrieves the refund condition and the course-specific exception separately. The content owner supplies effective-date ranges, so the filter selects the historical version rather than whichever policy was uploaded last.

The team rejects the universal 40-passage proposal because it does not address missing lexical candidates, dropped constraints, or historical eligibility. The architect owns the three-second trade-off; the retrieval owner owns fusion and depth, and the content owner owns version metadata. Requests with unresolved dates ask for clarification instead of answering from a guessed version.

Common exam traps

• Treating hybrid retrieval and reranking as the same operation.
• Expecting reranking to recover a passage outside its candidate set.
• Copying experimental candidate counts as product defaults or limits.
• Allowing rewriting to discard identifiers, dates, or a request clause.
• Assuming the last uploaded document is the applicable version.
• Increasing final context before diagnosing eligibility and candidate coverage.
• Using retrieval depth to repair an aggregation’s missing completeness guarantee.

What to remember

Improve the stage that fails. Collect complementary candidates, rerank what you actually collected, preserve intent through query transformations, filter by authoritative validity, and tune candidate and final depth separately. Defend the chosen configuration with end-to-end quality and latency measurements.

Knowledge check

References

• Anthropic, Contextual Retrieval

• Anthropic, Define tools

• Anthropic, Effective context engineering for AI agents

• Anthropic, Parallel tool use