Design hybrid indexes that track source changes

Learning objective

Design dense, lexical, and hybrid indexes with metadata filtering, plus re-indexing for updates, deletions, freshness, and embedding-model changes.

Core idea

An index is a derived view of approved source content, not the source of record. Choose representations for the queries you must answer, then design how that view changes when documents, permissions, or embedding configuration change. A strong similarity score does not establish that a passage is permitted, effective, or still published.

A production design needs both a retrieval strategy and a lifecycle. Dense, lexical, and hybrid retrieval determine which candidates are discoverable. Version identities, publication state, deletion checks, and migration routing determine whether those candidates should still exist and whether queries use the matching representation.

How it works

Build representations for different query behavior

Dense retrieval embeds text into vectors and searches for nearby representations. It helps when a query paraphrases a source. Anthropic does not offer its own embedding model; its embedding guide illustrates a separate provider, Voyage AI by MongoDB. In that Python example, document inputs use input_type="document" and query inputs use input_type="query" with the chosen model.

Lexical retrieval scores terms in indexed text. Anthropic’s contextual retrieval article describes BM25 and the value of combining it with semantic retrieval, particularly for identifiers whose exact spelling matters. Lexical retrieval also depends on how text is analyzed; do not assume every punctuation-sensitive identifier survives preprocessing unchanged. Keep identifier lookup tests in the evaluation set.

Hybrid retrieval combines dense and lexical candidates over the same approved chunks. Deduplicate by stable chunk identity and combine their rankings through a defined application strategy. Raw scores from different retrievers need not have equivalent scales; specify how you combine them rather than adding numbers whose meaning differs. Reranking and query-specific retrieval tactics belong to 3.6; here, maintain the shared searchable corpus they consume.

Filter eligibility within candidate selection

Apply the retrieval authorization policy from 3.2.2 and the requested effective date to both hybrid branches. A lexical branch that forgets the access condition can leak a passage even when the dense branch is correct. Both hybrid branches apply the same eligibility conditions before their candidates are deduplicated and their rankings combined.

For a concrete dense mechanism, Elasticsearch’s knn query accepts filter. The docs describe it as a pre-filter applied during approximate search. Filters elsewhere in the Query DSL tree can instead be post-filters, returning fewer than k results even when enough eligible documents exist. Inspect the deployed query shape; the presence of a filter somewhere is not sufficient.

Eligibility and similarity serve different purposes. A highly similar superseded document can be appropriate for a historical question and wrong for today’s policy. Model freshness as an explicit applicability condition, and preserve a distinct withdrawal state for material that must never be served. Ingestion defines those labels; see 3.5.2.

Replace source versions without splitting the corpus

Record document_id, source version, chunk identities, and processing configuration in an application ingestion ledger. On an update, rebuild affected chunks and any contextual prefixes that depend on the document’s changed meaning. Publish the replacement version only after both dense and lexical representations are ready.

If your platform stores the two representations separately, define a release identifier and query routing that selects a consistent pair. An application-level release is not an automatic cross-database transaction. If the second write fails, keep the replacement unpublished and surface the failed job instead of routing queries through mismatched versions.

Track source-to-searchable lag from the source revision event to visibility in the query path. A completed worker job does not prove that a query can retrieve its output. Probe the serving index with known identifiers and reconcile approved source versions against published index versions. Permission-only changes also enter this lifecycle even when the text hash stays the same.

Remove withdrawn data and migrate embeddings explicitly

Deletion must find every derivative chunk by source identity. In Elasticsearch, POST /INDEX/_delete_by_query removes documents matching a query. The operation uses a snapshot and internal versioning; concurrent changes can produce version conflicts, and completed deletions are not rolled back if later work fails. Inspect conflicts and failures, then verify that serving queries no longer return the target. INDEX represents your actual index name.

A source tombstone in the application ledger can deny retrieval while deletion work completes. Keep it outside the model prompt: telling Claude to ignore already supplied withdrawn text is too late. Apply withdrawal decisions to caches and rollback candidates as well as the active index.

Version model, input_type, output_dimension, output_dtype, preprocessing, and the query encoder together. For supported Voyage models, output_dimension=256 requests shorter Matryoshka vectors; output_dtype selects quantization such as int8, uint8, binary, or ubinary. A change to either output parameter changes the indexed representation even if the model name stays the same, so build a replacement index with a matching query representation. Re-embed the corpus into a replacement index unless the provider explicitly documents compatibility for the chosen configuration. Equal vector dimensions alone do not establish compatibility. Do not alternate query encoders against one mixed index and hope similarity scores remain comparable.

Elasticsearch’s reindex API copies documents and can process them through an ingest pipeline set on dest. Whether that produces replacement-model vectors depends on the pipeline and mapping you configure; build and verify the re-embedding step. Elasticsearch aliases support remove and add actions in one atomic serving-index swap, avoiding downtime or simultaneous routing to both targets when the swap succeeds. Check the response’s errors and action_results: the aliases docs show that a failed action can coexist with a successful one. Verify routing rather than assuming acknowledgment proves the intended swap.

Key distinctions and trade-offs

The following choices incur different maintenance costs:

A document revision, an access revocation, and an embedding migration are different events. The first changes source meaning, the second changes eligibility, and the third changes representation. All can require publication work, but only the text or representation changes necessarily need an embedding call.

Hybrid quality gains have a continuing cost: more derived state to maintain and more paths to validate. Choose it from retrieval evaluations, not as an automatic requirement for every corpus. Retain historical source versions only when the business needs them and the authorization policy permits them.

Architecture / decision scenario

A financial-services assistant searches product explanations and numbered policy clauses. Its platform owner must publish revisions within five minutes and stop serving withdrawn clauses within one minute. A dense-only trial misses clause identifiers; a hybrid trial improves identifier recall, but one lexical replica still serves an obsolete version.

The architect rejects both a dense-only shortcut and an uncoordinated hybrid rollout. The replacement design publishes one corpus release only after both branches are ready, applies identical effective-date and access predicates, and probes visibility from the serving path. Withdrawals enter an immediate deny list while deletion tasks remove derivative records and report conflicts.

For a planned embedding migration, the retrieval team builds a separate index and query encoder, compares metrics, and validates the alias response and serving probes before switching traffic. Rollback uses only versions that also honor withdrawals. The operations owner measures the five-minute and one-minute requirements independently; a fast normal update cannot excuse a late revocation.

Common exam traps

• Trusting semantic similarity to enforce permissions or select effective versions.
• Filtering only one branch of hybrid search.
• Calling every query filter a pre-filter.
• Assuming a reindex re-embeds documents without a configured embedding step.
• Treating deletion submission or alias acknowledgment as verified serving-state completion.

What to remember

Choose representations from query behavior, then operate them as one approved corpus. Filter eligibility during candidate selection, verify updates and deletions from the serving path, and migrate document and query embeddings together. Source identity and publication state are as important as relevance scores.

Knowledge check

References

• Anthropic, Embeddings
• Anthropic, Introducing Contextual Retrieval
• Elastic, kNN query
• Elastic, Reindex indices
• Elastic, Delete documents by query
• Elastic, Aliases