Cut passages without losing their meaning

Learning objective

Choose chunking strategies — fixed-size with overlap, structure-based, sentence-based — and enrich chunks with surrounding context.

Core idea

A chunk is both a retrieval unit and a possible source for an answer. Choose its boundaries around the information a question requires, then measure whether retrieval can find that unit. Token counts are useful controls, but they cannot tell you whether a passage contains its own subject, qualifiers, and exceptions.

The architect balances precision, surrounding context, ingestion cost, and traceability. A tiny chunk may match a question precisely while omitting the condition that changes the answer. A large chunk may preserve the condition while mixing topics and consuming the space needed for other sources. There is no universal size that resolves both problems.

How it works

Establish a fixed-size baseline with overlap

Fixed-size splitting caps each chunk by a measured unit such as tokens. Overlap repeats part of the preceding chunk in the next one, reducing the chance that a sentence or local explanation becomes unusable at a boundary. It is a reproducible baseline for prose whose structure cannot be recovered reliably.

Treat chunk size and overlap as application configuration, not Claude API settings. For example, a trial can use 400-token chunks with 80-token overlap. These are experiment values, not Anthropic limits or recommended defaults. Success depends on the corpus and the relevance labels used to evaluate it.

With stride equal to size minus overlap, that example advances 320 tokens per chunk. Away from document ends, it stores roughly 400/320, or 1.25 times, the original text volume. That duplication increases embedding input and can fill an answer context with near-identical passages. Retrieval evaluation must not reward repeated copies of the same supporting fact as independent coverage.

Overlap is a local repair. It cannot reconnect a footnote several pages away or make a table row meaningful if the column headers lie outside the repeated region. Increase overlap only when measured boundary errors justify the extra text, not as a substitute for preserving document structure.

Use structure where it carries meaning

Structure-based splitting follows headings, subsections, list items, or tables. Carry the heading path with the resulting passage so a clause about eligibility is not detached from the product or jurisdiction it describes. Keep access boundaries from 3.5.2 intact; context expansion must not merge material with different entitlements.

Structure alone does not bound size. A long subsection needs secondary splitting, while a short heading with one sentence may need its immediate parent context. Split a large table by coherent row groups and repeat the relevant caption, headers, and units. Do not concatenate unrelated sections merely to hit a preferred token count.

Preserve source locators through these transformations. If repeated headers and added headings help retrieval, mark them as derived representation rather than pretending they are contiguous text in the original source. The ingestion owner should be able to reconstruct which source ranges formed each chunk.

Group complete sentences when structure is weak

Sentence-based splitting preserves sentence endings and groups sentences until a target size is reached. It is useful for narrative prose and transcripts with weak heading structure. Compared with blind token cuts, it avoids splitting an individual assertion, but it does not guarantee that a group forms one topic.

A sentence may begin with “this exception” or “they,” referring to the preceding group. Abbreviations, code, decimal numbers, and OCR artifacts can also confuse sentence detection. Keep representative parser fixtures, inspect such failures, and retain surrounding sentences where they are necessary to interpret the selected unit.

Sentence grouping can operate inside structure-based sections. You do not need one splitter for every content type. Use the simplest reproducible boundary rule that preserves the unit of meaning and meets measured retrieval quality.

Enrich each chunk with surrounding context

Deterministic enrichment can add a title, section path, nearby sentences, or a table caption. Anthropic’s Contextual Retrieval takes a different approach: Claude receives the document and a selected chunk, generates a short explanation specific to that chunk, and the pipeline prepends it before both embedding and BM25 indexing.

The article describes contextual text as usually 50–100 tokens. That is the experimental pattern’s typical length, not a service limit. A generic document summary is not interchangeable with an explanation identifying the particular chunk’s subject, period, or relationship to the rest of the document.

Control repeated document processing with prompt caching of the reference-document prefix; see 2.5.1. The article estimates a one-time contextualization cost of $1.02 per million document tokens, assuming 800-token chunks, 8,000-token documents, 50 tokens of context instructions, and 100 context tokens per chunk. This is the article’s estimate under those assumptions, not a tariff or budget guarantee.

An alternative is Voyage’s voyage-context-4, called with contextualized_embed(), which produces chunk vectors with full-document context without manual metadata augmentation. Its 120,000-token limit applies when enable_auto_chunking=true; otherwise all inputs together are limited to 32,000 tokens. Unlike a generated textual prefix, this approach changes the vectors only: a BM25 index still scores the bare text unless you enrich it separately.

Keep the raw passage and the generated contextual prefix separate in storage. Inspect prefixes against the source: enrichment can introduce unsupported claims or attach the wrong date. The source text remains authoritative. If a revision changes the document-wide meaning, an unchanged local chunk can still need a regenerated prefix. Version the enrichment process with the parser and splitter, and evaluate it before publishing replacement representations.

Key distinctions and trade-offs

Use the following comparison to choose a trial strategy:

Generated context changes the indexed representation; retrieving a larger neighboring passage changes what you return after a hit. They solve different problems. The former can make a chunk discoverable, while the latter can complete an answer after discovery. Do not assume that one automatically replaces the other.

Compare candidates at a consistent delivered context budget, not only at the same chunk count. Ten large chunks and ten small chunks expose very different input volumes. Evaluate both relevance and complete answer support; see 3.5.5. Keep exact table values and identifiers in the indexed text, even when semantic enrichment improves paraphrase matching.

Architecture / decision scenario

An education publisher needs answers over textbooks, narrative lecture transcripts, and fee-policy tables. Its retrieval context budget is 6,000 tokens. In a reviewed test set, 14% of errors come from table rows without units and 9% from transcript passages whose subject appears in a preceding sentence.

The platform lead proposes 400-token chunks with 80-token overlap everywhere. This provides a fast baseline but cannot reliably include a distant table header, and duplicate overlaps spend the limited answer budget. Replacing every chunk with a full chapter would preserve context but leave room for too few subjects.

The architect keeps textbook subsection boundaries, splits fee tables by row groups with repeated headers, and groups transcript sentences with limited neighboring context. A separate trial prepends document-aware contextual explanations before dense and lexical indexing. The publisher’s content owner reviews generated prefixes for wrong subjects and dates; the retrieval owner compares recall and accuracy under the same 6,000-token budget. The architect compares cached contextual-prefix generation with contextualized_embed() and its vector-only gain; enrichment ships only if measured quality offsets ingestion and validation costs.

Common exam traps

• Calling one chunk size optimal without a corpus-specific evaluation.
• Using overlap to repair a missing distant header or cross-reference.
• Assuming every sentence group is a self-contained topic.
• Treating generated context as source authority or an access label.
• Comparing strategies at equal chunk count but unequal context volume.

What to remember

Choose boundaries that preserve meaning, then measure findability and complete answer support. Overlap repairs local cuts, structure preserves source relationships, and sentence grouping protects assertions. Contextual Retrieval adds a chunk-specific prefix to both embedding and BM25 input, with its own validation and versioning cost.

Knowledge check

References

• Anthropic, Introducing Contextual Retrieval
• Anthropic, Embeddings
• Anthropic, Prompt caching
• Microsoft, Chunk large documents for vector search solutions