Choose corpus embeddings, chunks, and refresh

Learning objective

Choose embedding models, chunking strategies, and index refresh policies that match a corpus and its retrieval workload. Retrieval tuning, reranking, and product selection are outside this lesson.

Core idea

Treat corpus indexing as one versioned contract with three decisions:

• Representation: The embedding model maps a chunk and a query into a vector space where relevant items should be close.
• Boundary: The chunking strategy defines the smallest evidence unit that retrieval can return.
• Freshness: The refresh policy defines how quickly source changes become searchable and how obsolete evidence disappears.

These decisions are coupled. A better model cannot recover a rule split from its exception, and good chunks do not help if their embeddings are stale. Changing the model, output dimension, parser, or chunking rules creates a new corpus version, not a routine content update.

How to choose the contract

Choose an embedding model for the workload

Compare candidates on representative queries and known relevant chunks. Check:

• supported modalities, languages, and domain vocabulary;
• retrieval task support and maximum input length;
• quality on the application’s query–evidence pairs;
• dimension, storage, latency, throughput, cost, and model lifecycle.

Use a retrieval configuration rather than a general semantic-similarity configuration. Google Cloud’s embedding task-type guidance specifies RETRIEVAL_DOCUMENT for corpus content and query-oriented task types such as RETRIEVAL_QUERY or QUESTION_ANSWERING for requests. These asymmetric roles help relate questions to answer passages even when their wording differs.

Keep the corpus and query configurations compatible: pin the model version, task role, and output dimension in index metadata. Vectors from a replacement model belong to a different learned space, even if their dimensions happen to match. A dimension change also requires a correspondingly configured index. Google Cloud documents both configurable output dimensionality and model input limits in Get text embeddings. Smaller vectors reduce storage and computation, but require quality evaluation.

Choose chunks as evidence units

First parse the source into logical structures, then apply a size limit. Prefer boundaries such as a policy clause, a table with headers, or a function with its signature. A layout-aware parser can preserve headings, lists, and tables; Google Cloud’s layout parser documentation describes the resulting chunks as context-aware and semantically coherent.

Use fixed-size token windows when documents lack reliable structure. Tune two separate controls:

• Chunk size balances focus against context. Smaller chunks isolate a fact. Larger chunks preserve qualifications but may dilute the vector and consume more generation context.
• Chunk overlap repeats text across adjacent fixed windows to protect facts near a boundary. More overlap can improve continuity, but it increases embedding and storage cost and can return duplicates.

The embedding model’s input limit constrains usable chunk length; it is not a recommended chunk size, and some APIs truncate oversized input. Google Cloud exposes chunk_size and chunk_overlap as independent token controls in Fine-tune RAG transformations. Measure whole-passage retrieval and boundary failures on the evaluation set; there is no universally correct token count.

Choose a refresh policy from the freshness requirement

Define a freshness service-level objective (SLO), such as “approved policy changes are searchable within 15 minutes.” Then select the least expensive update path that meets it:

Incremental refresh processes only changed, added, or deleted source objects. A full rebuild derives every chunk and vector again. These are update scopes. Batch and streaming describe how updates reach the index; they do not define how often the source is checked.

Use stable source and chunk identifiers, content hashes, source versions, and deletion records so retries are idempotent and removed content does not remain searchable. Verify that the deployed index received an update before declaring the freshness SLO met. Vector Search supports complete replacement, partial batch updates, and streaming upserts, with different propagation behavior, as documented in Update and rebuild an active index.

Key distinctions and trade-offs

Model quality versus vector size: More dimensions increase index storage and computation. A smaller vector is a cost optimization, not automatically a smaller model.

Token chunking versus structural chunking: Token windows control length. Structural chunking preserves meaning. A practical pipeline often uses structure first and a token limit second.

Incremental update versus migration: Source edits can update selected chunk IDs. A representation change requires a versioned rebuild because old and new vectors or chunk definitions must not be mixed silently.

Architecture / decision scenario

A global support assistant indexes product manuals in English, German, and Japanese. Manuals change monthly, while safety notices must appear within ten minutes.

The team evaluates multilingual embeddings on real support questions. It selects a versioned model and dimension that meet quality, latency, and cost targets. Documents use layout-aware chunking: each procedure stays with its heading and warnings, tables retain column headers, and oversized sections split at paragraph boundaries with limited overlap.

The ingestion pipeline sends new or withdrawn safety notices through a streaming update path. A nightly job finds ordinary manual edits, and a weekly reconciliation compares source hashes and removes orphaned chunk IDs. Both jobs deliver corrections through the same update path. For a new embedding model, the team builds a separate full index and switches traffic after validation.

This design spends low-latency capacity where staleness creates risk and prevents a model migration from mixing incomparable vectors.

Common exam traps

• Selecting an embedding model from a benchmark alone without testing the corpus languages, content, and query style.
• Embedding documents and retrieval queries with unrelated configurations.
• Treating the embedding input limit as the ideal chunk size.
• Increasing overlap without accounting for duplicates and cost.
• Choosing streaming updates for every corpus when a batch policy already meets the freshness SLO.
• Updating changed documents but never deleting superseded chunks.
• Mixing vectors from a new embedding model into an existing index instead of rebuilding and switching a versioned index.

What to remember

• Choose representation, chunk boundaries, and freshness as one versioned corpus contract.
• Evaluate embedding candidates on real query–evidence pairs and keep corpus and query configurations compatible.
• Preserve semantic structure first; use token size and overlap as controls, not universal answers.
• Derive refresh cadence from a freshness SLO and source change pattern.
• Use incremental updates for content changes and a validated full rebuild for representation changes.

Knowledge check

References

• Google Cloud, Choose an embeddings task type
• Google Cloud, Get text embeddings
• Google Cloud, Model versions and lifecycle
• Google Cloud, Fine-tune RAG transformations
• Google Cloud, Use Document AI layout parser with RAG Engine
• Google Cloud, Manage Vector Search indexes
• Google Cloud, Update and rebuild an active index