Choose a vector index by recall and capacity

Learning objective

Choose vector index types: flat, IVF, HNSW, and GPU (CAGRA).

Core idea

An index decides which vectors receive distance calculations and how much work finding those candidates requires. Exhaustive search evaluates every vector; approximate nearest-neighbor (ANN) search avoids some comparisons by searching clusters or traversing a graph. Skipping work buys speed at the risk of missing a true neighbor.

Choose the index against an exact reference, a query workload, and a resource budget. A fast index that misses required passages fails the retrieval requirement. An accurate index that consumes the GPU reserved for generation fails the deployment requirement. Embedding selection and reranking belong to 6.1.3; this lesson holds the vectors fixed and chooses their search structure.

How it works

Establish the exhaustive reference

Milvus calls its exhaustive floating-vector index FLAT. Configure it through prepare_index_params(), add_index(index_type="FLAT", metric_type=...), and create_index(). Its index-specific params is empty: there are no clusters or graph edges to tune. Every query compares against the dataset, making it useful for small collections and exact nearest-neighbor reference results.

Exactness refers to the chosen vectors and metric. It does not prove that the embedding captures business relevance. Use FLAT results to measure how much an approximate index loses, then evaluate answer relevance separately. A nearest-neighbor reference cannot identify a missing source document or repair an unsuitable embedding model.

Reduce comparisons with clusters

The inverted-file family groups vectors around centroids. Milvus IVF_FLAT uses k-means clustering, retains the original vectors, and searches selected clusters. The construction parameter nlist sets the cluster count; the search parameter nprobe determines how many clusters a query visits.

The FLAT suffix means that vectors within the searched clusters are compared without vector compression. It does not make the overall search exhaustive when other clusters are skipped. A neighbor across a cluster boundary can be missed. Increasing nprobe explores more clusters and can recover it, spending more query time. Reserve parameter sweeps for 6.2.2; the selection question here is whether clustering provides useful pruning for your dataset.

Navigate a CPU graph

Milvus HNSW implements a hierarchical navigable small world graph. Upper layers provide coarse navigation; the bottom layer contains all points and refines the search. Graph traversal can deliver low latency and high recall, while storing graph connections adds substantial memory overhead.

Construction uses M for graph connectivity and efConstruction for the candidate pool considered while building connections. Query-time ef controls search breadth. A larger construction budget can produce a better graph, but it costs build time and memory. Increasing query breadth spends latency; it does not rebuild the graph. Treat graph memory and construction time as admission criteria before choosing HNSW for an entire corpus.

Separate a GPU graph from GPU serving

NVIDIA cuVS implements CAGRA, a graph index optimized for GPUs that Milvus exposes as GPU_CAGRA. NVIDIA recommends CAGRA when the index fits in GPU memory and approximate search meets the requirement. The Milvus index uses construction parameters intermediate_graph_degree, graph_degree, and build_algo. The final graph_degree must be smaller than intermediate_graph_degree. These settings shape the graph before queries arrive; search parameters such as itopk_size and search_width govern retrieval through it.

GPU deployment has its own capacity boundary. Milvus loads collection data into GPU memory for GPU searches. Its GPU-index overview estimates GPU_CAGRA memory usage at approximately 1.8 times the original vector data and warns that incomplete loading out-of-memory protection can let excessive data crash a QueryNode. That page limits GPU_CAGRA search limit to 1,024 and excludes range and grouping search from GPU indexes. Validate these requirements before benchmarking throughput.

NVIDIA RAG Blueprint documents Milvus as an optional backend; its default backend is Elasticsearch. NVIDIA cuVS can convert a CAGRA graph to HNSW for CPU search; the Blueprint exposes this GPU-build/CPU-search path through Milvus’s adapt_for_cpu. A GPU-built index therefore does not, by itself, justify reserving a query-serving GPU. See 6.2.2 for the configuration and query-parameter consequences.

Key distinctions and trade-offs

Use the structure that removes enough work while meeting the required accuracy:

GPU acceleration changes the execution resource; approximate search changes candidate coverage. They are separate dimensions. Compare CPU and GPU configurations at the same required recall, query batch pattern, and concurrent load. Otherwise, a throughput comparison can reward a less accurate search or a workload your application never sends.

Architecture / decision scenario

An industrial maintenance assistant must search 8 million equipment passages on premises. The project assumes 1,024-dimensional float32 vectors, a 96 GB RAM allowance, and 80 GB of GPU memory available to Milvus. The raw vectors occupy approximately 32.8 GB; the 1.8-times CAGRA guideline gives approximately 59 GB before temporary search data and other allocations. Its retrieval budget is 80 ms at the 95th percentile, with ANN recall@10 of at least 0.97 against an exact reference. These are project acceptance targets, not product guarantees. One GPU can be assigned to retrieval, but using it removes that GPU from the generation pool.

The retrieval owner measures four candidates on the same held-out queries. In this hypothetical trial, FLAT takes 420 ms, IVF_FLAT reaches 0.94 recall at 45 ms, HNSW reaches 0.98 at 65 ms using a measured 74 GB of its 96 GB RAM allowance, and GPU CAGRA reaches 0.98 at 24 ms. FLAT remains the reference; the tested IVF configuration fails quality; both graphs meet the target.

The tempting proposal is CAGRA because it wins the latency chart. The platform owner shows that allocating its GPU leaves generation below its own capacity requirement. Choose HNSW for this deployment because it meets retrieval requirements without that loss. Reconsider CAGRA when traffic or GPU availability changes, including GPU construction with CPU search as a separate candidate. The decision records accuracy, memory, build duration, and the displaced generation capacity, rather than declaring one algorithm universally fastest.

Common exam traps

• Reading IVF_FLAT as exhaustive search across the entire collection.
• Treating HNSW’s query-time ef as a construction parameter.
• Choosing a GPU index without budgeting loaded data and index memory.
• Assuming CAGRA construction requires every subsequent query to execute on GPU.
• Comparing index speed at different recall targets or query batch sizes.

What to remember

FLAT establishes exact neighbors; IVF prunes clusters; HNSW navigates a hierarchy; CAGRA provides a GPU-oriented graph. Choose from measured accuracy and capacity, including build resources and the hardware that search displaces. Tune the chosen operating point in 6.2.2 and manage its lifecycle in 6.2.3.

Knowledge check

References

• Milvus, FLAT
• Milvus, IVF_FLAT
• Milvus, HNSW
• Milvus, GPU_CAGRA
• Milvus, GPU Index Overview
• NVIDIA, CAGRA
• NVIDIA, Milvus Configuration for NVIDIA RAG Blueprint