Tune retrieval precision and recall

Learning objective

Tune similarity scoring, top-k, metadata filters, and score thresholds across the precision–recall trade-off. This lesson covers candidate selection, not embedding and chunk design, reranking, or product selection.

Core idea

Treat retrieval as a candidate funnel. The similarity metric defines what “near” means, metadata filters define which chunks are eligible, top-k caps how many ranked chunks can pass, and a threshold rejects candidates that are too weak. This is a mental model, not a guaranteed execution order inside every search engine.

Tune the funnel against labeled query–chunk pairs:

• Retrieval precision = relevant retrieved chunks / all retrieved chunks.
• Retrieval recall = relevant retrieved chunks / all labeled relevant chunks.

Precision measures focus; recall measures coverage. Tightening the funnel often raises precision and lowers recall; widening it often does the reverse. The right operating point depends on the cost of noise, missed evidence, and empty results. These definitions follow the standard precision and recall metrics, applied to retrieved chunks at one fixed configuration.

How the controls shape the funnel

Choose one scoring convention

A vector index ranks chunks with its configured distance measure:

• Cosine distance compares vector direction and is 1 - cosine similarity.
• Dot-product distance is the negative dot product, so vector magnitude can affect ranking unless vectors are normalized.
• Squared L2 distance compares geometric separation.

Vertex AI Vector Search notes that dot product with unit L2 normalization produces the same ranking as cosine distance in its index configuration. Use the measure and normalization expected by the embedding configuration.

Do not treat a raw score as a probability of relevance. Some APIs return similarity, where larger is better; others return distance, where smaller is better. The Vertex AI RAG API therefore supports a similarity threshold that keeps scores above the limit and a distance threshold that keeps distances below it, as shown in the RAG retrieval filter reference. A threshold validated for one model, metric, normalization, or corpus is not automatically valid for another.

Apply metadata filters deliberately

A metadata filter removes ineligible chunks before they become evidence. Useful fields include tenant, product, language, region, status, and effective date. Filtering can improve precision when semantic similarity cannot represent applicability. For example, similar instructions can exist for several product versions, but only one manual applies.

Filters also create false negatives when metadata is missing, stale, or too narrow. Validate both the filter expression and metadata coverage. In Vector Search categorical filtering, query tokens use OR within one namespace and AND across namespaces; numeric restricts support comparisons. See Filter vector matches.

Authorization is not a relevance preference. Enforce tenant and document access as mandatory policy. Tune only among permitted chunks.

Tune top-k and the threshold together

Top-k is the maximum number of highest-ranked candidates requested. Raising it can recover relevant chunks, but can also add weak chunks, consume context, and increase downstream work. Lowering it can cut off supporting evidence.

A score threshold sets a minimum match strength. A strict threshold usually improves precision but can return fewer than k results—or none. A loose threshold usually improves recall but admits more noise. Vertex AI RAG Engine exposes both controls: top_k selects the number of top contexts, while its distance filter returns only contexts below the configured distance, as documented in the RAG Engine API.

Tune with a sweep, not one guess:

1. Build an evaluation set with representative queries and known relevant chunks.
2. Hold the corpus, embedding version, scoring measure, and filters constant.
3. Test a grid of top-k and threshold values by query segment.
4. Record precision, recall, empty-result rate, latency, tokens, and answer quality.
5. Choose the smallest candidate set that meets recall and quality targets, then validate it on held-out queries.

Inspect errors before changing a control. If correct chunks rank just below k, raise k. If weak matches pass, tighten the threshold. If the correct version is excluded, repair the filter or metadata. If relevant chunks never rank near the top, changing k alone masks a representation or scoring problem. Each error points to a different control.

Key distinctions and trade-offs

Top-k versus threshold: Top-k limits quantity relative to other candidates. A threshold tests absolute score strength under one scoring setup. Top-k can return k poor matches; a threshold can return zero good-enough matches.

Relevance recall versus ANN recall: Relevance recall asks whether retrieval found the labeled evidence. Approximate-nearest-neighbor (ANN) recall asks whether an approximate index returned the same neighbors as exact search. Vector Search documents index-search controls such as approximateNeighborsCount and fractionLeafNodesToSearch as affecting ANN recall, latency, and cost in its query-time performance guidance. High ANN recall does not prove that the embedding ranks business-relevant chunks.

Filters versus thresholds: A filter expresses applicability using structured facts. A threshold expresses minimum vector closeness. Do not encode product version or access policy as a similarity threshold.

Architecture / decision scenario

A support agent searches router manuals. Wrong-model instructions can cause an outage, while an empty result routes the case to a technician.

The team labels real queries and segments results by model, language, and query length. It uses the metric and normalization expected by its embedding model. Mandatory access rules remain outside tuning; model and firmware metadata restrict applicable manuals.

An initial top_k of 12 gives high recall but fills the prompt with weak passages. A stricter threshold removes noise but creates too many empty results for short queries. The team keeps the filters, lowers top-k to 6, and validates different thresholds for short and detailed queries. It then checks held-out retrieval metrics, answer quality, latency, and prompt tokens.

This preserves enough candidates without treating context as free. If recall is adequate but the best passage remains below weaker candidates, use reranking rather than an ever-larger k.

Common exam traps

• Treating similarity or distance as a calibrated confidence probability.
• Applying a “greater than” threshold to a distance, where smaller is better.
• Assuming a larger top-k always improves the final answer.
• Using a threshold when an exact metadata fact should determine applicability.
• Tuning filters without checking missing or stale metadata.
• Calling high ANN recall proof of high relevance recall.
• Comparing thresholds across embedding models or distance measures as if the numeric scales were interchangeable.

What to remember

• Tune retrieval as a funnel: scoring, eligibility, quantity, and minimum match strength work together.
• Measure precision and recall on labeled queries at a fixed configuration.
• Match threshold direction to the API’s similarity or distance convention.
• Use metadata for structured applicability and policy, not as a substitute for semantic relevance.
• Choose the smallest candidate set that meets recall and answer-quality needs; allow zero results when no evidence is strong enough.

Knowledge check

References

• Google for Developers, Classification: Accuracy, recall, precision, and related metrics
• Google Cloud, Index configuration parameters
• Google Cloud, Filter vector matches
• Google Cloud, RAG Engine API
• Google Cloud, RAG retrieval filter API reference
• Google Cloud, Query a public Vector Search index