Budget evidence before injecting retrieved context

Learning objective

Retrieve and inject relevant context under explicit token, provenance, and relevance budgets

Core idea

Retrieval produces candidates; context assembly decides which candidates become evidence for a response. An architect must budget three things independently: the tokens that fit, the sources that can support the answer, and the relevance needed for this task. Returning five documents is not a five-document guarantee of useful grounding.

Microsoft Foundry’s Azure AI Search tool exposes controls for candidate count, query type, and filtering. Azure AI Search exposes lower-level query controls when the application needs to assemble context itself. Foundry memory provides another retrieval path for learned user context. None of those candidate counts replaces a token budget over the rendered input.

See 0.8 for the distinction between similarity and support. This lesson concerns selecting and injecting retrieved material; memory authorization belongs to 2.2.6.

How it works

Allocate the inference budget first

Reserve space for trusted instructions, the current question, required recent history, and output before allocating retrieved evidence. See 0.2 for inference token budgeting. Count the evidence after formatting, because source labels, entity identifiers, excerpts, and delimiters also consume input tokens.

Use the selected model’s tokenizer for a precise application budget where available. Agent Framework’s CharacterEstimatorTokenizer uses a four-characters-per-token heuristic and accepts replacement by a model-specific tokenizer through TokenizerProtocol. A character estimate is useful for planning, but should not be described as an exact token measurement.

Make the remaining evidence allowance explicit in the context assembler. If the candidates exceed it, select smaller supported passages or retrieve more targeted evidence. Cutting the final serialized string at an arbitrary position can sever a citation from its passage or remove the exception that changes the answer.

Choose candidate controls deliberately

For the Foundry Azure AI Search tool, project_connection_id identifies the project connection and index_name selects the index. top_k defaults to 5. query_type defaults to vector_semantic_hybrid; documented alternatives include simple, vector, semantic, and vector_simple_hybrid. The tool’s filter applies to every query the agent makes to that index.

These controls determine the retrieval path and candidate set. top_k is a result-count control, not a maximum number of tokens. Five long chunks might exceed your allowance, while several short chunks might fit comfortably. If a hard, inspectable passage budget is required, use an application-controlled retrieval and assembly stage rather than assuming that a managed tool’s result count proves the bound.

The tool targets one index, and the Foundry agent and search resource must be in the same tenant. Its prerequisites include retrievable content and a retrievable source URL field, optionally with a title, for linked citations. Do not design a single tool definition that silently searches unrelated indexes or tenants.

With direct Azure AI Search queries, vectorQueries.k controls vector matches and top controls returned results for hybrid queries with a search parameter. The documented default result count when unspecified is 50. When using semantic ranking with vectors, Microsoft recommends k=50 because the semantic ranker accepts up to 50 candidates. This candidate pool is distinct from the Foundry tool’s returned-result control, top_k, and from your injected-token allowance. select chooses returned fields; use it to return the content and source metadata needed for evidence rather than unnecessary fields.

Apply relevance and provenance gates

Nearest-neighbor retrieval can return the requested number of results even for an off-topic query. A complete result list is therefore not proof that any passage answers the question. Require the selected passage to support the particular claim, preserve qualifications, and match the entity or policy version being discussed.

Azure AI Search documents vectorQueries.threshold as preview. The form {"kind": "vectorSimilarity", "value": 0.8} is an example threshold, not a generally valid relevance cutoff. Microsoft distinguishes pure single-vector score thresholds from hybrid ranking, whose fused score ranges differ. Calibrate the retrieval configuration against representative questions; do not copy a vector cutoff into a hybrid path and infer comparable meaning.

Keep each retained excerpt with its source ID, URL or record locator, and relevant version metadata. The context assembler should preserve that association when it deduplicates overlapping passages. Two overlapping chunks from the same document do not provide two independent sources.

For the Foundry tool, verify citations in the generated response and inspect url_citation annotations when streaming. The documented citation form is [message_idx:search_idx†source]. Validate that the linked source actually supports the answer; displaying a citation is not itself a relevance check.

Retrieve learned context separately

Foundry memory is preview and is intended for learned long-term context. Python’s project_client.beta.memory_stores.search_memories() accepts a store name, scope, and MemorySearchOptions(max_memories=5) to bound the number of memories returned. Again, the count does not cap the rendered tokens.

For static user profile context, search_memories with scope and without items or previous_search_id returns profile memories. For contextual retrieval, supply recent messages as items. Retrieve stable preferences near conversation start and task-relevant historical summaries when needed. Keep those memories distinguishable from curated policy documents: a remembered preference cannot establish the company’s refund policy.

Key distinctions and trade-offs

Candidate count, relevance, and tokens measure different things. Increasing top_k can improve coverage but also add redundancy and cost. Reducing the passage size can fit more sources but omit a decisive exception. Tightening a relevance gate can improve precision while leaving the question unsupported.

Provenance survives only if it accompanies the retained content. Carrying a list of URLs elsewhere in the prompt does not show which URL supports which claim. A practical evidence packet binds the excerpt, source, and version before injection.

Authorization precedes assembly. Apply the permitted search filter or memory scope before material enters model context, then perform relevance and token selection on the allowed results. Removing unauthorized passages from the answer does not undo their exposure during inference.

Assembly therefore filters for authorization first, then selects for relevance and the token allowance, and binds each excerpt to its source before injection.

Architecture / decision scenario

A warranty assistant must answer whether a particular device qualifies for a replacement. Its application grants 2,000 tokens to retrieved context after reserving instructions, recent interaction, and response space. That figure is an application budget, not a service quota.

The assembler queries the permitted index with the device family and policy version. A managed prototype uses top_k=5 and query_type="vector_semantic_hybrid"; the production path uses direct retrieval when exact passage accounting is required. It retains content, source URL, policy version, and device-family metadata through select.

The candidates contain a general warranty, an exception for refurbished devices, and several near-duplicates. The assembler retains the general rule and applicable exception, removes repeated material, and counts the complete evidence packet. If both passages cannot fit without losing their meaning, it asks a narrower question or abstains from the eligibility decision.

A scoped memory lookup supplies the customer’s preferred language, bounded separately by max_memories. It never replaces the policy source. The response cites the policy passages and explains which exception applies. Acceptance requires correct decisions for supported and unsupported devices, intact claim-to-source links, and a packet within the token allowance.

Common exam traps

• Treating top_k, k, or max_memories as token caps.
• Assuming nearest neighbors necessarily answer the question.
• Presenting preview vector thresholds as a universal hybrid cutoff.
• Deduplicating text while dropping its source linkage.
• Using a remembered preference as organizational policy.
• Filtering sensitive evidence only after generation.

What to remember

Retrieve candidates, authorize them, select support, and measure the rendered evidence packet. Count controls improve retrieval discipline; provenance and relevance gates determine whether the retained material is usable. Reserve a token allowance explicitly and preserve decisive qualifications when fitting evidence into it.

Knowledge check

References

• Microsoft, Azure AI Search tool
• Microsoft, Create a vector query
• Microsoft, Create and use memory in Foundry Agent Service
• Microsoft, Compaction