Evaluate embeddings as a retrieval contract

Learning objective

Evaluate embedding selection, dimensionality, metadata, and representation quality.

Core idea

For an architect selecting a retrieval representation, an embedding model is part of the index contract. It converts particular input content into a particular vector space. The query must use a compatible representation, and the index must declare the matching dimensions and similarity metric. A successful embedding request proves that a vector was produced, not that it retrieves useful evidence.

Evaluate models and dimension settings against the questions, languages, and document types your agents must handle. Preserve source text and metadata alongside vectors so you can authorize, inspect, cite, and filter retrieved evidence. A vector alone does not explain which policy version it represents.

See 2.2.8 for chunk boundaries. Here, hold those boundaries stable while comparing representations; otherwise, an apparent model improvement might come from a different evidence unit.

How it works

Define the embedding-space contract

Microsoft’s Azure OpenAI embedding example calls openai.embeddings.create(model=..., input=...) and reads response.data[0].embedding. The model value is the Azure deployment name, which need not equal the underlying model name. Record both in your deployment configuration so an index rebuild uses the intended representation.

Azure AI Search describes an embedding space as vector fields populated from the same embedding model. Use the same compatible model and dimension setting for documents and queries. Two models producing equal-length vectors do not thereby produce coordinates with the same meaning. A length check can catch a shape error without detecting this semantic mismatch.

For integrated indexing, the Azure OpenAI Embedding skill exposes deploymentId, modelName, and dimensions. The skill’s dimensions must equal the target vector field’s dimensions. Query vectorization must use the corresponding representation too. When changing the model, build and evaluate a compatible corpus rather than querying vectors produced by one model with another model’s outputs.

Choose dimensions from measured quality and capacity

The Embedding skill documents these supported ranges:

For supported models, the skill defaults to the maximum dimensionality. Reducing dimensions changes the representation and can reduce the raw vector data you store and move. It does not establish a fixed reduction in service cost or an acceptable quality level. Compare measured retrieval behavior at the dimension settings you can afford.

Capacity controls also include vectorSearch.compressions with scalarQuantization (using scalarQuantizationParameters.quantizedDataType="int8") or binaryQuantization, and vector-field stored=false to omit an extra retrieval copy. These storage choices differ from reducing embedding dimensions; see 2.2.10 for rescoringOptions and evaluation of compressed-vector retrieval.

In Azure AI Search, a typical floating-point vector field uses Collection(Edm.Single), a declared dimensions, and a vectorSearchProfile. For Azure OpenAI embeddings, configure the algorithm’s metric as cosine. Keep the metric tied to the embedding provider’s documented representation rather than choosing it because another index used it.

Dimension changes affect indexed documents and incoming queries together. Test the full path, including generated query vectors, ingestion failures, and expected source matches. A migration that updates only a skill or only a field leaves an incompatible system even if each edited definition looks reasonable in isolation.

Preserve metadata outside the vector

Azure AI Search requires a document key, represented by an Edm.String field with key=true. Add human-readable content for answer generation and inspection. The service cannot reconstruct source text from a vector, and semantic ranking works on textual fields rather than embedding coordinates.

Store the source identifier, chunk position, document version, language, and applicable access metadata as separate fields according to your retrieval contract. These names are application schema choices, not reserved Azure fields. Mark a field filterable when the query must constrain it and retrievable when downstream evidence handling needs its value. Vector fields themselves cannot be filterable, facetable, or sortable.

Embedding a product name into text can help similarity retrieval, but it does not replace an exact version filter. Similarly, text describing an access group does not authorize the document. See 2.2.11 for source authorization. Keeping access and version metadata separate lets you enforce those constraints before deciding which relevant passage to use.

Consider whether titles and section labels belong in the embedded text. They can disambiguate short passages, but unrelated boilerplate can dominate many chunks. Treat this as a representation experiment: preserve the original evidence, vary the embedded text deliberately, and compare the resulting retrieval errors.

Evaluate the evidence that the representation retrieves

Create an application evaluation set with known relevant passages. Include paraphrases, domain vocabulary, exact identifiers, multilingual questions where applicable, and requests with no supported answer. Measure whether the relevant evidence appears among the candidate results, its position, and whether it retains the detail needed to answer.

Hold the corpus, chunking, query settings, and authorization constraints constant when comparing model or dimension candidates. Use Azure AI Search’s vectorQueries.exhaustive=true to run exhaustive nearest-neighbor search when you need a reference comparison against approximate retrieval on an HNSW-indexed field. This helps distinguish a representation problem from an approximate-index recall problem; see 2.2.10 for query controls.

Inspect the returned source fields, not only @search.score. A high similarity score can identify a related policy that applies to the wrong product release. Separate relevant-topic matching from answer-bearing evidence, and inspect failures by document family before accepting an aggregate improvement.

Input validation is part of representation quality. The direct Azure OpenAI embedding guide states an 8,192-token per-input limit, while the Azure AI Search Embedding skill documents an 8,000-token maximum. Apply the constraint of the interface you use rather than importing the direct API limit into the skill pipeline.

Key distinctions and trade-offs

Model identity versus vector length: Dimensions specify shape. The model and preprocessing specify the meaning of coordinates. Both must agree between indexing and querying.

Representation quality versus search approximation: An embedding can encode the wrong emphasis even when exhaustive search finds its exact nearest neighbors. Approximate retrieval can miss a useful neighbor even when the representation is suitable.

Embedded context versus filter metadata: Context in the text helps similarity. Separate filterable fields enforce exact selection and access constraints. Neither mechanism replaces the other.

Higher dimensionality provides a different capacity-quality trade-off, not a universal quality guarantee. Additional metadata improves control and diagnosis but requires consistent ingestion and refresh. Compare the operational cost with evidence quality rather than selecting the largest vector automatically.

Architecture / decision scenario

A product-support team compares text-embedding-3-small at 1,536 dimensions with two supported dimension settings of text-embedding-3-large. It freezes manual chunks, embeds the same titles and passages, and creates compatible vector fields and query configurations for each candidate.

Its evaluation includes paraphrased symptoms, exact error codes, translated questions, and policies for two product versions. Filterable version and access metadata restrict the candidate corpus; retrievable source text and identifiers show whether a match actually supports the answer. The team compares ranking positions and answer completeness, rather than averaging unrelated similarity scores across candidates.

When one candidate loses a useful passage, exhaustive search reveals whether approximate indexing caused the miss. When exhaustive search still prefers the wrong passage, the team inspects representation and source text. It chooses the dimension setting that meets its measured retrieval target within its resource budget, then rebuilds documents and query vectorization together.

Common exam traps

• Assuming equal dimensions make vectors from different models compatible.
• Passing a model family name where the request expects an Azure deployment name.
• Changing skill dimensions without matching the vector field and query path.
• Treating an embedding as a recoverable copy of the source passage.
• Replacing authorization or version filters with embedded natural-language labels.
• Applying the direct embedding API input limit to the Embedding skill.

What to remember

Evaluate the model, dimensions, embedded content, and metadata as one representation contract. Keep document and query vectors compatible, retain inspectable source fields, and separate approximation errors from representation errors. Choose from measured evidence quality under the actual interface’s constraints.

Knowledge check

References

• Microsoft, Generate embeddings with Azure OpenAI
• Microsoft, Azure OpenAI Embedding skill
• Microsoft, Create a vector index
• Microsoft, Create a vector query