Reuse answers within verified freshness scopes

Learning objective

Implement response and semantic caching with freshness, invalidation, and tenant isolation.

Core idea

A response cache returns an answer without repeating the operation that produced it. Exact caching requires the same normalized request contract; semantic caching allows a sufficiently similar prompt to reuse a stored answer. Both require evidence that the answer is still valid for this caller and this version of the system.

Similarity alone does not establish equivalence. Two tenants can ask the same question and require different answers. A document can change while its question remains identical. A tool can require a side effect that returning a previous response would skip.

See 2.4.7 for provider prompt caching, which reuses input computation while still generating output. This lesson concerns stored answers and the lookup rules that determine whether you can reuse them safely.

How it works

Define the exact request contract

For exact response caching, construct an application-owned key from the trusted tenant or access scope, normalized request, model identifier, prompt version, tool configuration, safety-policy version, and relevant evidence revision. Normalize only differences that cannot change meaning. Removing an account number to increase reuse changes the task, not merely the cache key.

Azure Managed Redis can store keyed responses. Microsoft’s cache-aside example uses StringGetAsync for lookup, StringSetAsync to store a value, KeyExpireAsync to set its lifetime, and KeyDeleteAsync for invalidation. These are the .NET Redis-client operations shown in that architecture pattern, not Foundry agent methods.

Apply cache-aside at the model-call or read-only application-operation boundary: look up the scoped key, validate the entry, return a valid hit, and compute and store on a miss. Store the answer with its creation time, evidence revision, and contract version. Those metadata fields are your schema. Do not cache a success receipt as a substitute for executing a requested transaction.

Match semantic requests inside a partition

Azure API Management provides llm-semantic-cache-lookup before the model backend and llm-semantic-cache-store after its response. These policies use a configured external cache. The lookup policy supports OpenAI Chat Completions and Responses, Google Vertex AI, and Anthropic Messages; only Anthropic Messages requires API Management v2 tiers.

Configure embeddings-backend-id to identify the embeddings backend and embeddings-backend-auth="system-assigned" for its documented authentication mechanism. Set score-threshold between 0.0 and 1.0; lower values require higher similarity. Microsoft suggests starting at 0.05 and warns that values above 0.2 can cause mismatches. These are tuning guidance, not guarantees that a hit is correct. Linked access is not checked for embeddings-backend-id: a policy author can route calls through any available backend using the instance’s managed identity, so restrict policy-writing permission as well as cache partitions.

Use <vary-by> to partition entries by a trusted tenant, user group, or subscription scope before reusing output. The documentation’s context.Subscription.Id example is suitable only when that subscription actually matches the intended sharing scope. If one subscription serves multiple tenants, add the verified tenant identity rather than accepting a client-supplied tenant header as authority.

ignore-system-messages="true" removes system messages before comparing prompts. If you choose this optimization, preserve prompt and safety-policy versions in vary-by so changed instructions do not share the same response partition. max-message-count skips caching when the remaining dialogue exceeds the configured count; a multi-turn conversation is not equivalent to its final question alone.

For application-owned semantic caching, Microsoft’s Azure Managed Redis tutorial documents RedisVL SemanticCache, with check, store, ttl, metadata, and filterable fields or separate indexes for partitions. RedisVL distance_threshold uses COSINE distance from 0 to 2. Do not copy an API Management threshold into RedisVL and assume the units are interchangeable.

Bound freshness and invalidate changed contracts

API Management’s llm-semantic-cache-store duration="..." sets entry time-to-live in seconds. The store policy’s cache-response attribute accepts a policy expression to control caching of the current response; when omitted, only HTTP 200 OK responses are cached. Enforce answer eligibility through the storage policy rather than treating transport success as answer validity. RedisVL supplies a default ttl and per-entry overrides. Choose expiration from the source’s change rate and the harm of serving stale content, not from the longest technically possible lifetime.

Expiration does not react immediately to a document correction. For exact entries, update the authoritative source before removing the associated cache key with KeyDeleteAsync. Microsoft’s cache-aside pattern explains that deleting first can let another request reload and cache the unchanged source. Still account for concurrent fills; cache-aside is not a transactional consistency guarantee.

For semantic entries, version the index name, key prefix, filterable partition, or API Management vary-by value when evidence or the response contract changes. This makes preceding entries ineligible for subsequent lookups. Old entries can expire under their retention policy; they must not remain selectable under the updated contract. Version routing is application invalidation, not an assertion that a semantic policy has deleted every related entry.

If deletion is required for privacy or retention, separately remove the stored data from its owning cache. Making an entry unreachable by a different version is not deletion. Avoid semantic caching for personalized, regulated, safety-critical, time-sensitive, or side-effect-dependent work, as the Managed Redis guidance recommends.

Measure correctness as well as reuse

Record cache outcome, partition identifier, model and prompt versions, lookup latency, and similarity score or distance when available. Evaluate similar-looking questions with different entities, negation, dates, and access scopes. A higher hit rate with wrong answers is a regression.

Use API Management tracing to verify gateway cache behavior. Place rate-limit or rate-limit-by-key immediately after lookup, as Microsoft recommends, to protect the model backend when the cache is unavailable. A cache outage can change the workload from mostly hits to mostly model calls without increasing user traffic.

Measure avoided model calls against embedding and lookup overhead. Semantic misses still incur the lookup path before normal generation. Restrict eligible workloads if that overhead exceeds the savings or increases tail latency beyond your service target.

Key distinctions and trade-offs

Exact response caching offers a clearer equivalence contract but fewer hits when users paraphrase. Semantic caching increases reuse but adds embedding cost, threshold tuning, and false-match risk. Provider prefix caching can reduce input cost without trusting a previous answer; it belongs to a different decision.

Time-to-live bounds an entry’s lifetime, whereas invalidation reacts to changed truth or configuration. Tenant partitioning restricts which entries can match; it does not authenticate a caller. Establish identity first, then derive the cache partition from trusted authorization state.

A cache is an optimization over an authoritative source. Keep approval decisions, transaction outcomes, and access grants outside a response-cache hit unless the application explicitly verifies them anew.

Architecture / decision scenario

A support service answers public product questions and reads tenant-specific order status. The product knowledge base changes weekly, and order status can change between requests.

Use semantic caching for public product explanations under a product, evidence-revision, model, prompt, and safety-policy partition. Start with a strict API Management score-threshold; set duration to a business-approved freshness interval. On a knowledge-base correction, switch the evidence revision in vary-by before accepting answers under the revised contract.

Keep order-status requests out of semantic reuse. If their freshness contract permits short exact caching, use a key containing the authenticated tenant, order identifier, and relevant revision, with Redis expiration and explicit invalidation after the authoritative update. Otherwise read the source directly. A refund request always reaches its execution and approval path rather than returning a cached confirmation.

Test cross-tenant questions, corrected product facts, semantically similar questions about different products, and cache outages. Verify that partitions block foreign hits, version changes block stale hits, and rate limiting contains the miss surge.

Common exam traps

• Treating vector similarity as proof that two requests need the same answer.
• Using one global cache for tenant-specific responses.
• Assuming expiration immediately invalidates a corrected document.
• Confusing version-based ineligibility with deletion of stored data.
• Reusing a transaction response instead of performing the requested action.

What to remember

Define eligibility before optimizing hit rate. Use scoped keys for exact reuse, partitions and strict thresholds for semantic reuse, explicit lifetimes for freshness, and deletion or version changes for invalidation. Authenticate the caller before deriving any tenant cache scope.

Knowledge check

References

• Microsoft, Cache-Aside pattern.
• Microsoft, Use Azure Managed Redis as a semantic cache.
• Microsoft, llm-semantic-cache-lookup.
• Microsoft, llm-semantic-cache-store.