Measure reuse of stable prompt prefixes

Learning objective

Structure repeatable prompt prefixes and measure provider prompt-cache savings.

Core idea

Provider prompt caching reuses computation for repeated input tokens. It does not return a previously generated answer. Each request still generates its own output, so repeated instructions can benefit even when every user’s question and answer differ.

Azure OpenAI in Microsoft Foundry Models implements this reuse for supported models. Eligibility depends on the rendered prefix, model family, deployment type, and cache controls. Measure the provider’s reported cache tokens rather than inferring a hit from repeated wording or faster responses.

See 2.1.2 for rendering runtime prompts and 2.4.8 for caching completed responses. Here the architectural decision is which content can remain identical at the start of successive model requests, and whether the resulting reads outweigh cache costs.

How it works

Keep the reusable content first

Place stable instructions, reference content, and reusable tool definitions before request-specific evidence and the user’s variable input. Keep conversation context append-only where the interaction permits it. A timestamp, random identifier, or personalized greeting near the beginning can break reuse even if the rest of the prompt is unchanged.

Azure OpenAI requires a prompt of at least 1,024 tokens and identical first 1,024 tokens for prompt-cache reuse. A single-character difference in that initial content can cause a miss. On GPT-5.5 and earlier models, hits beyond the first 1,024 tokens occur in 128-token increments; that rounding does not apply to GPT-5.6 and later model families.

The cache can include messages and tool definitions. Image requests must preserve the image detail parameter across requests. Structured-output schemas are appended as a prefix to the system message, so schema changes can also alter prefix reuse. Stabilize the rendered request contract, not only the text file containing the instructions.

Do not reorder evidence or suppress required personalization solely to reach the token threshold. First preserve correctness and authorized context; then move content that is genuinely reusable. Padding a short prompt creates extra work without establishing useful repeat demand.

Select controls for the model family

On GPT-5.6 and later model families, prompt_cache_key improves matching for requests with common long prefixes. Keep a stable key-to-prefix mapping. Microsoft notes that traffic above approximately 15 requests per minute for one prefix/key combination can cause some misses; partition high-volume traffic across keys while retaining that mapping. This is a cache-reuse consideration, not an inference RPM quota.

Standard pay-as-you-go deployments with these families support explicit breakpoints. Set prompt_cache_breakpoint: { "mode": "explicit" } on a supported content block to include that block and everything before it in the reusable prefix. Responses supports input_text, input_image, and input_file; Chat Completions supports text, image_url, input_audio, and file blocks.

Request-wide prompt_cache_options.mode is implicit or explicit. The default implicit mode includes a breakpoint on the last message and uses explicit breakpoints too. explicit mode uses only supplied breakpoints. With no explicit breakpoint, explicit mode performs no prompt caching or cache-write charging.

Each request can create at most four cache writes. The implicit last-message breakpoint consumes one slot, leaving up to three explicit writes; explicit mode can write up to four. For reads, the service considers up to the last 50 conversation breakpoints. Breakpoints from previous turns are read-only.

These parameters are not portable to every deployment. Models before GPT-5.6 reject prompt_cache_options and prompt_cache_breakpoint with HTTP 400. Provisioned Throughput managed (PTU-M) deployments support prompt caching but not explicit breakpoints or cache_write_tokens reporting. Validate model and deployment support before adding an optimization parameter.

Separate minimum lifetime from retention policy

On GPT-5.6 and later families, prompt_cache_options.ttl accepts only 30m, also its default. It sets a minimum lifetime, not a maximum deletion deadline. prompt_cache_retention does not apply to these families and is deprecated for them.

For preceding model families, prompt_cache_retention selects in_memory or 24h when supported. In-memory caches typically clear after 5–10 minutes of inactivity and are always removed within one hour of their last use. Extended retention is available only on the models listed in Microsoft’s prompt-caching documentation and retains prefixes for up to 24 hours. GPT-5.5 enables extended retention by default; GPT-5.4 and preceding models default to in_memory when no policy is supplied.

Microsoft states that prompt caches are not shared between Azure subscriptions. This subscription scope does not replace application authorization or tenant-specific response-cache partitioning. Do not use a cache lifetime as an approval lifetime, freshness promise, or application retention policy. The prefix cache concerns provider computation; business evidence and deletion requirements need their own controls.

Measure reads, writes, and savings separately

For Chat Completions, record usage.prompt_tokens and usage.prompt_tokens_details.cached_tokens. On GPT-5.6 and later Standard pay-as-you-go deployments, also record usage.prompt_tokens_details.cache_write_tokens. Models before that family have no extra cache-write charge; GPT-5.6 and later families can incur cache-write charges in addition to discounted reads.

Compute a workload-level cached-input fraction as the sum of cached tokens divided by the sum of prompt tokens. Also count requests with nonzero cached tokens. These measure different things: many requests with tiny hits can produce a high request hit rate but little reused input computation.

Estimate read savings from reported cached tokens and the deployment’s applicable normal-input and cached-input rates. Subtract applicable cache-write charges and any change in prompt size. Treat this as an estimate and reconcile it with billed cost using the deployment’s actual rates. Cache reads receive discounted input pricing for Standard deployment types and up to a 100% input-token discount for Provisioned deployment types. Prompt-cache pricing is the same for in-memory and extended retention policies.

Compare cold and reused-prefix requests with the same model, prompt version, output constraints, and task distribution. Record end-to-end latency and output tokens as well as cache usage. Microsoft identifies generated-token count as a major latency driver, so faster output from a shorter answer does not prove improved prefix reuse.

Key distinctions and trade-offs

Provider caching reduces repeated input processing while preserving generation. Response caching avoids a model call by returning stored output. A stable prompt_cache_key improves matching but is not an authorization mechanism or evidence of a hit.

Long reusable instructions can improve cache efficiency, but a prefix used only once can add cache writes and cost without later reads. Explicit breakpoints give supported Standard deployments control over write placement; automatic caching is the appropriate mechanism where those controls are unsupported.

Evaluate net cost and latency together. A high cached-input fraction can coexist with slow generation, large output bills, or excessive write volume. Optimize the repeated prefix only after the task’s prompt and output contract are sound.

Architecture / decision scenario

A support orchestrator sends a shared 1,500-token policy prefix to several specialist agents, followed by customer-specific evidence. One team adds a request timestamp before the policy; another regenerates tool descriptions with unstable ordering. Both report disappointing savings.

Make the policy and tool contract deterministic and place the timestamp and customer evidence after reusable content. For a supported GPT-5.6 Standard deployment, place an explicit breakpoint after that prefix, select explicit mode, and map prompt versions to stable prompt_cache_key values. Partition high-volume keys if traffic for one combination exceeds the documented approximate cache-reuse threshold.

Compare the corrected version against a controlled baseline. Capture prompt, cached, cache-write, and output token counts, plus latency. Calculate read discounts minus write charges using the actual deployment rates. If specialists use PTU-M or a preceding model family, retain the stable prefix but remove unsupported breakpoint controls. The decision succeeds when measured net savings improve without changing the authorized evidence or answer quality.

Common exam traps

• Describing prompt caching as returning a saved answer.
• Expecting a cache hit with fewer than 1,024 prompt tokens or a changed initial prefix.
• Sending breakpoint parameters to models that return HTTP 400 for them.
• Treating 30m as a maximum retention deadline.
• Reporting savings from request hit rate while ignoring token volume and cache writes.

What to remember

Stabilize rendered prefixes, use controls supported by the model and deployment, and read the provider’s usage fields. Measure cached input, applicable write charges, output cost, and latency before claiming prompt-cache savings.

Knowledge check

References

• Microsoft, Prompt caching with Azure OpenAI in Microsoft Foundry Models.
• Microsoft, Latency optimization.