Remove waits from the critical path

Learning objective

Reduce perceived and actual latency with streaming, parallel calls, caching, asynchronous paths, and fewer sequential round trips

Core idea

For an architect improving an interactive Claude workflow, distinguish when useful text appears from when the result becomes complete and acceptable. Streaming helps a person start reading sooner. Parallel execution, prompt-cache reuse, and removing unnecessary round trips can shorten the work itself. A deferred path releases the caller but changes when the business receives the result.

Assume independent work can overlap and latency-tolerant work can be deferred. The additional decision here is how to combine delivery, prefix reuse, and call scheduling without counting an unfinished answer as a completed outcome. See 3.3.3 for enforcing the task’s deadline.

How it works

Stream useful text, then recognize completion

For the Claude Messages API, set stream: true to receive server-sent events. In the Python SDK, client.messages.stream(...) exposes text_stream for text increments. Measure the first nonempty text that the product actually displays, rather than the arrival of any event.

A stream can contain ping events and thinking blocks before answer text. With thinking.display: "omitted", thinking content is not streamed as readable reasoning text. A transport event therefore cannot establish that the user has received something useful. For the UI’s first-text measurement, use text content deltas with delta.type: "text_delta", and include all retrieval and queueing that occurred before the model call.

The stream ends with message_stop; a message_delta carries the stopping reason. An error event can occur during streaming, including overloaded_error. Do not report a partial paragraph as a successful complete answer. Completion accounting must include the final message outcome and any required application checks. A progress display and an accepted recommendation are separate product states.

Streaming does not remove a lookup that runs before inference or a verification call that must finish before acceptance. If a business rule prohibits displaying unverified advice, first useful text is measured after that rule permits display. The delivery strategy must honor that boundary rather than improve the metric by showing unusable content.

Reuse a stable processed prefix

Prompt caching reuses a processed prefix, not a saved answer; see 2.5.1 for cache_control, lifetime selection, eligibility, and warm-up request constraints. Here the decision is whether reuse removes prefix processing from the user-visible critical path under the route’s actual arrival pattern.

A cache miss adds a time-to-first-token penalty. The documented max_tokens: 0 pre-warm can move that processing before the first interactive request, but only if real traffic reads the entry it created. Use the same thinking configuration and output_config.effort for the warm-up and the interactive route: those values are rendered into the prompt, so warming a different configuration can create an entry that production never reads. For a comparison of effort levels, warm each candidate’s own prefix rather than attributing mismatched-cache delay to its reasoning setting.

Measure the arrival gap alongside first visible text and usage.cache_read_input_tokens. A busy route can sustain reuse, while an infrequent route can repeatedly arrive after its entry expires. An illustrative route whose calls usually arrive thirty seconds apart has a different warm-cache expectation from one whose calls arrive twenty minutes apart. Test the first request after an idle period as well as the steady stream; the steady-state result alone cannot support a first-interaction latency claim.

Pre-warming is worthwhile when its timing and configuration make the next interactive request a hit. Assign ownership for refreshing the warm-up after prompt, tool-definition, or reasoning-configuration changes, and confirm reads in the actual requests. The deployment decision should state which arrival conditions the latency target covers and whether a cold request still fits the completion budget. Cache write cost is part of that choice even when the interactive request itself becomes faster.

Shorten the chain while retaining its inputs

Independent reads can overlap, but the application must execute the requested calls concurrently; multiple tool_use blocks alone do not enforce that schedule.

Return all their tool_result blocks together in the next user message, each matched by tool_use_id, before any text content. Splitting completed results across separate user messages does not preserve the documented parallel-call history and can undermine subsequent parallelism. If one requested call is skipped, still return its result with is_error: true and a brief explanation, so the next continuation accounts for the missing input. This preserves a single continuation after the independent read group rather than inventing one response-and-result cycle per result.

Look for predictable inputs that the application can fetch together before generation. When an order identifier already identifies both the order and its published return policy, the integration can supply both together instead of asking Claude to request each in successive turns. Preserve the same authorization and freshness requirements. Do not prefetch a dependent query whose parameters only become known from the preceding result.

Reduce the output work too. Anthropic recommends concise prompts and shorter requested responses. Ask for the few fields the consumer needs rather than an essay. max_tokens is a hard output limit, so indiscriminately lowering it can truncate the response rather than produce a complete concise answer. A response with stop_reason: "max_tokens" reached that limit; increase the allowance or continue the response when the task’s remaining time permits. Measure accepted completion, not time until an artificially short fragment stops.

Place deferred work outside the interaction

Use POST /v1/messages/batches for independent work whose consumer accepts deferred completion, rather than treating an asynchronous SDK call or a submission receipt as a finished deliverable.

The latency delta comes from removing that deliverable from the interactive dependency chain. For example, a fleet-wide aggregation can consume a completed support interaction later without blocking the advice shown now. A verification step required before the advice is accepted cannot move to that later path. Write down which output the user needs immediately and which downstream report can wait, then measure accepted completion with only the genuinely optional downstream work removed. The resulting improvement comes from fewer blocking stages, not faster processing of the deferred analysis.

Key distinctions and trade-offs

Streaming reduces waiting before readable output; caching reduces repeated prefix processing; scheduling removes unnecessary serialized execution. None guarantees unchanged accuracy. A shorter format might omit a needed caveat, and a scheduling change might violate a dependency. Re-run the same task-specific checks after each change.

Treat cold-cache and warm-cache runs as separate operating conditions. A quiet route might rarely read a live entry even when a busy route benefits. State which condition underlies a latency result, and verify it from usage rather than assuming a warm cache.

Architecture / decision scenario

A technology support portal targets first useful text within two seconds and completed advice within seven seconds at the 95th percentile. Its illustrative baseline takes nine seconds to complete: two independent reads are serialized, stable product instructions are processed repeatedly, and a long explanation is withheld until generation finishes. A weekly fleet analysis can arrive the following day.

The tempting shortcut is to move every request to Message Batches. The support architect rejects it because immediate advice is a blocking deliverable. The architect overlaps the independent reads, supplies both results together, caches the stable prefix, requests a compact response, and streams permitted advice. Fleet analysis uses the deferred path separately.

The platform owner tests warm and cold starts and records both first visible text and accepted completion. If streaming meets two seconds but advice still completes after seven, the configuration fails the completion requirement. The support owner also checks that compact responses retain troubleshooting warnings. Release requires both timing targets and preserved answer quality; the optimization is not justified by a smoother display alone.

Common exam traps

• Measuring a ping or thinking event as the first useful answer text.
• Claiming streaming removes pre-inference or verification delay.
• Confusing a processed-prefix cache with an answer cache.
• Launching a cold parallel burst and assuming every request reads the prefix cache.
• Treating an asynchronous client or pending-job receipt as completed business work.
• Cutting max_tokens until incomplete answers appear fast.

What to remember

Measure the first useful display and accepted completion separately. Reuse stable prefixes, reduce needless round trips, and overlap only independent work. Stream what the product permits, verify cache reads, and defer only deliverables whose consumers can tolerate delay.

Knowledge check

References

• Anthropic, Streaming Messages
• Anthropic, Prompt caching
• Anthropic, Parallel tool use
• Anthropic, Reducing latency
• Anthropic, Batch processing
• Anthropic, Handling stop reasons