Choose delivery by the completion contract

Learning objective

Compare synchronous, streaming, asynchronous, and batch inference.

Core idea

For an architect choosing inference delivery, ask when the caller needs a usable result and how it learns that processing completed. Synchronous inference returns one response on the invocation path. Streaming returns incremental output on that path. Asynchronous inference accepts work and exposes completion later. Batch inference processes a dataset of independent requests as a job.

These labels describe related but different dimensions. A batch job is asynchronous, but one deferred invocation is not automatically a batch. A stream can improve the wait for visible output while still keeping the request active. Select a mode only after verifying that the model supports its operation. This lesson establishes the contracts; see 2.2.2 for SageMaker AI serving choices and 4.1.4 for deadline-based selection.

How it works

Synchronous inference returns one completed response

Amazon Bedrock Converse sends messages to a selected model and returns its response. The response includes output, stopReason, usage, and metrics.latencyMs. The caller waits for that response before it has the complete candidate output. A native-model path can instead use InvokeModel; the model-specific request and response contract belongs to 1.3.7.

This fits a short operation whose result must be checked before the next step, such as extracting a bounded record or classifying an incoming request. The caller can validate the assembled result without first delivering partial text. However, the caller’s timeout and end-user wait must accommodate the generation path. Synchronous delivery is not a promise of a particular latency.

Do not equate successful response delivery with successful inference for the business purpose. stopReason: max_tokens, for example, can indicate an incomplete result. The application needs its completion and acceptance checks regardless of how the response arrived; see 0.5.

Streaming returns progress before completion

Bedrock ConverseStream returns an event stream. contentBlockDelta carries incremental content, messageStop reports the model’s stop reason, and metadata includes usage and latency information. A client should preserve the event contract rather than concatenate every event as if it contained prose. Tool and other content blocks require their own handling.

Streaming is useful when the interface benefits from showing generated text as it arrives. It changes time to visible content, not whether the eventual response is correct or complete. The user can receive a partial answer before a later error or unfavorable stop reason. The system therefore needs an explicit partial-output state and a final acceptance state.

Before selecting this path, check model support. AWS directs callers to GetFoundationModel and the responseStreamingSupported field. The existence of ConverseStream does not mean every model can use it. Applications using native request formats can evaluate InvokeModelWithResponseStream under that model’s contract; see 2.4.4 for delivery through the application interface.

A stream is also not a deferred job. The connection carries ongoing output and can fail while generation is underway. If a final validated record is required before any action, buffer and validate it before acting even if text is streamed for display. Never execute a tool argument assembled from only the first few chunks of an incomplete result.

Asynchronous inference separates acceptance from completion

Bedrock StartAsyncInvoke starts a supported asynchronous invocation. It accepts modelId, modelInput, and outputDataConfig, and returns an invocationArn. That ARN identifies accepted work, not its completed output. GetAsyncInvoke exposes status values of InProgress, Completed, and Failed.

Amazon Nova Reel provides a concrete capability boundary: its video-generation documentation supports the Async Invoke Model API and does not support the Invoke Model API. A request for generated video therefore cannot be made synchronous merely by calling a different operation. Use the documented model path, track the invocation, and retrieve the configured output after completion.

The application must store the work identifier and correlate the eventual result with the request. StartAsyncInvoke.clientRequestToken provides an idempotency token to prevent duplicate requests. Reuse the appropriate identity for a retry of the same submission rather than creating unrelated work every time the caller loses its response. Detailed durable worker design belongs to 2.4.2.

SageMaker AI asynchronous endpoints are a different mechanism with their own limits; see 2.2.2.

Batch inference treats a dataset as the work unit

Bedrock batch inference submits independent prompts as a dataset job and writes results to S3. The work has a job identity plus individual record identities, so job tracking and record acceptance remain separate responsibilities. See 4.1.4 for the JSONL record contract and job states.

Use GetModelInvocationJob to inspect completion. Its PartiallyCompleted status means not all records could be processed in time; inspect the output and reconcile which records still need work. A job-level status does not replace record-level validation of the generated results.

Batch fits offline processing of independent prompts, such as overnight summaries of archived reports. It is not a substitute for an interactive tool loop: the Bedrock batch guide says records are processed independently without multi-turn interaction and does not support tool calling. Check the model and Region’s batch support before choosing this route. A deadline that tolerates deferred delivery must still leave room for queueing, processing, and output acceptance.

Key distinctions and trade-offs

The contracts divide as follows:

First output versus accepted output: Streaming can expose content earlier. A validated complete result can still arrive later, especially when acceptance requires examining the whole response.

One invocation versus dataset: A deferred video generation has one invocation identity. A batch has a job identity plus individual record identities. Their recovery and completion checks have different units.

Timing versus capability: A relaxed deadline does not make an unsupported model eligible for batch. A short deadline does not add synchronous support to a model documented only for asynchronous generation.

Architecture / decision scenario

A training portal needs four functions: classify a short support request, display an explanatory answer, generate a video, and summarize archived reports overnight.

The classifier uses synchronous Converse and validates the bounded result before routing. The explanatory interface uses ConverseStream on a supported model and marks deltas as partial until completion checks pass. The video path uses Nova Reel’s supported StartAsyncInvoke contract, stores invocationArn, and waits for a completed result. The report pipeline creates a batch job with stable per-record identities and reconciles the generated results.

The team keeps four completion states rather than one generic “request succeeded” flag. A dropped answer stream is partial, a returned video ARN is pending, and a partially completed batch requires record reconciliation. Moving the report job to streaming would not fix its offline recovery needs, while moving the live explanation to batch would not meet its interaction requirements.

Common exam traps

• Treating streaming as an asynchronous accepted-work job.
• Equating the first streamed content with a completed answer.
• Assuming an invocation ARN or job ARN proves successful generation.
• Applying SageMaker AI asynchronous limits to Bedrock operations.
• Treating partial batch completion as completion of every input record.
• Treating independent batch records as an interactive tool conversation.
• Selecting delivery mode before verifying model support.

What to remember

Choose the completion contract that matches the task. Synchronous waits for one response; streaming exposes incremental progress; asynchronous separates submission from completion; batch processes an independent dataset. Preserve the correct work identity and validate the result at the appropriate boundary.

Knowledge check

References

• AWS, Converse
• AWS, ConverseStream
• AWS, StartAsyncInvoke
• AWS, GetAsyncInvoke
• AWS, Generating videos with Amazon Nova Reel
• AWS, GetModelInvocationJob
• AWS, Process multiple prompts with batch inference