AWSAWS Certified Generative AI Developer - Professional2026-102.2 Implement model deployment strategies
Match inference modes to caller deadlines
Completed by the knowledge check
Learning objective
Choose SageMaker real-time, serverless, asynchronous, or batch inference by latency and job size.
Core idea
Choose the response contract before the compute configuration. An interactive caller waits for a response. An asynchronous caller submits work and retrieves a result later. An offline dataset needs a job that processes records without maintaining a request-serving endpoint.
Amazon SageMaker AI has a deployment mode for each pattern, but they are not interchangeable ways to run every foundation model. Hardware support, memory, payload size, and processing time eliminate unsuitable modes before cost enters the decision. In particular, SageMaker Serverless Inference does not support GPUs; irregular demand does not make it eligible for a GPU-dependent model.
Distinguish a request’s payload from a dataset’s total size. A terabyte of records can be processed through bounded batch requests. One large indivisible input can require asynchronous inference even if the daily dataset is small.
How it works
Serve interactive requests with real-time inference
Real-time inference fits interactive, low-latency workloads. The caller invokes the deployed endpoint through InvokeEndpoint, identifying EndpointName and the container’s input format through ContentType. SageMaker AI’s feature matrix documents GPU support and autoscaling for this mode.
For the ordinary synchronous request path, plan within the documented 60-second response boundary and a payload smaller than 6 MB. A long-running report that cannot complete within the synchronous contract needs a different path, even when the model itself fits the instance.
Interactive latency remains a measured property of the deployed model. Choosing real-time inference does not remove model loading, saturation, or token-generation costs. Use 2.2.5 to size the container and 2.2.1 to compare capacity economics.
Use serverless inference only for eligible models
On-demand Serverless Inference launches compute when needed and scales to zero during idle periods. It fits intermittent workloads that can tolerate cold starts. Startup depends on model download, model size, and container startup.
Configure the deployment with CreateEndpointConfig and a production variant’s ServerlessConfig. MemorySizeInMB accepts 1024 through 6144 MB in 1024-MB steps. MaxConcurrency accepts 1 through 200 concurrent invocations per endpoint; that configurable maximum is not proof that your account’s regional quota grants the same total capacity.
Serverless requests and responses are limited to 4 MB, and the container’s invocation timeout is one minute. The mode excludes GPUs and VPC configuration. A large GPU-hosted language model is therefore not a serverless candidate merely because it receives occasional traffic.
For eligible models with predictable bursts, ProvisionedConcurrency keeps the configured amount of serverless capacity warm. It must be no greater than MaxConcurrency and adds provisioned-capacity charges. This is SageMaker Serverless Inference provisioned concurrency, a different resource from Bedrock Provisioned Throughput and from Lambda provisioned concurrency.
Queue individual jobs with asynchronous inference
Asynchronous inference decouples submission from completion. AsyncInferenceConfig marks the endpoint configuration for asynchronous invocations. For large inputs, store the payload in Amazon S3 and pass its URI in InvokeEndpointAsync.InputLocation. The service queues the request and returns locations for output and failure artifacts; the application must distinguish acceptance from a completed result.
The mode supports payloads up to 1 GB and processing up to one hour. The per-request InvocationTimeoutSeconds controls processing time: its default is 900 seconds and its maximum is 3600 seconds. The per-request RequestTTLSeconds controls how long the request can wait in the queue, with a default and maximum of 21,600 seconds. Increasing processing time does not extend the queue deadline.
The API also accepts an inline Body of up to 128,000 bytes. Body and InputLocation are mutually exclusive. Use the S3-backed path for large requests; do not state that every asynchronous request must upload an object first.
Asynchronous endpoints support GPU instances and scaling to zero. That makes them candidates for intermittent GPU work when callers accept queue and startup delay. It does not give the caller an interactive streaming answer.
Process offline datasets with batch transform
Batch transform runs inference over a dataset without a persistent endpoint. CreateTransformJob defines the job. SageMaker AI distributes input S3 objects across the configured instances and writes output files to S3.
Input layout matters. If you supply one input file and several instances, only one instance processes that file; the others remain idle. Adding instances does not automatically parallelize one object across them. Partition the dataset into appropriate objects when you need distributed work.
For splittable records, SplitType: Line divides input files into mini-batches. BatchStrategy and MaxPayloadInMB control request grouping. MaxPayloadInMB must not exceed 100 MB, and MaxConcurrentTransforms * MaxPayloadInMB must also not exceed 100 MB. These are mini-batch request constraints, not a 100-MB limit on the entire offline dataset.
Verify that the container supports the selected record format and batching behavior. A model that runs behind an endpoint is not automatically configured for a useful batch-transform data contract.
Key distinctions and trade-offs
The following choices start from the caller’s required outcome:
| Caller requirement | Candidate mode | Deciding boundary |
|---|---|---|
| Interactive response from a GPU-hosted model | Real-time | Synchronous latency and payload contract |
| Sporadic small CPU model requests | Serverless | No GPU; 6 GB maximum RAM; cold-start tolerance |
| Large individual request with delayed completion | Asynchronous | Up to 1 GB input and one hour processing |
| Offline inference over many records | Batch transform | Dataset partitioning and bounded mini-batches |
Asynchronous versus batch: A queue of independently submitted requests differs from a job over an identified dataset. Both can write S3 results, but the application’s completion and ownership contracts differ.
Warm capacity versus eligibility: Provisioned concurrency reduces serverless cold starts; it does not add GPU support or expand the payload limit.
Architecture / decision scenario
A document platform needs four inference paths. Its chat assistant uses a GPU model and must respond interactively, so the team selects real-time inference and tests the synchronous request budget. A small CPU classifier receives sporadic short inputs; it fits serverless memory and payload limits, and the caller accepts cold starts.
Large uploaded documents need several minutes of processing. The platform invokes an asynchronous endpoint with InputLocation, sets InvocationTimeoutSeconds, and tracks the returned output or failure location instead of holding a synchronous connection open. A nightly archive contains millions of independent records, so the team uses batch transform and partitions S3 objects to keep multiple instances busy.
The architect rejects “serverless for everything” because the chat model requires GPUs. The team also rejects sending the nightly archive as one asynchronous request: the dataset is independently partitionable and has an offline job boundary.
Common exam traps
- Choosing serverless for a GPU model because demand is intermittent.
- Confusing asynchronous acceptance with successful inference completion.
- Treating processing timeout and queue lifetime as the same setting.
- Applying mini-batch payload limits to an entire dataset.
- Expecting multiple batch instances to share one input S3 file.
- Assuming provisioned concurrency changes serverless feature exclusions.
What to remember
- Start with interactive response, delayed result, or offline dataset.
- Eliminate modes using hardware, memory, payload, and timeout constraints.
- Asynchronous inference has separate queue and processing deadlines.
- Serverless warm capacity does not enable GPUs.
- Batch-transform object layout determines usable parallelism.
Knowledge check
A chat assistant on a GPU-hosted model receives only a few dozen requests a day, and each user waits for the answer. To avoid paying for idle compute, an engineer proposes SageMaker Serverless Inference with ProvisionedConcurrency to hide cold starts. Which mode should the team choose?
ProvisionedConcurrency set to the expected peakMemorySizeInMB raised to 6144Users upload scanned contracts of up to 400 MB, and each takes about 20 minutes to process as one indivisible input. Users accept a notification when the result is ready. Which SageMaker AI path fits?
InputLocationBodyBody is capped at 128,000 bytes; real-time inference keeps a synchronous 60-second, 6 MB contract that a client timeout cannot extend; and Serverless allows 4 MB and one minute.An asynchronous endpoint’s requests set RequestTTLSeconds to 1800 and InvocationTimeoutSeconds to 900. During a backlog, many requests expire before processing starts. A team member raises InvocationTimeoutSeconds to 3600. What should change instead?
InvocationTimeoutSeconds again, past its 3,600-second valueBody to InputLocation so they queue fasterInvocationTimeoutSeconds so earlier requests finish soonerRequestTTLSeconds , which limits time in the queue, up to 21,600RequestTTLSeconds is how long a request may wait in the queue, and InvocationTimeoutSeconds is how long it may be processed. These requests expire while queued, so only the queue lifetime helps; InvocationTimeoutSeconds cannot exceed 3,600 anyway.A nightly batch transform job runs on eight instances over one 50 GB file of independent JSON Lines records. Monitoring shows one instance busy and seven idle. What change spreads the work?
MaxPayloadInMB above 100 so each request carries more recordsMaxConcurrentTransforms so each instance sends more requestsMaxPayloadInMB and its product with MaxConcurrentTransforms are capped at 100 MB.