AWSAWS Certified Generative AI Developer - Professional2026-104.1 Implement cost optimization and resource efficiency strategies
Choose inference delivery by deadline slack
Completed by the knowledge check
Learning objective
Choose batch inference versus on-demand inference based on delivery deadlines and capacity.
Core idea
A batch discount is useful only when the job can produce acceptable results before the business deadline. Batch inference accepts a set of independent requests for asynchronous processing. On-demand inference serves individual invocations through runtime APIs. Choose between them from the delivery contract and supported features before comparing prices.
For an architect scheduling Bedrock workloads, the relevant budget is deadline slack: the time available for submission, validation, queueing, processing, result inspection, and recovery. A nightly task can have substantial slack; an interactive question usually cannot. A task’s description as “offline” does not by itself prove that its deadline can tolerate a batch queue.
Capacity also differs between the paths. A batch job can wait for scheduling, while an on-demand route must manage its own runtime demand and throttles. Neither path creates a universal guarantee that work finishes on time. Use measured completion behavior and the applicable model, Region, and quota constraints.
How it works
Check whether the workload fits batch inference
Amazon Bedrock batch inference takes input from Amazon S3 and writes results to S3. Each record is processed independently. AWS states that batch inference does not support tool calling or structured output through response_format, and it is not supported for provisioned models. A request needing a model-client tool exchange belongs on another eligible execution path, even if the customer would tolerate a delayed answer.
Batch input uses JSONL records containing recordId and modelInput. The modelInput format follows the selected invocation type. CreateModelInvocationJob.modelInvocationType supports InvokeModel or Converse; the latter means a batch record uses the Converse request format, not that it becomes an interactive conversation with the client.
Assign stable recordId values from your own work identifiers. AWS does not guarantee that output record order matches input order. Joining results by line position can associate one customer’s generated result with another customer’s work. Keep the record-to-business mapping outside any assumption about processing order.
Check the supported model and Region combination and the applicable batch quotas before admitting work. Record-count and input-size requirements can determine whether a small workload is eligible or whether a large workload must be partitioned. Do not assume the limits of one model apply to every model, or that a job accepted under one Region’s configuration can be submitted unchanged elsewhere.
Separate job timeout from delivery promise
CreateModelInvocationJob identifies the model, S3 input and output configuration, and service role, and returns jobArn. Its timeoutDurationInHours has a documented range of 24 to 168 hours. That parameter bounds when the job is forced to time out; it is not a promised completion duration or a way to request a five-minute service deadline.
The submission also supports clientRequestToken for idempotency. Stable submission identity prevents accidental duplicate job creation for a retried submission, but it does not replace your application’s result-publication policy. Do not confuse one accepted API request with one successfully published result for every record.
Use GetModelInvocationJob.status to distinguish Submitted, Validating, Scheduled, and InProgress. Scheduled means the job has passed validation and is waiting in a queue. Expired means it did not begin before the timeout. PartiallyCompleted means not all records were processed in time. The delivery contract must account for these outcomes instead of assuming every accepted job ends as Completed.
The status response includes totalRecordCount, processedRecordCount, successRecordCount, and errorRecordCount. Use these fields with the output records to reconcile completion. A job that has started is not a completed business deliverable. Preserve successful records and identify unfinished or failed work before deciding what to rerun.
Compare deadline and capacity evidence
Estimate batch completion from observed submission-to-result times for comparable workloads, including queueing and validation. Reserve time for result checks and a bounded recovery path. Define an application cutoff before the customer deadline: if unfinished work reaches that cutoff, either move only the remaining eligible records to an approved on-demand route or report that the deadline cannot be met.
That recovery route needs its own capacity budget. Sending an entire overdue batch to runtime inference at the last moment can create a burst that overwhelms the alternate path. For bedrock-runtime, CloudWatch InvocationThrottles, InvocationLatency, InputTokenCount, and OutputTokenCount help evaluate on-demand pressure. EstimatedTPMQuotaUsage is an approximation, and AWS warns against using it as the sole capacity-planning indicator.
Interactive on-demand requests can use Converse or streaming APIs where supported. Their execution and retry boundaries are covered in 2.4.1 and 2.4.5. Here, the decision is whether observed runtime capacity and latency support the delivery deadline, including retries and output validation.
For supported models, the Flex service tier offers discounted on-demand runtime inference for workloads that tolerate longer processing times. It preserves the selected runtime API contract rather than turning requests into independent batch records; it can serve eligible workloads requiring features batch excludes. The tier guide names service_tier: flex; see 2.2.1 for API-specific tier mechanics. Priority, Standard, and Flex share the model’s on-demand quota, so moving delayed work to Flex does not isolate it from interactive or recovery demand.
AWS advertises batch inference for select foundation models at a 50% lower price than corresponding on-demand inference. Keep the qualifier: this is not proof that every model or workflow qualifies. Compare the complete delivery cost, including delayed work, recovery attempts, and result-processing overhead, rather than the advertised rate alone.
Key distinctions and trade-offs
Asynchronous versus deadline-safe: A task can tolerate asynchronous delivery yet still require a tighter deadline than observed batch scheduling supports.
Converse format versus multi-turn interaction: Batch supports a Converse-shaped record, but records remain independent and tool-calling exchanges are unsupported.
Timeout versus guarantee: The 24–168-hour job setting controls timeout behavior. Your business deadline is a separate application requirement.
Batch versus Flex: Batch handles eligible independent sets asynchronously. Flex keeps eligible work on the runtime path with longer tolerated processing times and shared on-demand quota.
Capacity versus discount: Lower batch pricing does not remove queueing. On-demand recovery does not create unused runtime quota automatically.
Completion versus publication: Job states and record counts describe processing. Your application must still validate and map results before releasing them.
Architecture / decision scenario
A retailer needs overnight descriptions for a large catalog and immediate answers for shoppers. It also receives a smaller set of urgent catalog corrections due within an hour. The team verifies that descriptions use a supported batch model and need neither tools nor response_format structured output.
The overnight work has enough measured slack for validation, queueing, processing, and checks. The scheduler submits JSONL with stable recordId values, records jobArn, and polls GetModelInvocationJob. It joins output by identifier and compares success and error counts rather than assuming line order or job submission proves delivery.
Shopper requests use on-demand inference because their interactive deadline cannot tolerate a batch queue. Urgent corrections also use on-demand processing when observed batch completion cannot meet the one-hour target. The team tests Flex for eligible corrections that tolerate slower runtime processing, retaining Standard when Flex does not meet the deadline. Both consume the shared on-demand quota. The gateway reserves capacity for those requests and watches InvocationThrottles and completion latency.
Before the overnight deadline, an application cutoff examines incomplete records. Only the remaining eligible work can move to the reserved recovery route; already successful records are retained. If that route lacks capacity, the scheduler reports the missed-delivery risk rather than promising that a 24-hour timeout ensures completion. The cost comparison includes both the batch discount and any recovery invocations.
Common exam traps
- Choosing batch solely because a task is described as offline.
- Treating
timeoutDurationInHoursas a guaranteed completion time. - Assuming batch Converse records support live tool exchanges.
- Joining results by output line order rather than
recordId. - Treating
ScheduledorInProgressas completed delivery. - Assuming an overdue batch can always spill into unlimited on-demand capacity.
What to remember
Choose an eligible path that meets the delivery contract under measured load. Budget queueing, validation, and recovery, not only generation. Batch jobs require job-state and record-level reconciliation; on-demand recovery requires reserved operating headroom. Compare price only after compatibility and deadline feasibility are established.
Knowledge check
A set of urgent catalog corrections must be published within one hour. The work needs no tools, and the model supports batch inference. Observed batch jobs for similar workloads spend 40 to 90 minutes between submission and results. Which path fits?
timeoutDurationInHours set to its minimum valueExpiredtimeoutDurationInHours bounds when a job is forced to time out, with a 24-hour minimum; it is not a completion promise. Waiting for Expired leaves no time to recover.A nightly enrichment workflow lets the model call client-side tools. It tolerates slower processing, and the team wants a discount over standard on-demand pricing. The model supports both batch inference and the Flex service tier. Which option fits?
modelInvocationType set to ConverseA batch job’s output file lists records in a different order from the input file. How should the scheduler map each result back to its customer’s work item?
recordId assigned from the work identifierrecordId taken from your own work identifier keeps the mapping independent of processing order.Near the application cutoff, a batch job reports PartiallyCompleted. An on-demand recovery route has a tested capacity budget. What should the scheduler do?
PartiallyCompleted means not every record was processed in time. Reconciling record counts and outputs preserves finished work, and moving only the remainder keeps the recovery burst inside the route’s capacity budget; a fresh batch job would face the same queue again.