AWSAWS Certified Generative AI Developer - Professional2026-104.2 Optimize application performance
Bound demand before increasing inference throughput
Completed by the knowledge check
Learning objective
Improve throughput through bounded concurrency, quota planning, batching, and token capacity
Core idea
Throughput is completed, acceptable work per unit of time. Adding callers increases offered load; it does not necessarily increase completed work. If callers exceed model quotas, retries consume time and compete with useful requests. If callers serialize everything, available capacity remains unused.
Treat concurrency, request rate, token demand, and batch eligibility as separate controls. Concurrency limits requests in flight. Rate limits requests admitted over time. Token accounting captures differences between a short classification and a long document analysis. Batching changes the delivery contract for work that can finish asynchronously.
The useful operating point keeps the model busy while preserving deadlines and response quality. Choose it from representative load tests, then enforce it at the admission boundary. Hosting economics belong to 2.2.1; this lesson addresses how to use the available capacity without creating a retry storm.
How it works
Bound workers and requests separately
For an SQS-backed Lambda caller, configure the event source mapping’s ScalingConfig.MaximumConcurrency through UpdateEventSourceMapping. This caps concurrent function instances for that event source. Maximum concurrency is independent of function reserved concurrency; the reserved concurrency must cover the sum of configured maxima across the function’s SQS sources. See 2.2.4 for function-level concurrency budgets. Maximum concurrency and SQS provisioned mode cannot be used together.
A function-instance cap is not automatically a model-request cap. If each invocation processes several messages in parallel, multiply worker concurrency by parallel model calls per worker. Include tool loops and retry attempts in the load calculation. An application semaphore can bound in-flight calls inside each worker, while a shared admission policy bounds aggregate request and token demand across workers.
Use those controls together. A concurrency cap alone permits a higher request rate when requests become faster. A requests-per-minute limit alone permits long-running calls to accumulate. Choose limits below the tested operating boundary and reserve headroom for variable prompt sizes and recovery attempts. See 2.4.5 for bounded retry behavior and 2.4.2 for durable worker completion.
Plan against the applicable quota
On bedrock-runtime, AWS documents per-model, per-Region on-demand and cross-Region tokens-per-minute quotas. Requests-per-minute quotas apply to some models, rather than every model. There is also a cross-model daily quota scoped to the account and Region. Calls through different inference APIs share the selected model’s runtime quota; changing from InvokeModel to Converse does not create another allocation.
Inspect the actual account values in Service Quotas. AWS states that assigned quotas can be lower than published defaults and that increases are not automatic. Keep the model, Region, endpoint, and inference route attached to every capacity estimate. A quota for a cross-Region inference profile is not interchangeable with a single-Region on-demand quota. Routing boundaries belong to 1.2.4.
Token quota accounting also differs by endpoint. bedrock-mantle has separate per-model quota allocations and separate input and output token quotas. The combined runtime accounting described here must not be copied into a mantle admission controller.
Account for reservations and generated tokens
For bedrock-runtime, the initial quota deduction is input tokens plus the requested maximum output tokens. During processing, AWS adjusts usage; at completion, the documented calculation is InputTokenCount + CacheWriteInputTokenCount + (OutputTokenCount × burndown rate). Cache-read tokens do not contribute to that final quota calculation. The output burndown rate is model-specific.
An oversized output allowance can therefore reduce admitted concurrency even when responses are short. Set the allowance from observed completion lengths and task requirements, rather than always requesting the model maximum. For Converse, the output limit is inferenceConfig.maxTokens; the quota documentation calls the corresponding limit max_tokens. Preserve enough room for valid answers and monitor truncation before lowering it further.
Use CountTokens with the target modelId and compatible input.converse or input.invokeModel input where supported. Its inputTokens result describes the model-specific input count, not future output length or available quota. Combine input counts, output distributions, and the applicable burndown rate when estimating capacity. Token reduction without losing meaning belongs to 4.1.2.
Move eligible work into batch inference
Bedrock batch inference processes independent prompts asynchronously from S3 input and writes results to S3. CreateModelInvocationJob identifies the model through modelId and the locations through inputDataConfig and outputDataConfig; roleArn supplies the job’s access role. Inputs can use the InvokeModel or Converse format.
Batch inference has its own quotas and a model-and-Region support matrix. It does not support provisioned models, tool calling, or structured output through response_format. Independent batch records also do not provide multi-turn interaction. Check these constraints before treating an interactive agent workflow as offline work.
Separate two meanings of batching: a Lambda invocation receiving several SQS messages, and a Bedrock batch inference job. The former can still make many ordinary runtime calls and exhaust the same online quotas. Only the latter changes the service processing mode. See 4.1.4 for deadline-based selection.
Key distinctions and trade-offs
The controls answer different questions:
| Control | Decision it supports | Limitation |
|---|---|---|
| Worker concurrency cap | How much processing can run at once? | Does not count model calls hidden inside each worker |
| Request admission rate | How quickly can calls begin? | Does not represent variable token demand |
| Token admission budget | Can this prompt and output allowance fit? | Must reflect reservation and model-specific accounting |
| Batch inference job | Can independent work finish asynchronously? | Requires supported models, Regions, and features |
More headroom reduces utilization but improves tolerance of bursts and long prompts. Smaller output allowances admit more work but can invalidate responses. Judge both through accepted completions, rather than successful HTTP responses alone.
Architecture / decision scenario
A support platform serves live ticket triage and an overnight document classification backlog. The team measures each route separately and checks the selected model’s actual quotas. It gives the online route priority and leaves an explicit recovery budget.
For queued online work, MaximumConcurrency bounds Lambda workers and an internal semaphore bounds each worker’s model calls. Admission uses model-specific input estimates plus an output allowance. A load test increases concurrency until accepted throughput stops improving or latency and throttling cross the release gates.
The overnight classifier needs no tools or multi-turn interaction. After checking batch support and quotas, the team submits it through CreateModelInvocationJob, rather than letting backlog workers compete with interactive calls. It compares online InputTokenCount, OutputTokenCount, InvocationThrottles, and InvocationLatency in AWS/Bedrock with application completion records. EstimatedTPMQuotaUsage is useful telemetry, but AWS warns that it does not represent reservation-based throttling and must not be the sole capacity signal.
Common exam traps
- Increasing Lambda concurrency while leaving downstream token demand unbounded.
- Treating SQS message batches as Bedrock batch inference.
- Assuming all models enforce the same requests-per-minute quota.
- Using billed tokens as the complete quota-consumption calculation.
- Setting the maximum output allowance to the model limit for every task.
- Counting retries as throughput improvements instead of additional offered load.
What to remember
- Bound model requests as well as worker instances.
- Plan quotas by model, Region, endpoint, and inference route.
- Account for upfront reservations and model-specific output burndown.
- Use batch inference only for compatible independent asynchronous work.
- Increase concurrency only while accepted throughput and deadlines improve.
Knowledge check
An SQS event source mapping sets MaximumConcurrency to 10. Each invocation processes its messages by calling the model four at a time, and throttles appear well before the tested limit of 25 concurrent model calls. What should the team add?
MaximumConcurrency caps function instances, not model calls: ten workers making four calls each offer up to 40 in flight, before retries. A per-worker semaphore bounds hidden parallelism, and a shared admission policy bounds aggregate request and token demand. Raising reserved concurrency adds load, and longer backoff only spreads the excess.A team is close to a model’s tokens-per-minute quota on bedrock-runtime. It plans to send half the traffic through Converse and half through InvokeModel to double its capacity. What happens?
bedrock-runtime are shared across every inference API that calls it, even though the quota names mention InvokeModel . Capacity planning attaches to the model, Region, endpoint and route, and increases are requested rather than automatic.A triage feature’s answers rarely exceed 300 tokens, but each request sets inferenceConfig.maxTokens to the model maximum. The route throttles at moderate traffic. Which change admits more concurrent work without truncating valid answers?
maxTokens so the model uses its default limitmaxTokens from the observed completion distribution, with headroombedrock-runtime deducts input tokens plus the requested output maximum when a request starts, so an oversized allowance reserves quota that short answers never use. The default is the model maximum, billed tokens miss the reservation, and retries add offered load. Watch truncation after lowering the allowance.An overnight classification backlog is processed by Lambda workers that each receive SQS message batches and make ordinary runtime calls. These calls compete with live traffic for quota. The task needs no tools or multi-turn interaction. Which change moves it onto Bedrock’s batch processing mode?
CreateModelInvocationJob job from S3References
- AWS, Configuring scaling behavior for SQS event source mappings
- AWS, Quotas for the bedrock-runtime endpoint
- AWS, Quotas for Amazon Bedrock
- AWS, How tokens are counted in Amazon Bedrock
- AWS, CountTokens
- AWS, InferenceConfiguration
- AWS, Process multiple prompts with batch inference
- AWS, CreateModelInvocationJob
- AWS, Supported Regions and models for batch inference
- AWS, Monitor Amazon Bedrock runtime metrics