AWSAWS Certified Generative AI Developer - Professional2026-104.2 Optimize application performance
Scale hosted capacity from measured demand
Completed by the knowledge check
Learning objective
Plan and auto-scale FM capacity using queue depth, token throughput, prompt length, and utilization
Core idea
A foundation model request is not a uniform unit of compute. A short prompt with a short completion and a long prompt with sustained generation can occupy the serving system for very different durations. Requests per minute alone can hide the demand that determines whether your application meets its latency target.
Plan capacity using representative input lengths, completion lengths, concurrent work, and accepted token throughput. Then choose a scaling signal that changes meaningfully when you add serving capacity. Queue depth reveals unfinished demand; utilization reveals resource pressure; neither replaces a measured service envelope.
The actuator depends on the hosting path. SageMaker AI supports Application Auto Scaling for hosted models. Bedrock on-demand callers manage admission and applicable quotas rather than endpoint instance counts. Do not interpret the objective as a universal automatic capacity control across every FM service. See 4.2.5 for caller-side throughput controls.
How it works
Measure work by length and service rate
Benchmark the deployed model and container with the workload’s prompt-length and output-length distributions. Keep model identity, hardware, serving configuration, and acceptance criteria fixed. Measure sustainable input and output tokens per second, response latency, and the concurrency at which the quality and latency gates still pass.
For hosted models, collect token counts from the serving container’s response contract and publish application measurements through CloudWatch PutMetricData. Use an application namespace and MetricData entries whose MetricName and Dimensions identify the model deployment and workload class. Token throughput and prompt-length measurements are application-defined here; they are not universal built-in SageMaker token metrics.
For a Bedrock runtime route, AWS/Bedrock supplies InputTokenCount, OutputTokenCount, InvocationLatency, and InvocationThrottles. AWS also publishes EstimatedTPMQuotaUsage, but warns that this estimate does not reflect reservation-based throttling and must not be the sole capacity-planning signal. See 4.2.5 for the reservation calculation.
Use the benchmark to forecast required capacity. For example, if a tested instance sustains an acceptable mix at 40 requests per minute, an illustrative 160-request-per-minute workload requires four instances before headroom. That estimate is valid only for the tested length distribution. Doubling prompt length invalidates the assumption that every request costs the same amount of work.
Register the actual scaling resource
For a SageMaker production variant, use Application Auto Scaling RegisterScalableTarget with ServiceNamespace: sagemaker, ResourceId: endpoint/ENDPOINT_NAME/variant/VARIANT_NAME, and ScalableDimension: sagemaker:variant:DesiredInstanceCount. MinCapacity sets the serving floor and MaxCapacity bounds expansion. Choose the bounds from required availability, tested capacity, instance quotas, and the workload’s budget.
Apply a policy through PutScalingPolicy with PolicyType: TargetTrackingScaling and a TargetTrackingScalingPolicyConfiguration. For similar request sizes, PredefinedMetricType: SageMakerVariantInvocationsPerInstance tracks per-instance invocation demand. Its TargetValue comes from your load test; an example value in documentation is not a recommended capacity target for your model.
For long-running generation, concurrent work often represents pressure better than completed request counts. SageMakerVariantConcurrentRequestsPerModelHighResolution uses ConcurrentRequestsPerModel. AWS documents that it includes requests queued inside the container and tracks streaming requests until the last token. The metric emits every 10 seconds, compared with once per minute for standard InvocationsPerInstance metrics.
Inference components have a different resource and policy surface. For their corresponding concurrency policy, the predefined metric type is SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution. Do not substitute a per-copy metric into a production-variant policy without matching the resource being scaled.
Use backlog for asynchronous work
For SageMaker Asynchronous Inference, AWS recommends target tracking with ApproximateBacklogSizePerInstance through CustomizedMetricSpecification. The example uses namespace AWS/SageMaker, dimension EndpointName, and statistic Average. This normalizes waiting demand by capacity rather than treating a queue of 100 requests identically at one instance and at ten instances.
Asynchronous endpoints can scale to zero with MinCapacity: 0; requests received at zero capacity remain queued. AWS documents an additional step-scaling policy triggered by a CloudWatch alarm on HasBacklogWithoutCapacity. Without that policy, scaling up from zero waits until backlog exceeds the target-tracking value. A sparse queue can therefore wait unnecessarily even though work has arrived.
Queue depth still counts requests rather than tokens. If a backlog shifts toward long documents, the same depth can require far more processing time. Track prompt-length buckets and output demand beside the backlog metric. Use this evidence to revise the tested target or separate workload classes, rather than pretending the queue metric already measures token work.
Interpret utilization and stabilize the controller
SageMaker endpoint metrics include GPUUtilization, GPUMemoryUtilization, CPUUtilization, and ModelLatency. GPU and CPU utilization values can be sums across devices or cores rather than a single 0–100 percentage. Normalized metrics such as GPUUtilizationNormalized are scoped to endpoints with active inference components. Check the metric’s units and scope before using a percentage threshold.
High GPU memory utilization is not proof of productive throughput. Pair resource pressure with queueing, concurrent work, token completions, and latency. A custom target-tracking metric must change in proportion to capacity; raw aggregate token throughput can rise with demand rather than fall as instances are added. Use it to establish the service envelope, not automatically as a utilization target.
Set ScaleOutCooldown and ScaleInCooldown from observed readiness and traffic behavior. A short cooldown cannot make model loading instantaneous. Keep a warm floor for response deadlines that cannot tolerate provisioning, and test bursts, length-distribution changes, scale-out, and scale-in before relying on the controller.
Key distinctions and trade-offs
Demand versus productivity: Queue depth and concurrent requests indicate work awaiting or receiving service. Completed tokens measure output rate. A rising backlog with flat token throughput indicates unmet demand; a higher token rate alone does not prove acceptable latency.
Capacity floor versus maximum: The floor protects readiness during quiet periods. The maximum limits spending and resource expansion. When the maximum is reached, callers still need admission and deadline policies.
Hosted autoscaling versus Bedrock provisioning: Application Auto Scaling changes hosted SageMaker capacity. Bedrock Provisioned Throughput follows its own provisioning modes: AWS documents Model Unit capacity as not editable through the existing resource, while token-based provisioning permits modifying input and output tokens per minute. This is not a SageMaker instance-count policy; see 4.1.5 for purchased-capacity decisions.
Architecture / decision scenario
A platform hosts an interactive FM and an asynchronous document processor. The team benchmarks short, medium, and long prompts and records accepted token throughput, latency, and utilization. A long-prompt surge requires more capacity despite unchanged requests per minute.
For the interactive production variant, it registers DesiredInstanceCount with a warm minimum and a tested maximum. A high-resolution concurrent-request policy captures sustained streaming demand. ModelLatency, token-completion measurements, and length buckets confirm whether expansion restores the service target rather than merely adding idle instances.
For the asynchronous endpoint, it uses ApproximateBacklogSizePerInstance target tracking and a HasBacklogWithoutCapacity step policy for wake-up from zero. Long-document workload shifts trigger a review of the backlog target against measured drain rate. The team tests scale-in cooldowns so a brief drop in queue depth does not remove capacity needed for the next burst.
Common exam traps
- Sizing FM capacity from request counts while ignoring prompt and completion lengths.
- Treating every endpoint metric as a percentage between 0 and 100.
- Using aggregate token throughput as an untested utilization target.
- Applying per-copy policy types to the wrong scaling resource.
- Assuming a zero-capacity asynchronous endpoint immediately serves a single queued request.
- Treating Bedrock quotas or provisioning as SageMaker instance autoscaling.
What to remember
- Benchmark the token and length distribution that production actually serves.
- Scale the registered resource using a tested per-capacity pressure signal.
- Use concurrent requests for sustained generation and backlog for asynchronous work.
- Pair utilization with accepted throughput and latency.
- Test readiness, cooldowns, zero-capacity wake-up, and maximum-capacity behavior.
Knowledge check
A load test showed one instance of a hosted model sustaining 40 requests per minute within the latency target, so the team runs four instances for 160 requests per minute. Request volume stays flat, but prompts and completions roughly double in length, and latency misses its target. What should the team revise?
A SageMaker production variant serves long-lived streaming generations, and many requests wait inside the container. Its target-tracking policy uses SageMakerVariantInvocationsPerInstance and reacts slowly. Which predefined metric better reflects the pressure?
SageMakerVariantConcurrentRequestsPerModelHighResolutionSageMakerInferenceComponentConcurrentRequestsPerCopyHighResolutionSageMakerVariantInvocationsPerInstance with a lower target valueApproximateBacklogSizePerInstance as a customized metricA SageMaker asynchronous endpoint scales to zero overnight using target tracking on ApproximateBacklogSizePerInstance with a target of 5. A single request arrives at 2 a.m. and waits for hours. What fixes it?
ScaleOutCooldown so the target-tracking policy reacts soonerHasBacklogWithoutCapacity alarmHasBacklogWithoutCapacity that starts capacity whenever work is queued and no instance exists; retries only add duplicate work to the queue.An engineer proposes a target-tracking policy whose metric is the endpoint’s total generated tokens per minute, with the target set at the benchmark’s sustainable rate. Why is this metric unsuitable as the target?