AWSAWS Certified Generative AI Developer - Professional2026-102.2 Implement model deployment strategies

Size containers beyond the model weights

Completed by the knowledge check

Learning objective

Size model containers by GPU memory, model load behavior, context, and generated-token throughput.

Core idea

A model that loads successfully is not necessarily a model that can serve your workload. Its weights consume part of device memory; runtime buffers and active generation consume more. Context length and simultaneous sequences change that working set. A sizing decision must prove both readiness and sustained service under representative requests.

For a decoder-only language model, distinguish input processing from output generation. Long inputs stress prompt processing and attention state. Long outputs keep sequences active as tokens are generated. The same request rate can therefore imply very different memory use and response time.

Amazon SageMaker AI provides deployment parameters and CloudWatch metrics to observe these boundaries. Your inference engine supplies additional scheduling and memory controls. Keep their names and scopes separate: a container’s sequence limit is not an endpoint instance count, and a deployment health-check timeout is not an inference request timeout.

How it works

Budget device memory for active work

Begin with the exact artifact and serving configuration: model release, weight representation, inference-engine version, and GPU layout. Estimate weight memory, then reserve room for runtime allocation and active attention state. Validate the estimate by loading that artifact in the intended container; an artifact’s compressed S3 size is not a measurement of GPU memory use.

AWS Prescriptive Guidance describes the key-value (KV) cache as attention state retained for generation. Its size grows as a sequence adds tokens. More simultaneous sequences and longer prompt-plus-output sequences consume more cache capacity. A test with one short prompt cannot prove that a container supports the application’s longest concurrent requests.

Treat the accepted context as a serving policy, not just a model capability. For an engine using vLLM, AWS documents --max-model-len as the prompt-plus-output sequence limit and --max-num-seqs as the active sequence cap. These are engine controls, not fields of CreateEndpointConfig. Check the selected container/backend before applying them.

Another vLLM control, --gpu-memory-utilization, sets the memory allocation fraction. Raising it can leave more space for active KV state but less headroom for transient allocations. Do not copy a universal value from an example. Sweep the relevant settings under your measured context distribution and reject configurations that fail or violate latency targets.

Distinguish model parallelism from replication

If one model’s serving working set does not fit one GPU, adding independent model replicas does not make that one copy fit. Tensor parallelism splits each model layer across multiple GPUs, which work on the same request simultaneously; the vLLM control is --tensor-parallel-size. More GPUs can enable a larger model but add communication work. A replica adds another independently served copy with its own memory requirements.

Choose the layout using both memory and measured throughput. A multi-GPU model copy can pass the memory gate yet deliver poor economics for short requests. Several smaller eligible copies might serve more independent requests, but only if each copy fits and meets the task’s quality requirement.

Tensor parallelism splits one model copy across GPUs for the same request, while each replica is another copy serving independent requests.

SageMaker AI inference components declare each copy’s allocation through ComputeResourceRequirements. NumberOfAcceleratorDevicesRequired specifies accelerators, including GPUs or AWS Inferentia; MinMemoryRequiredInMb is required, and MaxMemoryRequiredInMb is optional. These memory fields allocate model memory, not a KV-cache-specific budget. Set replica count separately through RuntimeConfig.CopyCount: more copies do not enlarge one copy’s allocation or configure the engine’s tensor parallelism.

Keep adapter deployment choices in 1.2.6. This lesson’s sizing question is the total memory and generation behavior of the release actually being served, including its chosen runtime configuration.

Prove loading and readiness separately

SageMaker AI makes model artifacts available to the inference code under /opt/ml/model; the container has read-only access to that directory. Its serving process listens on port 8080 and implements /invocations and /ping. Model loading is part of readiness, not something a static health response should conceal.

Use /ping to check that the artifact is loaded, critical resources are available, and the inference path is functional. AWS documents a two-second timeout for individual health-check attempts. A server that returns HTTP 200 before it can generate results can appear healthy while failing requests.

For large models, configure the production variant’s ModelDataDownloadTimeoutInSeconds and ContainerStartupHealthCheckTimeoutInSeconds when the measured download or initialization time requires them. VolumeSizeInGB controls the instance’s attached EBS volume where applicable. These fields appear in CreateEndpointConfig under ProductionVariants.

AWS recommends increasing the EBS volume beyond the model’s size when the model exceeds 30 GB on an instance without local disk. It recommends increasing the model-download timeout for models larger than 40 GB. These are sizing recommendations with conditions, not universal maximum model sizes.

A longer initialization timeout cannot repair an out-of-memory load failure. A larger disk cannot enlarge GPU memory. Use startup logs to determine whether download, disk capacity, runtime initialization, or device allocation failed before changing a parameter.

Measure generated-token service capacity

AWS distinguishes prefill, which processes the prompt, from decode, which generates successive output tokens. Time to first token and time per output token describe different portions of the user experience. Measure both alongside total completion time; fast first-token delivery does not prove fast completion of a long answer.

Use representative prompt lengths, output limits, concurrency, and request mixes. Record completed output tokens per second and accepted results per second from the test harness. Do not count requested output tokens as generated work, or failed requests as successful capacity. Reuse the quality gates from 1.2.2 rather than inventing a new benchmark standard.

Observe CloudWatch GPUMemoryUtilization, GPUUtilization, ModelLatency, and OverheadLatency alongside those application measurements. GPU utilization measures device use, not answer quality or token throughput. On multi-GPU instances, the unnormalized GPU metrics sum utilization across devices: four GPUs can report a range up to 400 percent. Do not read a value over 100 percent as impossible or as evidence that every device is equally saturated.

Continuous batching can admit and remove sequences during generation. For vLLM, --max-num-batched-tokens controls the token budget per iteration. Larger admitted work can improve total throughput while delaying individual sequences. Tune it with the latency target and context mix rather than maximizing occupancy alone.

Key distinctions and trade-offs

Weights fit versus workload fits: Weight allocation is the first gate. Concurrent context, generation state, and runtime headroom determine whether the deployed model serves the workload.

Download timeout versus readiness timeout: The former budgets artifact transfer. The latter budgets container startup health. Neither extends the ordinary synchronous inference response contract; see 2.2.2.

Aggregate throughput versus per-request latency: More active sequences can raise total generated tokens per second while reducing the speed of each answer. Select the configuration that meets both the throughput requirement and the latency target.

Increasing context support is not free capacity. If most requests are short but a few are very long, measure the mixed workload explicitly. A configuration that passes only separate short-request and long-request tests might fail when those requests compete.

Architecture / decision scenario

A team loads its specialist model successfully on a GPU endpoint. Short single-user tests pass, but long-context requests under concurrency cause memory failures and slow generation.

The team first verifies that /ping reports model readiness and inspects load logs. It adjusts ModelDataDownloadTimeoutInSeconds only for measured artifact-transfer delay, and ContainerStartupHealthCheckTimeoutInSeconds only for legitimate initialization. Neither setting is used to hide runtime memory failures.

For serving, the team tests bounded prompt-plus-output lengths through --max-model-len, active sequences through --max-num-seqs, and memory headroom through --gpu-memory-utilization. It compares a fitting multi-GPU layout against eligible replicas, then sweeps batching settings. The chosen configuration meets tail latency and accepted-result throughput while retaining memory headroom on the mixed workload; a configuration with higher aggregate token output but stalled interactive answers is rejected.

Common exam traps

Selecting an instance from compressed artifact size alone.
Assuming a successful load proves concurrent long-context capacity.
Adding replicas when one model copy needs tensor parallelism to fit.
Increasing startup timeouts to fix GPU memory exhaustion.
Treating GPU utilization as generated-token throughput.
Maximizing batching throughput without checking per-request generation latency.

What to remember

Budget weights, runtime allocation, KV state, and headroom together.
Verify context and concurrency limits in the selected inference engine.
Separate download, startup readiness, and request processing budgets.
Measure actual generated tokens and accepted outputs under mixed load.
Use CloudWatch resource metrics to explain measured service behavior.