AWSAWS Certified Generative AI Developer - Professional2026-102.2 Implement model deployment strategies
Assign inference operations before choosing hosting
Completed by the knowledge check
Learning objective
Choose SageMaker AI managed hosting versus ECS, EKS, or EC2 inference operations
Core idea
For an architect choosing where to serve a model, the deciding question is who owns the serving system after deployment. A working container proves that inference can run. It does not prove that the team can replace unhealthy replicas, introduce a model safely, preserve available capacity, or explain a latency regression.
SageMaker AI provides a model-hosting interface with managed endpoint resources. Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and Amazon EC2 give you different ways to operate your own serving application. Compare them against the same model, traffic, and recovery requirements. A lower compute estimate is insufficient if it omits engineering effort or leaves reliability work unassigned.
Choose the option with the least operational surface that satisfies the required runtime and operating contract. Existing container or Kubernetes expertise can change that choice, but it does not remove the need to design inference-specific readiness and capacity controls. Inference mode selection belongs to 2.2.2; this lesson concerns ownership of the hosting layer.
How it works
SageMaker AI: declare the model and endpoint resources
The conventional SageMaker AI deployment path uses CreateModel, CreateEndpointConfig, and CreateEndpoint. The model identifies the inference image and artifacts. The endpoint configuration names the model in ProductionVariants and specifies resources with fields such as InstanceType and InitialInstanceCount. Creating the endpoint provisions those resources and deploys the model.
This separates an application’s invocation contract from instance administration. The application calls InvokeEndpoint with EndpointName; it does not discover individual serving instances. SageMaker AI also supports inference components, which allocate endpoint resources to models. Choose that topology deliberately rather than treating every endpoint as one container; see 1.2.6 for adapter-serving decisions.
SageMaker AI endpoint auto scaling adjusts provisioned instance counts as workload changes. Deployment guardrails provide managed blue/green updates with all-at-once, canary, or linear traffic shifting, and rolling deployments with built-in safeguards such as automatic rollback. Guardrails apply to eligible real-time and asynchronous endpoints; check their exclusions and see 1.2.8 for release decisions. These are platform features you configure instead of building deployment orchestration yourself.
Managed hosting still leaves meaningful work with you. You select artifacts, container behavior, instance capacity, and scaling policy. You define the acceptance test and decide whether an endpoint’s response meets the application contract. A platform cannot determine whether the deployed model answers the right business question.
Use SageMaker CloudWatch metrics to separate container processing from platform overhead. ModelLatency includes container inference and communication with the container; OverheadLatency measures the remaining SageMaker overhead. Both use microseconds. Measure client latency as well: neither platform metric represents every hop between a user and the application. A healthy endpoint and acceptable tail latency are separate release conditions.
ECS: schedule a serving container
For GPU inference on ECS, AWS documents GPU-capable EC2 container instances and a GPU-optimized Amazon Machine Image (AMI) with NVIDIA drivers and a Docker GPU runtime. GPU allocation is a task-definition concern. A ResourceRequirement identifies type: GPU; its value specifies the physical GPU count reserved for the container. The total reservation across a task cannot exceed the GPUs available on its container instance.
ECS therefore provides a concrete scheduling mechanism, not a model-specific performance guarantee. A reserved GPU does not mean that the model fits with its request-dependent memory or that throughput meets the target. Measure that serving envelope as described in 2.2.5.
Choose ECS when a container-oriented team wants to package inference as an application service without introducing a Kubernetes operating model. Assign ownership for traffic admission, health behavior, replacement readiness, artifact loading, and service releases. AWS’s GPU-optimized AMI reduces setup work; it does not specify your model’s readiness condition.
EKS: use Kubernetes with explicit accelerator support
EKS offers Kubernetes scheduling and extensibility. NVIDIA GPUs require a supported device-management path. AWS documents the NVIDIA Kubernetes device plugin and the NVIDIA Dynamic Resource Allocation (DRA) driver. The plugin advertises integer nvidia.com/gpu resources; DRA describes devices through ResourceSlice objects.
Their support boundaries differ. AWS recommends DRA for Kubernetes 1.34 and later with static capacity provisioning in Karpenter, managed node groups, or self-managed nodes. DRA is not supported with EKS Auto Mode. AWS directs EKS Auto Mode and dynamic Karpenter provisioning to the NVIDIA device plugin. Do not select an accelerator mechanism independently of the cluster version and provisioning mode.
Choose EKS when existing Kubernetes operations or a required Kubernetes integration justify the additional configuration. Name who maintains the serving workload, GPU allocation, node capacity, readiness, and rollout behavior. Cluster management does not establish model compatibility or safe request concurrency.
EC2: own the serving host
Direct EC2 hosting gives you the host as the deployment boundary. An NVIDIA GPU instance needs the appropriate NVIDIA driver; AWS documents installing a driver or choosing an AMI with it preinstalled. Driver choice depends on the instance and workload.
Use EC2 when a required host-level configuration or specialized serving arrangement warrants direct control. Then explicitly assign operating-system maintenance, runtime compatibility, process supervision, traffic distribution, host replacement, and model release duties. Starting an inference process on one GPU instance is an experiment, not a complete availability design.
Key distinctions and trade-offs
The following comparison identifies the decision each option makes you own:
| Option | Primary deployment contract | Main reason to choose it |
|---|---|---|
| SageMaker AI | Model, endpoint configuration, endpoint or inference component | Platform scaling and guarded rollout reduce custom operations |
| ECS | Container task and GPU reservation on supported container instances | Container-service operations fit the team and workload |
| EKS | Kubernetes workload and supported accelerator allocation | Kubernetes integrations and operating practices are required |
| EC2 | Host, driver, runtime, and serving process | Direct host control is a concrete requirement |
These are operating boundaries, not a ranking of model quality. Moving an unchanged model to Kubernetes does not improve its answers. Moving to managed hosting does not remove evaluation, artifact approval, or capacity planning.
Compare total operating cost at equivalent accepted throughput and recovery behavior. Include idle capacity, rollout headroom, observability, and engineering ownership. A cheaper host that cannot sustain demand or recover within the required time is not the cheaper implementation of the same requirement.
Architecture / decision scenario
A document-processing team needs GPU inference, workload-responsive capacity, and guarded model updates. It has a container image but no team capacity to build and maintain inference rollout orchestration. A load replay and replacement test compare all four hosting options.
If the container and endpoint are eligible, SageMaker AI meets that ownership constraint with endpoint auto scaling and deployment guardrails. The team configures capacity and release safeguards, then compares ModelLatency, OverheadLatency, and client timing. Existing use of a SageMaker API is not the deciding requirement.
The choice changes if an established EKS platform team requires a Kubernetes-only serving integration and already owns scaling and rollout operations. EKS then fits the integration and ownership contract; dynamic Karpenter provisioning uses the NVIDIA device plugin. If the requirement instead is integration with an existing ECS container service, ECS is the candidate, with GPU ResourceRequirement reservations and an assigned readiness owner. A demonstrated host-level requirement selects EC2 only with a funded host-maintenance and recovery plan.
Without a required integration or host customization, the team chooses the eligible managed endpoint. With one, it chooses the matching platform only when its owners can demonstrate the required capacity and guarded replacement behavior.
Common exam traps
- Treating a managed endpoint as responsibility-free inference.
- Choosing ECS GPU inference without GPU-capable container capacity and reservations.
- Assuming every EKS GPU mechanism works with every provisioning mode.
- Equating direct EC2 control with a complete production serving design.
- Comparing host prices while omitting equal reliability and accepted throughput.
- Using platform latency metrics as the entire user-visible latency budget.
What to remember
Choose hosting from runtime requirements and operational ownership. SageMaker AI declares model-hosting resources; ECS schedules container resources; EKS adds a Kubernetes and device-management contract; EC2 makes the host your responsibility. Require evidence that the selected option can load, serve, replace, and observe the model under representative demand.
Knowledge check
A document-processing team has a GPU inference container that fits SageMaker AI’s endpoint contract. It needs workload-responsive scaling and canary model releases with automatic rollback, and it has no capacity to build or run release orchestration. It has no Kubernetes or host-level requirement. Which hosting option fits?
A container-service team moves a GPU model onto Amazon ECS. Which configuration reserves accelerators for the model container?
nvidia.com/gpu resource request in the container definitionResourceRequirement of type: GPU on GPU-capable container instancesmemory value in the task definition for the model weightsResourceRequirement whose value is the physical GPU count, placed on GPU-capable container instances such as those using the GPU-optimized AMI. nvidia.com/gpu is the Kubernetes device-plugin resource, and task memory is host memory, not a GPU reservation.A platform team runs EKS on Kubernetes 1.35 and provisions GPU nodes with Karpenter’s dynamic capacity provisioning. It wants to schedule a GPU inference workload. Which device-management path does AWS direct it to?
A SageMaker AI endpoint shows a steady p95 ModelLatency and low OverheadLatency, but users report slow answers. What should the team measure next?
GPUUtilization , which shows how quickly answers are producedModelLatency again, converting it from milliseconds to secondsModelLatency covers container inference and its local communication, and OverheadLatency covers the rest of SageMaker’s handling; both are in microseconds. Neither includes the hops between the user and the endpoint, so client timing is the missing measurement, and GPU utilization is not a latency figure.