AWSAWS Certified Generative AI Developer - Professional2026-102.2 Implement model deployment strategies
Check compilation fit before tuning loading
Completed by the knowledge check
Learning objective
Consider SageMaker Neo only for supported models and target hardware; otherwise optimize model loading separately
Core idea
For an architect investigating inference performance, compilation and loading answer different questions. Compilation transforms a supported model for a selected execution target. Loading moves artifacts into a serving environment and initializes the runtime before requests can succeed. A model can execute efficiently and still take too long to become ready.
SageMaker Neo consists of a compiler and a target runtime. Its compiler translates exported framework operations into an intermediate representation, optimizes them, and produces target code and model artifacts. Its runtime loads and executes the compiled model. This is a compatibility-dependent path, not a universal accelerator for foundation models.
First establish whether the model, export format, framework version, inputs, and target form a supported combination. If that case cannot be established, exclude Neo and investigate loading or a separate eligible LLM optimization path. Even when Neo is eligible, measure both steady-state execution and deployment readiness. The optimization must improve the part of the system that actually violates the requirement.
How it works
Establish the complete support tuple
A framework name is only the beginning of the eligibility check. AWS’s Neo cloud support table distinguishes framework versions, model categories, artifact formats, and deployment instance families. For example, its PyTorch row includes versions 1.13 and 2.0 with support for object detection, Vision Transformer, and HuggingFace models. That wording does not establish that every Hugging Face architecture, operator, or export is supported.
The cloud table lists CPU and GPU targets such as ml_c5, ml_m5, ml_p3, and ml_g4dn. It treats Inferentia targets separately. Do not infer that a GPU instance available for ordinary SageMaker hosting is a Neo compilation target. Likewise, a valid API target name is not proof that the selected framework and model support it.
Record the architecture, export, framework version, and intended hardware. Check target-specific support before compiling; changing a request field does not repair an unsupported combination.
Avoid broad claims that Neo excludes all language models or supports all transformers. Neither follows from the support table. The architectural decision is conditional: use a documented, tested combination and verify the result on your own workload.
Make the compilation contract explicit
CreateCompilationJob uses InputConfig and OutputConfig to describe the source model and intended output. InputConfig.S3Uri points to a single gzip-compressed tar archive. InputConfig.Framework identifies the framework. FrameworkVersion is supported for MXNet, PyTorch, TensorFlow, and TensorFlow Lite; it is not a universal field for every framework.
DataInputConfig describes expected input names and shapes in a framework-specific JSON dictionary. This is part of the compilation contract, not a place for a sample natural-language prompt. Input shape and data type must match the exported model and the serving request transformation. A compilation experiment with the wrong input contract does not establish production compatibility.
OutputConfig.S3OutputLocation selects where to write the compiled artifact. Specify either TargetDevice or TargetPlatform: the fields are mutually exclusive. TargetDevice names a listed device or instance target; TargetPlatform describes an operating system, architecture, and accelerator. CompilerOptions carries target-specific compiler settings. AWS documents it as required for NVIDIA accelerators and highly recommended for CPU compilations.
These settings explain why compilation is tied to a deployment target. You cannot treat the output as an arbitrary framework archive and assume it runs unchanged wherever the original model ran. Preserve the original artifact and the compilation configuration alongside the produced artifact so that the release remains reproducible.
After compilation, compare correctness, execution latency, throughput, and readiness against the baseline on representative inputs. Successful compilation alone does not establish deployment acceptance.
Optimize loading on its own evidence
For SageMaker large-model hosting, distinguish artifact transfer, decompression, runtime initialization, and readiness. Observe the container’s CloudWatch logs to identify the failing phase. The documented endpoint controls live in ProductionVariants of CreateEndpointConfig.
ModelDataDownloadTimeoutInSeconds gives the artifact download more time. AWS recommends increasing it when model size exceeds 40 GB. ContainerStartupHealthCheckTimeoutInSeconds gives a correctly configured container more time to respond to health checks. VolumeSizeInGB provides artifact storage; AWS recommends increasing it beyond the model size when a model exceeds 30 GB and the instance has no local disk.
Those values have different jobs. Extending a timeout lets a valid but slow startup finish; it does not make downloading or initialization faster. Increasing storage does not increase GPU memory. Keep this startup diagnosis separate from the request-dependent memory and token-capacity decisions in 2.2.5.
SageMaker also supports uncompressed model deployment through CreateModel.PrimaryContainer.ModelDataSource.S3DataSource. For an artifact prefix, use S3DataType: S3Prefix, CompressionType: None, and an S3Uri pointing to the model’s files. For one uncompressed object, use S3DataType: S3Object. This avoids the archive-decompression step for a large artifact.
Scope that option carefully. AWS documents that ModelDataSource cannot be used with AWS Marketplace, batch transform, Serverless Inference endpoints, or multi-model endpoints. Check the hosting mode before proposing it. Reducing decompression work does not eliminate download time, model initialization, or the need for sufficient local storage.
For eligible large language models (LLMs), the SageMaker AI inference optimization toolkit provides fast model loading. It pre-shards the model and stores weights in equal-sized chunks, then streams them from Amazon S3 directly onto GPUs. This bypasses download-to-disk, host-memory loading, and host-side sharding, improving deployment and scale-out readiness. CreateOptimizationJob applies techniques through OptimizationConfigs; ModelShardingConfig is its sharding configuration. Deploy and measure the optimized output rather than treating this as a timeout setting. Check support for the specific model and target; the overview does not provide a universal model list.
Fast model loading streams pre-sharded weights onto GPUs without download-to-disk, host-memory loading or host-side sharding.
The toolkit also offers separate ahead-of-time LLM compilation: TensorRT-LLM for GPUs and the AWS Neuron SDK for Trainium or Inferentia. It reduces deployment and scale-out time by avoiding just-in-time compilation. This uses ModelCompilationConfig in optimization-job OptimizationConfigs, not Neo’s CreateCompilationJob. Unsupported Neo compatibility does not rule out this separate supported path.
Key distinctions and trade-offs
The following choices act on different parts of deployment:
| Decision | Mechanism | What it does not establish |
|---|---|---|
| Compile for a supported target | Neo InputConfig and OutputConfig | Universal model or hardware compatibility |
| Prepare LLM weights for faster loading | Optimization-job ModelShardingConfig | Support for every model or target |
| Compile an eligible LLM ahead of time | Optimization-job ModelCompilationConfig | Neo compatibility |
| Avoid archive decompression | ModelDataSource with CompressionType: None | Faster network transfer or runtime initialization |
| Permit a valid slow startup | Download and startup health-check timeouts | Lower startup latency |
| Provide artifact space | VolumeSizeInGB | Enough accelerator memory for inference |
Compilation adds an artifact and target-compatibility boundary. Uncompressed deployment changes packaging. Timeout changes alter how long the platform tolerates startup. Choose the mechanism that addresses the observed phase, and retain the simpler path when the measured benefit is insufficient.
Architecture / decision scenario
A team serves two models: a supported image classifier and a large generative model with slow deployment readiness. It considers Neo for both rather than assuming one optimization applies everywhere.
For the classifier, the team checks its framework version, export format, input shapes, and TargetDevice against the Neo support table. It compiles only after those checks pass, then compares prediction agreement and steady-state performance on the intended hardware.
For the generative model, the team cannot establish Neo support. It separately checks optimization-toolkit eligibility: fast model loading can address host-side staging and sharding; ModelCompilationConfig can avoid just-in-time compilation on its supported target. It tests the relevant optimized output rather than assuming either technique supports the model.
If no toolkit path fits and logs show archive extraction is slow, the eligible real-time endpoint can still use uncompressed ModelDataSource packaging. The team provides adequate storage and startup timeouts, then measures replacement readiness. A longer timeout is not a speed improvement, and the classifier’s compilation result proves nothing about the generative model.
Common exam traps
- Selecting Neo because the source model uses a named supported framework.
- Treating a target enum as proof of model-and-framework compatibility.
- Supplying both
TargetDeviceandTargetPlatform. - Confusing Neo compilation with optimization-job LLM compilation.
- Treating timeout extensions as performance optimizations.
- Confusing artifact storage with GPU inference memory.
- Recommending uncompressed
ModelDataSourcefor an excluded hosting mode.
What to remember
Neo requires a supported model, framework, input contract, and target. Compilation and loading remain separate measurements. For unsupported combinations, optimize the observed transfer, packaging, storage, or initialization problem with the hosting mechanism that actually applies. Validate readiness and output quality before accepting any optimization.
Knowledge check
A team wants to compile a Hugging Face image model exported from PyTorch with SageMaker Neo. The Neo cloud support table lists PyTorch 2.0 with HuggingFace models. What must the team establish before compiling?
TargetDevice and TargetPlatform are both set, to cover more hardwareTargetDevice and TargetPlatform are mutually exclusive.A large model on a real-time endpoint becomes ready slowly. Logs show that the download finishes quickly, then several minutes are spent extracting the model.tar.gz archive. Steady-state inference meets its target. Which change shortens that startup phase?
ModelDataDownloadTimeoutInSeconds on the production variantVolumeSizeInGB so the archive has more room to extractModelDataSource with CompressionType: NoneModelDataSource is available on real-time endpoints, though not on Serverless, batch transform, multi-model endpoints, or Marketplace.A large language model has no established Neo support. When the endpoint scales out, logs show each new instance spending most of its startup downloading weights to disk, loading them into host memory, and sharding them on the host. Which path should the team evaluate?
CreateCompilationJob with TargetPlatform instead of TargetDeviceCreateOptimizationJob with ModelShardingConfigModelCompilationConfig in an optimization jobContainerStartupHealthCheckTimeoutInSeconds for new instancesA large model’s endpoint fails to become healthy. Logs show the download completed, and runtime initialization is progressing normally but takes longer than the configured startup health-check window. Which change is appropriate?
ContainerStartupHealthCheckTimeoutInSeconds to fit the measured startupModelDataDownloadTimeoutInSeconds to fit the measured startupVolumeSizeInGB above the model size/ping before initialization completes