NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.4 Scale deployments using containerization (Docker, Kubernetes) with load balancing.
Admit GPU Pods only when inference works
Completed by the knowledge check
Learning objective
Deploy GPU containers on Kubernetes with the GPU Operator, health probes, and resource requests.
Core idea
A container image packages an inference service; it does not provision the node’s GPU software, reserve suitable hardware, or establish that the model can answer. Treat those as three separate decisions: prepare the node, schedule the resources, and admit traffic after model initialization.
The NVIDIA GPU Operator automates the node software stack. Kubernetes resource requirements control placement, while probes control startup, restart, and readiness. A Pod can pass one decision and fail the next. Increasing its replica count helps only after you identify which decision prevents usable capacity. See 0.8 for the image, workload, and Service objects; this lesson focuses on the GPU-specific admission path.
How it works
Provision the GPU software on the node
The GPU Operator manages NVIDIA drivers, the Kubernetes GPU device plugin, NVIDIA Container Toolkit, GPU feature discovery, and DCGM-based monitoring. The driver enables GPU access; the device plugin exposes schedulable GPU resources; the toolkit connects containers to the GPU runtime. Installing an application image alone does not perform those node-level steps.
Choose installation values from the node’s actual ownership model. When drivers are preinstalled, set the Operator Helm value driver.enabled=false. When both drivers and NVIDIA Container Toolkit are preinstalled, also set toolkit.enabled=false and follow the installation page’s runtime prerequisite. These settings assign responsibility to the platform’s node-image process instead of asking two mechanisms to manage the same installation.
The installation guide requires a container engine such as containerd or CRI-O. When driver configuration uses ClusterPolicy, the GPU driver-container path requires GPU worker nodes to use the same operating-system version; preinstalled drivers permit differing operating systems. Node Feature Discovery is a dependency and is deployed by default; set nfd.enabled=false if the cluster already runs it. Where Pod Security Admission applies, the guide requires the Operator namespace’s enforcement policy to be privileged. Keep that infrastructure requirement distinct from the permissions of the inference workload.
Reserve resources that match the replica
See 7.2.3 for NIM deployment with Helm or the NIM Operator; the NIM Helm chart exposes GPU limits through resources.limits and model-cache storage through persistence. After device-plugin installation, Kubernetes exposes nvidia.com/gpu. Set the container’s resources.limits entry for that resource to the GPU count required by the selected model deployment. Kubernetes uses the GPU limit as the request when only the limit is supplied. If you supply both GPU requests and limits, their values must be equal; a GPU request without a limit is invalid for this device-plugin model.
A GPU count is a scheduling quantity, not a GPU-memory specification. For a heterogeneous cluster, use GPU Feature Discovery’s nvidia.com/gpu.product (model) and nvidia.com/gpu.memory (memory in MiB) as nodeSelector or node-affinity keys. Sharing can modify the product label; under the MIG single strategy, the product includes partition information and memory describes each MIG device. A GPU count alone cannot distinguish memory capacities. See 3.4.2 for model and precision selection and 7.2.2 for profile-to-GPU fit.
Also set CPU and host-memory requests. Kubernetes schedules using requests, while CPU limits can throttle execution and host-memory limits can cause out-of-memory termination. Tokenization, request handling, and model loading still need host resources even when inference uses a GPU. A healthy GPU does not compensate for an undersized CPU or host-memory allocation.
If the Pod remains Pending, inspect kubectl describe pod and the resource and placement requirements before changing health probes. A probe cannot repair an unschedulable GPU reservation. Conversely, a scheduled Pod that cannot initialize the model needs initialization diagnostics rather than another scheduler constraint.
Separate startup, readiness, and liveness
Kubernetes provides different reactions for different failures:
| Probe | Decision | Mechanism and consequence | Main tuning cost |
|---|---|---|---|
startupProbe | Has initialization completed? | Suppresses liveness and readiness checks until it succeeds; repeated failure restarts the container | A short allowance can restart a valid model load |
readinessProbe | Should this replica receive Service traffic? | Failed readiness removes the Pod from ready service endpoints | Too permissive admits an unloaded model; too strict removes usable capacity |
livenessProbe | Does the container need restarting? | Repeated failure causes restart according to the workload’s restart policy | An overload-sensitive check can create restart churn |
NIM’s GET /v1/health/live returns 200 when its container is running and does not require the model to be loaded. GET /v1/health/ready returns 200 when the model is loaded and inference is available. Use that distinction when connecting NIM to Kubernetes probes. A successful /v1/health/live check alone is insufficient for admission to inference traffic.
Choose the startup endpoint for the initialization milestone you want to protect. A startup check against model readiness can reserve time for loading before the ordinary liveness policy begins. Kubernetes documents the startup allowance through failureThreshold × periodSeconds: a threshold of 30 and period of 10 seconds gives an approximately 300-second allowance. Set the actual allowance from measured initialization, including a start without a populated model cache (the chart’s persistence volume), rather than copying that example.
When startup repeatedly fails because a legitimate load exceeds the allowance, increase the allowance after measuring it. When readiness fails because model loading is broken, inspect the container’s initialization logs and correct the loading failure. Merely lengthening the probe cannot make a failed initialization succeed.
Key distinctions and trade-offs
The Operator provisions node capabilities; the scheduler assigns those capabilities to Pods; readiness decides which assigned Pods serve traffic. Each has a different owner and repair path. Keep the cluster platform team responsible for drivers and runtime integration, and the serving team responsible for the model’s resource requirements and usable-health definition.
A Pod can pass one decision and fail the next, and each failure has its own repair path.
A longer startup allowance reduces premature restarts but delays detection of a stuck initialization. A larger resource request can improve predictability but reduce the number of replicas that fit. These settings should reflect the same measured replica, so the scheduler and health checks describe a deployment that can actually become ready.
Architecture / decision scenario
A hospital’s document-retrieval assistant runs on a fixed four-GPU cluster. In this scenario, the unnamed serving configuration has been validated on one of these GPUs per replica. Measured model initialization takes 240 seconds on a cold start, while process liveness succeeds after 8 seconds. The service requires two usable replicas before a maintenance rollout proceeds.
A proposal to route traffic when the container becomes live would expose an unloaded model for roughly four minutes. The serving owner instead uses /v1/health/ready for readiness and a startup probe with failureThreshold: 36 and periodSeconds: 10, allowing approximately 360 seconds for the measured initialization. /v1/health/live serves the subsequent liveness check.
The platform owner keeps preinstalled drivers under the node-image process with driver.enabled=false. Each Pod reserves nvidia.com/gpu: 1, validated host resources, and the appropriate accelerator placement. A fifth replica stays pending on this fixed cluster; increasing the startup allowance cannot create a fifth GPU. The team verifies both scheduling and ready endpoints before counting a replica toward maintenance capacity.
Common exam traps
- Treating a Docker image as installation of the host driver and device plugin.
- Specifying unequal GPU requests and limits, or a GPU request without a limit.
- Interpreting GPU count as proof that the model fits the accelerator’s memory.
- Using container liveness as proof of model readiness.
- Restarting slow model loads because the startup allowance is too short.
- Repairing
Pendingplacement failures by changing HTTP probes.
What to remember
Provision GPU capabilities with the Operator, reserve a compatible replica with Kubernetes resources and placement, and admit inference only after model readiness. Startup protects initialization; readiness controls traffic; liveness controls restart. Diagnose the failed stage before adding replicas.
Knowledge check
A new NIM replica with a limit of nvidia.com/gpu: 1 has stayed Pending for ten minutes. A teammate proposes raising the startup probe’s failureThreshold so the replica has longer to come up. What should the serving owner do first?
failureThreshold as proposed/v1/health/live insteadkubectl describe podPending phase alone does not identify the cause. A startup probe cannot repair an unschedulable reservation, and adding replicas does not supply missing GPU resources.A NIM replica’s process passes /v1/health/live after 6 seconds, but a cold model load takes up to 210 seconds. Liveness currently checks /v1/health/ready every 10 seconds and restarts the container after three failures. Keep that liveness policy, admit no traffic before the model can answer, and protect valid loading. Which configuration meets these requirements?
/v1/health/ready ; startup on /v1/health/live with a 300-second allowance/v1/health/ready ; startup on /v1/health/ready with a 300-second allowance/v1/health/ready ; no startup probe, retaining the current liveness timing/v1/health/live ; startup on /v1/health/live with a 300-second allowance/v1/health/live returns 200 once the container runs, before the model loads, so it cannot gate traffic; /v1/health/ready returns 200 when the model is loaded and inference is available. A startup probe on model readiness suppresses liveness and readiness checks throughout loading, with an allowance above the measured 210 seconds. A startup probe on process liveness succeeds after 6 seconds and releases the aggressive liveness checks too early; its nominal 300-second allowance does not keep it active after success.A platform reviewer checks four container specs for a one-GPU inference replica on a cluster that exposes GPUs through the NVIDIA device plugin. Which spec is valid?
requests of nvidia.com/gpu: 1 , with no GPU limitrequests of nvidia.com/gpu: 1 , with a GPU limit of 2limits of nvidia.com/gpu: 1 , with no GPU requestrequests of nvidia.com/gpu: 2 , with a GPU limit of 1nvidia.com/gpu , Kubernetes uses the limit as the request when only the limit is set. When both are set they must be equal, and a GPU request without a limit is invalid. Each of the other specs breaks one of those rules.A cluster mixes nodes with 24 GB and 80 GB GPUs. The model configuration is validated on one 80 GB GPU per replica, but some replicas land on 24 GB nodes and fail during model loading. Which change places replicas correctly while keeping one GPU per replica?
nodeSelector on the nvidia.com/gpu.memory labelnvidia.com/gpu limit on each replica’s containernvidia.com/gpu.memory (and nvidia.com/gpu.product ), which a nodeSelector or node affinity can require. A longer probe allowance does not make a failing load succeed, and host memory is a different resource from GPU memory.References
NVIDIA, GPU Feature Discovery
NVIDIA, Helm and Kubernetes
NVIDIA, About the NVIDIA GPU Operator
Kubernetes, Schedule GPUs
Kubernetes, Resource Management for Pods and Containers
Kubernetes, Configure Liveness, Readiness and Startup Probes
NVIDIA, API Reference