NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.5 Optimize deployment costs while ensuring high availability.
Fund headroom for failures and recovery
Completed by the knowledge check
Learning objective
Balance high availability, replica headroom, failover, and recovery costs.
Core idea
High availability costs more than the GPUs needed for normal traffic. You pay for capacity that can survive a defined failure, for a path that sends requests to that capacity, and for restoring the lost reserve before another disruption occurs.
Start with the latency-qualified capacity measurement from 4.5.1. Then remove the largest failure domain you promise to tolerate and calculate what remains. Several replicas on one failed node provide no surviving capacity; surviving replicas that are already saturated provide no useful failover headroom.
Make the product owner choose the service target and the platform owner price its failure model. An availability percentage is a requirement to demonstrate, not a property conferred by a replica count.
How it works
Turn availability into a recovery budget
Define the service-level indicator (SLI) around an observable result: successful agent tasks delivered within the agreed time. A live process is insufficient if its responses miss that target. Use the availability target to decide how much disruption the service can consume over its reporting window.
For a time-based example, 99.9% permits 0.1% unavailability: 43.2 minutes in a 30-day month. A proposed recovery that takes 20 minutes consumes nearly half of that allowance in one incident. If the SLI instead counts good requests, calculate the allowance from requests; do not substitute minutes without accounting for traffic distribution.
Write a recovery timing sheet: failure detection, removal of the failed target, routing to surviving capacity, and restoration of the failed replica. Measure those intervals in a failure exercise. The user-facing interruption and time to restore reserve are different: service can recover while the fleet remains one failure away from overload.
Size surviving capacity and place it deliberately
If each replica sustains five requests per second at the required latency and peak load is 12, three replicas cover ordinary traffic but only ten requests per second remain after losing one. Four replicas leave 15 after that failure. These are illustrative capacities; use your deployment’s measurements rather than this arithmetic as a hardware benchmark.
In Kubernetes, .spec.replicas requests the replica count. Use Pod topologySpreadConstraints to express placement across a chosen topologyKey, such as kubernetes.io/hostname for nodes or topology.kubernetes.io/zone for zones. With whenUnsatisfiable: DoNotSchedule, maxSkew constrains the permitted difference in matching Pod counts across eligible domains.
Placement does not create eligible GPU capacity. Hard spread constraints can leave a replacement unscheduled when a surviving domain lacks the required hardware. Price both the active serving replicas and the compatible capacity needed to restore them. If a model replica spans multiple GPUs, losing any required part of that allocation belongs in the failure test.
Separate failover from model startup
NIM’s GET /v1/health/live returns 200 when the container is running, without requiring the model to be loaded. GET /v1/health/ready returns 200 when the model is loaded and inference is available. Use model readiness for the routing decision; process liveness does not establish serving capacity.
Kubernetes readiness probes remove unready Pods from Service traffic, while startup probes can allow initialization before liveness checks begin. Set readinessProbe, startupProbe, and livenessProbe from measured behavior; see 4.4.1 for the deployment mechanics. Their economic role is to bound replacement startup without repeated restarts.
A cold replacement spends time starting and loading its model. Preloaded serving replicas cost more idle GPU hours but can absorb traffic without waiting for that initialization. NIM Operator’s NIMCache custom resource (nimcaches.apps.nvidia.com) pre-caches model artifacts on a persistent volume so multiple NIM Pods can use one cached model. NVIDIA documents improved microservice startup time from this local cluster cache.
The cache requires the NIM Operator and a persistent-volume provisioner backed by network storage such as NFS, S3, or vSAN. Price that storage and measure the loading interval with cached artifacts. This can shorten recovery without keeping another model replica continuously loaded, but cached artifacts are not a ready serving replica. Include storage availability in the failure exercise. See 4.5.2 for the container’s NIM_CACHE_PATH; restoring artifacts is also different from restoring workflow state or uncertain tool effects, covered in 2.4.4.
Failover and model startup are separate paths: routing to surviving capacity can recover service while the failed replica is still starting and loading its model.
Budget maintenance as well as failures
A PodDisruptionBudget (PDB) limits voluntary evictions through Kubernetes’ Eviction API. Its .spec.minAvailable or .spec.maxUnavailable expresses the protected count; specify one, not both. It cannot prevent involuntary disruptions such as a node failure.
Deployment rolling updates are not limited by PDBs. Set .spec.strategy.rollingUpdate.maxUnavailable and .spec.strategy.rollingUpdate.maxSurge for that path. A rollout with zero allowed unavailability and one surge Pod requires capacity for the extra Pod while the replacement becomes ready. If the GPUs are all allocated, the rollout can stall rather than preserve availability through an imaginary spare.
Keep separate entries for steady-state GPU hours, failure reserve, rollout surge, NIMCache persistent storage, and recovery operations in the cost sheet. Removing a reserve entry changes the failure or maintenance promise; it is not a free optimization.
Key distinctions and trade-offs
The availability choices protect different intervals:
| Choice | What it buys | What you pay or still need |
|---|---|---|
| Active replica headroom | Capacity for load after a tolerated failure | Idle allocation in normal operation |
| Node or zone spread | Fewer replicas lost to one domain failure | Eligible hardware in each required domain |
NIMCache artifact pre-caching | Shorter model startup from a shared persistent artifact cache | Network-backed storage and loading time; no ready serving capacity |
| Cold replacement | Lower standing GPU allocation | Detection, scheduling, loading time, and interim overload |
| Readiness-based routing | Traffic targets loaded models | Enough ready survivors to accept that traffic |
| PDB and rollout limits | Controlled voluntary maintenance and rollout availability | Separate policies and sufficient surge capacity |
Choose the smallest allocation that satisfies the promised failure case and restoration target. Stronger placement and faster recovery can cost more than the product’s outage budget warrants; weaker choices need an explicitly weaker service promise.
Architecture / decision scenario
A retail inventory assistant receives 12 inference requests per second at peak. Its evaluated NIM configuration sustains five per second per replica. The service promises to survive one GPU-node failure and targets 99.9% time-based availability. Procurement proposes three replicas and a cold replacement that takes 20 minutes to become ready.
Three replicas provide 15 requests per second normally, but a node loss leaves only ten if each replica occupies its own node. Recovery leaves a two-request-per-second shortfall. If the shortfall counts as unavailable minutes under the service’s time-based SLI, 20 minutes of recovery consumes a large fraction of the month’s 43.2-minute allowance. A request-based SLI would instead count failed or late tasks; partial degradation cannot automatically be booked as 20 minutes of total outage. The platform lead rejects the proposal on both capacity and recovery grounds.
The accepted design uses four serving replicas on separate eligible GPU nodes, with topologySpreadConstraints using kubernetes.io/hostname. After one loss, three replicas retain a measured capacity of 15. Readiness-based Service routing sends traffic only to loaded survivors. The team verifies latency at 12 requests per second during node loss.
For planned node drains, the application owner uses PDB minAvailable: 3. For updates, the Deployment uses maxUnavailable: 0 and maxSurge: 1, with compatible capacity reserved for that surge. The infrastructure owner prices a fifth allocation for maintenance and rebuilding reserve, plus network-backed persistent storage for NIMCache. A failure exercise measures model loading from that cache and return to four ready replicas; the cached model reduces startup work but does not replace the ready survivor capacity. This design promises one-node tolerance; a zone-wide promise would require another placement and surviving-capacity calculation.
Common exam traps
- Counting normal-load capacity as though all of it remains after a failure.
- Spreading Pods without ensuring eligible spare GPUs exist.
- Using NIM liveness as proof that a model can accept inference.
- Treating a PDB as protection against node loss or as the rollout policy.
- Pricing cold replacement without the service lost while it loads.
What to remember
Price the capacity that remains after the promised failure and the time needed to restore reserve. Use deliberate placement, model readiness, and separate disruption and rollout controls. A cheaper deployment is acceptable only when its measured failure behavior still satisfies the service target.
Knowledge check
Peak load is 12 requests per second, and each NIM replica sustains 5 at the latency target. The service promises to keep serving peak load after losing any one GPU node, and eligible GPU nodes are available. Which is the smallest deployment that keeps the promise?
During a node-failure exercise, the Service sends traffic to a replacement NIM Pod while it is still loading its model, and those requests fail. The Pod’s readinessProbe calls GET /v1/health/live. Which change stops the misrouting?
readinessProbe at GET /v1/health/readylivenessProbe at GET /v1/health/readystartupProbe that calls GET /v1/health/liveminAvailable set to 3A four-replica NIM Deployment is updated with maxUnavailable: 0 and maxSurge: 1, and a PodDisruptionBudget sets minAvailable: 3. Every GPU in the cluster is allocated, and the rollout’s new Pod stays Pending. What explains the stall?
maxUnavailable: 0 with maxSurge: 1 means an extra Pod must start before an old one stops, and that Pod needs a free GPU allocation. Readiness gates traffic to loaded models; it does not supply the missing GPU allocation. Reserve capacity for the surge, or accept a weaker rollout promise.A deployment of four ready NIM replicas, one per node, each sustaining 5 requests per second, must serve a 12-request peak through the loss of one node. To cut idle GPU hours, the team proposes running three replicas and relying on NIMCache, from which a replacement loads its model in six minutes. What does the change do to the one-node promise?
References
- NVIDIA, Caching NIM Models
- NVIDIA, API Reference
- Kubernetes, Pod topology spread constraints
- Kubernetes, Disruptions
- Kubernetes, Specifying a Disruption Budget for your Application
- Kubernetes, Deployments
- Kubernetes, Configure Liveness, Readiness and Startup Probes
- Google, Site Reliability Engineering: Service Level Objectives