NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.5 Ensure continuous uptime, transparency, and trust in live deployments.
Preserve service through updates and outages
Completed by the knowledge check
Learning objective
Deploy updates without downtime and fail over to fallback models during dependency outages.
Core idea
Continuous service requires two different recovery paths. A planned update replaces a working serving revision while preserving enough capacity and finishing accepted requests. A dependency outage requires another qualified inference path, or an explicit degraded outcome when that path cannot perform the task safely.
See 4.2.3 for canary, blue-green, and shadow delivery, and 8.4.3 for promotion and rollback ownership. Here you decide whether the live service can actually survive replacement and dependency loss. A successful rollout command is insufficient: users need complete responses, compatible tool behavior, and bounded waiting throughout the transition.
How it works
Keep capacity available during replacement
The NIM Helm chart deploys a StatefulSet by default (statefulSet.enabled: true). Set statefulSet.enabled: false to create a Deployment; only in that mode does the chart apply deployment.strategy. The Deployment controls below do not apply to the default StatefulSet. See 7.2.3 and 4.2.3 for workload-mode and rollout choices.
A Kubernetes Deployment uses .spec.strategy.type: RollingUpdate to scale replacement ReplicaSets up while scaling previous ones down. Set .spec.strategy.rollingUpdate.maxUnavailable to 0 when the rollout must preserve the desired available replica count. Give .spec.strategy.rollingUpdate.maxSurge a positive value so a replacement can start before an available replica leaves. These fields cannot both be zero.
This is a capacity commitment. A surge replica needs schedulable resources while the existing replicas still occupy theirs. If the replacement cannot obtain a GPU, inspect the pending Pod and provision the required headroom rather than deleting a healthy replica to make progress. See 4.5.3 for recovery capacity and placement. Account for terminating Pods too: Kubernetes documents that their resource use can persist during the grace period, exceeding the apparent replicas + maxSurge bound.
A NIM health probe distinguishes process health from inference readiness. GET /v1/health/live returns 200 when the container is running; GET /v1/health/ready returns 200 when the model is loaded and ready to accept inference requests. Use the latter for a Kubernetes readinessProbe. A startupProbe delays liveness and readiness probing until startup succeeds, avoiding premature restarts while initialization is legitimately slow. Size its allowance from measured cold starts; see 7.2.1 for startup diagnosis.
Use .spec.minReadySeconds when a replacement must remain ready without crashing before Kubernetes counts it as available. That stabilizing interval is distinct from a task-quality test. A model can load successfully yet produce invalid tool arguments, so keep the release’s regression checks as a separate condition for promotion.
Finish accepted work before removing its worker
A ready replacement does not transfer an existing token stream from the terminating Pod. Kubernetes graceful termination runs a configured preStop hook before sending the termination signal, within the Pod’s terminationGracePeriodSeconds. Configure the serving process and routing layer to stop admitting work and finish accepted requests during that interval. Test the shutdown behavior with real streamed responses, not only short health requests.
A replacement is ready and counted as available before a terminating Pod receives its preStop hook and finishes accepted requests.
The Pod-level grace period defaults to 30 seconds. If your accepted request can legitimately run longer, the default can end it before completion. Include hook time and request completion in the configured allowance. Terminating endpoints always report ready: false, so load balancers stop sending them regular traffic; a draining-aware load balancer reads the endpoint’s actual readiness from the EndpointSlice serving condition.
For a stalled rollout, .spec.progressDeadlineSeconds reports ProgressDeadlineExceeded; it does not automatically restore the previous revision. The Deployment controller keeps retrying. Let the release controller or operator respond to that condition, and use kubectl rollout undo deployment/DEPLOYMENT_NAME when restoring the previous Deployment revision is appropriate. DEPLOYMENT_NAME identifies your Deployment. Preserve the broader rollback target from 8.4.3 when configuration or workflow changes extend beyond that resource.
Qualify the alternate inference path
In a Python LangChain agent, ModelFallbackMiddleware passed through create_agent(..., middleware=[...]) supplies an ordered set of alternative models when the primary model fails. Its first_model and additional positional models accept model identifiers or BaseChatModel instances. NVIDIA documents LangChain integration with NeMo Agent Toolkit through nvidia-nat[langchain] and lists NVIDIA NIM among LangChain’s LLM providers. Use NIM-served models as primary or fallback through LangChain’s model interface; ModelFallbackMiddleware is configured in the Python LangChain agent, not as a toolkit YAML component.
An alternative must survive the failure you are covering. A second model behind the failed dependency is not independent recovery. Document its deployment location, resource availability, data-handling approval, and supported task classes in the service’s fallback policy. Test it with the same structured responses, tools, and output checks as the primary; a model that can answer prose may still be unsuitable for a workflow that must issue tool calls.
Keep recovery inside the remaining request deadline and retry budget from 2.4.2. Use the circuit-breaker policy from 2.4.3 to avoid repeatedly admitting traffic to a failing dependency. Fallback does not repair an unavailable retrieval source, and changing the model cannot establish whether an earlier external write succeeded. Preserve the action-recovery rules in 2.4.4. Make degradation visible; see 8.5.2.
Key distinctions and trade-offs
| Choice | Deciding condition | Mechanism | Cost or boundary |
|---|---|---|---|
| Rolling replacement | Deployment-mode revision can overlap with replacement | RollingUpdate, maxUnavailable: 0, positive maxSurge | Requires simultaneous capacity and tested draining |
| Recreate replacement | Service interruption is acceptable | .spec.strategy.type: Recreate | Existing Pods are killed before replacements are created |
| Qualified model fallback | Alternate path survives the dependency failure and passes task checks | LangChain ModelFallbackMiddleware | Reserved capacity, repeated inference cost, and possible reduced capability |
| Explicit degraded service | No qualified alternate can complete the affected task | Service fallback policy and terminal outcome | Lower coverage, but a clear limit rather than fabricated completion |
Architecture / decision scenario
A manufacturing maintenance assistant serves read-only troubleshooting queries. The team runs the NIM Helm chart in Deployment mode with statefulSet.enabled: false. Three NIM replicas each sustain 8 requests per second within the team’s measured latency target; peak demand is 20. Each replica consumes one identical GPU allocation, and a fourth allocation is reserved for updates. These are scenario measurements, not NIM performance guarantees.
The platform owner proposes maxUnavailable: 1 to save surge capacity. Two replicas supply only 16 requests per second, below peak demand. The release owner instead chooses maxUnavailable: 0, maxSurge: 1, readiness against /v1/health/ready, and a measured draining allowance. For 45-second accepted streams and a 5-second shutdown hook, the team tests a 60-second grace period rather than accepting the 30-second default.
A routing outage then makes the primary NIM endpoint unreachable. A separately deployed fallback handles only 12 requests per second and passed troubleshooting tests but not diagnostic-image analysis. Routing all 20 requests per second to it creates another overload. The application owner permits text-only requests within the measured capacity and reports image diagnosis as unavailable. A rollout preserves capacity for planned replacement; qualified, bounded degradation handles the different failure. Neither decision authorizes unsupported answers.
Common exam traps
- Expecting
maxSurgeto apply to the NIM chart’s default StatefulSet. - Setting
maxUnavailable: 0without providing schedulable surge resources. - Assuming a progress deadline performs automatic rollback.
- Expecting an in-flight stream to move to the replacement Pod.
- Sending every task to a weaker fallback regardless of capacity or capability.
What to remember
Preserve available capacity, wait for inference readiness, and drain accepted work. Treat fallback as a separately qualified service path with its own failure isolation and capacity limits. When that path cannot complete the task, report a bounded degraded outcome instead of turning transport availability into a claim of task success.
Knowledge check
A team runs the NIM Helm chart with statefulSet.enabled: false. Four replicas each sustain 10 requests per second within the latency target, peak demand is 35, and one spare GPU allocation is reserved for updates. The next model update must keep peak capacity available throughout the rollout. Which RollingUpdate setting meets that requirement?
maxUnavailable: 1 with maxSurge: 0maxUnavailable: 0 with maxSurge: 1maxUnavailable: 0 with maxSurge: 0maxUnavailable: 1 with maxSurge: 1maxUnavailable: 0 , the rollout never drops below four available replicas, and maxSurge: 1 lets the reserved GPU host each replacement before an old replica leaves. Allowing one replica to be unavailable permits capacity to fall to 30 requests per second against a peak of 35; a positive surge allowance does not require the controller to preserve all four available replicas. The two fields cannot both be zero.During a rolling update of a NIM Deployment, users get errors from replacement Pods that receive traffic while their model is still loading. The Pods’ readiness probe calls /v1/health/live. Which change stops traffic from reaching a replacement before it can serve inference?
terminationGracePeriodSeconds on the Pod speclivenessProbe at /v1/health/ready.spec.progressDeadlineSeconds on the DeploymentreadinessProbe at /v1/health/ready/v1/health/live returns 200 as soon as the container is running, while /v1/health/ready returns 200 only when the model is loaded and ready for inference, so it is the endpoint that should gate traffic. Using it for liveness instead would treat a slow load as a failure and risk restarting the container; the grace period governs terminating Pods, and the progress deadline only reports a stalled rollout.A NIM Deployment rolls out with maxUnavailable: 0 and a positive maxSurge. Replacements become ready and stay stable, yet users still see streamed answers cut off as old Pods terminate. Accepted streams can run for 90 seconds, the preStop hook stops admitting new work and takes 5 seconds, and the server is configured to finish accepted streams after the termination signal. The Pod spec leaves the grace period at its default. What fixes the truncation?
terminationGracePeriodSeconds for the hook and the longest streammaxSurge so that two replacements start in each rollout step.spec.minReadySeconds to the longest accepted stream durationA NIM Deployment update changes only the Pod template. The new revision’s Pods fail during model load because of a configuration error, and after the progress deadline the Deployment reports ProgressDeadlineExceeded. The old replicas still serve because maxUnavailable is 0. What should the release process do?
.spec.progressDeadlineSeconds so the rollback starts soonerkubectl rollout undo for the DeploymentmaxUnavailable so that the stalled rollout can completeA Python LangChain agent calls a NIM endpoint through a cluster gateway, its workflow must issue structured tool calls, and its customer data is approved only for in-house processing. You add ModelFallbackMiddleware to cover an outage of that gateway. Every candidate below has enough measured capacity for the fallback traffic. Which candidate qualifies as the fallback?
References
- Kubernetes, Deployments
- Kubernetes, Liveness, Readiness, and Startup Probes
- Kubernetes, Pod Lifecycle
- NVIDIA, Helm and Kubernetes
- NVIDIA, API Reference
- NVIDIA, Framework Integrations
- LangChain, Built-in middleware: Model fallback