NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.2 Apply MLOps practices for continuous integration and continuous delivery (CI/CD) workflows, monitoring, and governance.
Choose canary, blue-green, or shadow rollouts
Completed by the knowledge check
Learning objective
Roll out with canary, blue-green, or shadow deployments under change approval and rollback.
Core idea
A rollout strategy decides which candidate receives traffic, which output reaches users, and how quickly you can return to the approved baseline. Passing predeployment checks establishes eligibility for this decision; it does not establish acceptable behavior under production traffic.
Choose exposure and recovery together. A candidate that answers only a small cohort still has real authority if its tools can modify business records. A candidate whose answer is hidden can also cause damage if a duplicated request executes a write. Track user exposure, tool side effects, model-serving capacity, and release identity separately.
See 4.2.1 for predeployment checks and 4.2.2 for the versioned combination you promote or restore. This lesson owns the controlled transition between those combinations.
How it works
Choose what the candidate is allowed to affect
In a canary, the candidate serves a bounded subset of live work and its responses affect that subset. Kubernetes documents separate Deployments with track: stable and track: canary. A Service selects their common labels, omitting track, to reach both releases; changing replica counts controls their traffic ratio. Keep the Deployment selectors nonoverlapping so their controllers do not fight over Pods.
The shared Service selects Pods across both releases; it does not encode your agent-session release policy. When a conversation must retain its graph and tool expectations, assign its cohort through application routing and record a team-defined release_id with sessions and request logs.
In blue-green delivery, prepare separate baseline and candidate environments, then switch the production route after validation. A Kubernetes Service’s spec.selector selects its backend Pods by labels, so an application can use disjoint release labels to select the intended environment. Retain the previous environment until the recovery window closes. The traffic switch does not itself move in-flight agent state or reverse completed actions; define how existing sessions drain or retain their assigned release.
In shadow delivery, send a copy of selected inputs to the candidate for observation while the baseline alone supplies the user-facing response. Implement the duplication in the application dispatcher and direct candidate tool calls to read-only adapters or isolated test systems. Suppressing candidate text is insufficient if it can still send email, create bookings, or update records. Record the candidate’s proposed actions for comparison without executing production mutations.
In shadow delivery, the dispatcher duplicates selected inputs, and the baseline alone supplies the user-facing response.
Make approval an operational record
NIST AI RMF GOVERN 2.1 calls for documented risk responsibilities, while MANAGE 4.1 includes post-deployment monitoring, incident response, recovery, and change management. Apply those practices through a change record containing the candidate release, baseline release, passed checks, rollout mode, initial cohort, monitoring window, thresholds, approver, and rollback owner.
Approval must name what may expand automatically and what requires another decision. For example, a team can approve progression from a small cohort only after a minimum case count and an error threshold both pass. A mandatory policy violation can stop progression regardless of a favorable task-success average. These are organization-defined rules, not defaults supplied by Kubernetes or NeMo Agent Toolkit.
Join request results to release_id and cohort. The release dashboard should show completed and failed tasks, latency, safety outcomes, tool effects, and model-serving resource consumption for each release. Store the monitoring query and decision in the change record. See 8.1.2 for GPU and serving metrics and 8.3.2 for online comparison; here monitoring determines whether to continue this rollout.
Distinguish stalled rollout detection from recovery
Kubernetes RollingUpdate uses maxSurge and maxUnavailable to control replacement capacity and availability. These settings govern Pod rollout, not the application’s task quality or cohort assignment. Extra model-serving Pods need GPU headroom; see 4.5.3 for sizing it.
The NIM Helm chart defaults to statefulSet.enabled: true; set it to false to create a Deployment, where deployment.strategy applies. NVIDIA documents deployment.strategy.type: Recreate when no extra GPU Pod fits; deployment.strategy is ignored in StatefulSet mode. Recreate terminates existing Pods before creating replacements, interrupting that model service. Keep the baseline as a separate release when replacement downtime is unacceptable.
The following controls apply to Deployments, not the chart’s default StatefulSet. A Deployment’s .spec.progressDeadlineSeconds sets the deadline for reporting lack of progress. A stalled rollout receives a Progressing condition with reason: ProgressDeadlineExceeded. Kubernetes reports that condition but does not automatically roll back the Deployment. Your delivery automation or operator must decide and act. An available candidate can also need rollback because its answers or tool actions violate the approved rule.
kubectl rollout undo deployment/DEPLOYMENT_NAME restores a previous Deployment revision; DEPLOYMENT_NAME denotes the affected Deployment. Preserve revision history through .spec.revisionHistoryLimit if you depend on that mechanism. Restoring a Pod template does not restore every external prompt, corpus, or tool dependency. Use the release bundle from 4.2.2 and return the route to the approved baseline as part of the recovery plan.
Key distinctions and trade-offs
Choose the mode by the exposure you can accept:
| Mode | Deciding condition | Mechanism | Cost or limitation |
|---|---|---|---|
| Canary | You need bounded live behavior before broad exposure | Separate Deployments with common Service labels or application cohort routing | Exposes real users and actions; small cohorts accumulate results slowly |
| Blue-green | You need a prepared replacement and a route back | Separate environments plus Service spec.selector or application routing | Needs duplicate capacity and a plan for existing sessions |
| Shadow | You need production-shaped inputs before serving candidate answers | Dispatcher copies plus isolated candidate tool adapters | Adds inference and tool-read load; cannot directly measure candidate user satisfaction |
These modes can form a sequence. Shadow observation can precede a canary, and a prepared baseline can remain available during either. Each additional copy consumes capacity; count downstream model and tool work, not only frontend replicas. Sharing a saturated model endpoint can make a shadow test degrade baseline latency.
Architecture / decision scenario
A travel company’s booking assistant serves 60 requests per second. Its approved latency objective is p95 below 4 seconds, and the release owner requires tool-error rate below 1% with no duplicate bookings. The candidate and baseline initially share a model endpoint tested for 80 requests per second at the scenario’s request mix.
The team proposes full-traffic shadowing because candidate text would be hidden. That would raise offered model work to approximately 120 requests per second for this one-call-per-request scenario, exceeding the tested operating point. It would also duplicate booking attempts if the original write adapter remained available. The release owner rejects that proposal.
Instead, the team shadows 10%, approximately 6 extra requests per second, with booking writes replaced by an isolated recorder. After comparison and renewed approval, it serves a 5% live canary with stable session assignments. After the required minimum of 2,000 candidate requests, tool errors reach 3%, against 0.4% for the baseline, so the approved 1% stop condition triggers recovery.
The on-call owner returns candidate sessions through the application’s defined recovery path and stops assigning fresh sessions to it. The team restores the baseline release references and reconciles uncertain booking outcomes rather than assuming rollback canceled them. A blue-green replacement remains an option only after reserving enough capacity for both environments. The change record captures the observed rates, stopped progression, and recovery decision.
Common exam traps
- Equating
RollingUpdatewith application-level canary assignment. - Assuming shadow output suppression prevents candidate tool side effects.
- Doubling production inference without checking shared serving capacity.
- Treating
ProgressDeadlineExceededas an automatic rollback action. - Treating an available Pod as proof of safe task execution.
- Assuming
kubectl rollout undorestores external data or reverses completed business actions.
What to remember
Choose the rollout mode, monitor by release, and assign recovery ownership before routing traffic. Restore the approved combination and reconcile side effects separately. Check whether the serving workload is a StatefulSet or Deployment before choosing update and rollback controls.
Knowledge check
A team shadows a candidate travel agent on copies of live requests and discards its responses, so customers see only the baseline’s answers. The candidate’s tools include one that creates reservations. What else must the shadow setup do?
A support assistant serves 50 requests per second, and both baseline and candidate make one model call per request. Its shared model endpoint has been tested to 70 requests per second at this request mix. The team wants production-shaped inputs for a candidate without its answers reaching users or the shared endpoint exceeding its tested load. Which plan fits?
A NIM Deployment’s rollout stalls because the new Pods fail their readiness checks, and its status shows Progressing with reason: ProgressDeadlineExceeded. The approved recovery procedure requires restoring the previous revision. What initiates that recovery?
.spec.progressDeadlineSeconds so the rollout can finishmaxUnavailable so the previous Pods keep servingkubectl rollout undo . A longer deadline only delays the report for Pods that are failing, and maxUnavailable shapes replacement capacity rather than reverting the change.A team installs a NIM with the NIM Helm chart’s default values. No spare GPU fits a second model Pod during upgrades, so it adds deployment.strategy.type: Recreate, yet upgrades still do not use that strategy. Which additional setting makes the strategy apply?
statefulSet.enabled: false to use a DeploymentmaxSurge: 0 to remove the extra Pod allowanceprogressDeadlineSeconds to allow a longer rolloutrevisionHistoryLimit to retain the prior revisionstatefulSet.enabled: true , and deployment.strategy applies only in Deployment mode, so the team sets statefulSet.enabled: false for it to take effect. Recreate then terminates the old Pod before creating the new one, which fits a cluster with no spare GPU but interrupts the model service during the swap; maxSurge belongs to RollingUpdate .During a canary, a booking agent’s tool-error rate crosses the approved stop threshold. The candidate changed the image, a prompt file mounted from outside the image and the retrieval index, and some of its bookings ended with an uncertain outcome. The on-call engineer runs kubectl rollout undo on the agent Deployment. What else does recovery require?
kubectl rollout undo restores a previous Pod template; it does not restore an external prompt file or a rebuilt index, and it does not reverse bookings already made. Recovery returns every reference in the approved release bundle to the baseline and reconciles uncertain booking outcomes instead of assuming the rollback canceled them.References
- NVIDIA, Helm and Kubernetes
- Kubernetes, Managing Workloads
- Kubernetes, Deployments
- Kubernetes, Service
- NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0)