NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.4 Implement automated tuning, retraining, and versioning in production.
Promote reproducible releases with complete rollback
Completed by the knowledge check
Learning objective
Version and promote model, workflow, and dataset changes with regression gates and rollback.
Core idea
A production agent release is a combination of artifacts, not a model name. The same model with a different prompt, tool schema, retrieval snapshot, or generation setting can produce different outcomes. Version enough of that combination to reproduce the evaluated behavior and restore the previous combination when a candidate fails.
NeMo Agent Toolkit supplies evaluation artifacts that describe a run. Kubernetes supplies workload revisions and rollout commands. Your release process must connect them: evaluation approval must identify the exact artifacts deployed, and rollback must restore every behavior-changing dependency the release changed.
See 8.3.1 for constructing release regression datasets and 8.3.2 for online version comparison. This lesson uses their results to decide which artifact bundle may move into production and which bundle to restore.
How it works
Describe the evaluated release
Create a team-owned release manifest with immutable identifiers for the model checkpoint or adapter, base model, tokenizer, workflow code, prompt files, effective configuration, and tool schemas. Include training and evaluation dataset snapshot identifiers, preprocessing revision, evaluator configuration, and the preceding approved release. These are release-record fields you define, not a built-in NeMo manifest schema.
Keep dataset roles explicit. A training snapshot explains what supervision produced the checkpoint. An evaluation snapshot explains what measured its behavior. A retrieved knowledge snapshot can alter runtime answers without changing weights. Recording one “dataset version” for all three conceals distinct changes and makes rollback ambiguous.
For containerized components, Kubernetes supports image references by digest. A tag can be moved to another image, while a digest identifies fixed content. Use the digest in the Pod template and retain the artifact it identifies. Pinning the image alone does not pin a model or prompt downloaded from a mutable external location; the release manifest must identify those artifacts too.
Declare compatibility between bundle members. For example, a revised tool schema might require a revised workflow parser. Promote them as the combination actually tested rather than advancing independent “approved” aliases. A model-only rollback into an incompatible parser is not restoration of the earlier release.
Attach regression results to the candidate
See 3.1.4 for NeMo Agent Toolkit evaluation artifacts and the remote nat eval --endpoint boundary. The approval record points to that run’s retained effective configuration and the evaluated server’s release identifier, so the release owner can verify which combination earned approval.
Make the regression decision an explicit comparison result attached to the candidate manifest. Require the task-quality floor, critical safety cases, supported tool behavior, and latency or cost budgets chosen for the service. These are team acceptance rules, not implicit toolkit defaults. A pipeline should stop promotion when a required result is missing, even if the completed evaluators report good scores.
Promote the same artifacts
Deploy the approved artifact identifiers without rebuilding or retuning them after evaluation. If a required environment difference changes behavior, such as another model artifact or tool adapter, treat that combination as another candidate and test it. The release owner approves the named bundle, including its dataset snapshots and model selection, before the serving owner deploys it.
For a NIM-served LoRA adapter, mount its artifacts under NIM_PEFT_SOURCE and select a -feat_lora profile through NIM_MODEL_PROFILE when the model has a LoRA-capable profile. Query /v1/models to discover loaded adapter IDs. Promote the adapter by changing the workflow’s request model field to the approved ID. Preserve the connection between that ID and the versioned adapter artifact in the release manifest.
Static LoRA loads adapters at startup when NIM_PEFT_REFRESH_INTERVAL is unset. Dynamic LoRA sets that interval to poll the directory and enables runtime updates for vLLM; POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter provide manual control. Keep the preceding adapter loaded for request-level rollback when GPU memory permits; multiple loaded adapters consume that capacity. An API unload is not permanent removal: a watcher can reload an adapter that remains in the directory on its next scan.
Compare the serving release identifier and the adapter ID selected by the workflow with the approved manifest before declaring promotion complete. Deployment completion alone does not establish agent correctness. See 4.2.3 for rollout and rollback mechanics and 8.5.1 for availability during updates.
Restore the previous combination
See 4.2.3 for Kubernetes rollback commands, revision retention, and the complete rollback target. Recall that restoring a Deployment restores its Pod template; this release’s restoration checklist must also identify the compatible external artifacts.
Restore the prior base model and checkpoint or adapter, tokenizer, workflow code, prompt files, tool schemas, and effective configuration named by the approved predecessor. For a still-loaded NIM adapter, restore the prior adapter ID in the request model field along with the compatible workflow configuration. Before changing traffic, query /v1/models to verify that the ID is available; a retained file alone does not establish that NIM has loaded it.
Handle datasets according to their role. A training snapshot stays attached to the model it produced; changing the training file does not undo learned weights. Restore the earlier model artifact instead. An evaluation snapshot stays attached to its historical scores; a corrected test set creates another comparison record rather than rewriting those scores. A retrieval snapshot is a runtime dependency, so restore the approved source/index combination when that change caused the regression and its contents remain valid for the service.
If the preceding retrieval snapshot contains facts that must be withdrawn, the release owner cannot approve restoring it merely because the previous model passed its tests. Build and evaluate a combination with permissible source data, or restrict the affected task until that combination is available. The release manifest records that recovery choice so the incident does not silently change the meaning of the predecessor’s approval.
Assign the serving owner the restoration action and the release owner the acceptance decision. After restoration, verify both the selected artifact identifiers and the failed task slice. Freeze further promotion of the rejected candidate until its cause is understood.
Key distinctions and trade-offs
Keep the three changes independently identifiable and jointly testable:
| Changed artifact | Required version record | Regression concern | Rollback requirement |
|---|---|---|---|
| Model | Base model, checkpoint or adapter, tokenizer | Capability, tool behavior, serving cost | Restore the compatible model bundle |
| Workflow | Code, prompts, configuration, tool schemas | Routing, validation, execution compatibility | Restore the tested code and configuration combination |
| Dataset | Training, evaluation, and retrieval snapshots by role | Leakage, changed scoring, changed source facts | Restore runtime sources where appropriate and retain historical experiment inputs |
An evaluation dataset correction should remain visible even if the live model is rolled back. Historical scores must remain tied to the dataset that produced them; silently rescoring or relabeling the record destroys comparability. Storage retention costs money, but deleting the previous model or configuration can make an otherwise valid rollback command useless.
Architecture / decision scenario
A legal research assistant has a team target of 95% citation correctness and a three-second p95 response budget. A candidate changes the model adapter, citation parser, and retrieved-document snapshot. Its retained evaluation run records 97% correctness and 2.7 seconds, so the release owner approves that exact combination.
The tempting alternative is to deploy three independent mutable aliases for model, workflow, and documents. The serving owner rejects this because the parser and adapter were tested together. The candidate manifest instead records fixed artifacts, with their runtime references in the Deployment template.
After promotion, a critical jurisdiction slice drops from 98% to 90% citation correctness while all Pods remain ready. The release owner orders rollback despite successful rollout status. The serving owner restores the earlier workload revision and its model, parser, and retrieval references, then reruns that slice. Rolling back only the model would leave the changed parser and documents in place and would not recreate the approved predecessor.
Common exam traps
- Treating a mutable image tag or endpoint URL as a reproducible release identifier.
- Rebuilding an approved bundle after testing and treating the rebuilt artifacts as already approved.
- Promoting a candidate with missing mandatory evaluator results.
- Equating healthy Pods with acceptable agent quality.
- Treating an evaluation dataset correction as a reason to rewrite historical approval scores.
- Assuming an API unload permanently removes an adapter while its directory remains watched.
What to remember
Approve the exact combination of model, workflow, and dataset artifacts that passed regression checks. Retain the effective evaluation record and the predecessor’s artifacts. Deployment revisions support recovery, but complete rollback requires restoring the compatible behavior-changing bundle and verifying the failed task slice.
Knowledge check
An agent’s Deployment references the image agent:prod, which CI retags after every build; the registry lets any tag be pushed again. The release owner needs the image that runs in production to be the one that passed regression evaluation. What should the release process change?
agent:1.4.2 in the Pod templateimagePullPolicy: Always on the agent:prod referenceA tax-filing agent’s release changed its LoRA adapter, a tool schema with its matching workflow parser, and its prompt files, and was approved as that combination. After promotion, a critical slice regresses, and the on-call engineer proposes switching the request model field back to the previous adapter ID. What should the serving owner do instead?
model field to the previous adapter IDAfter promotion, every Pod of an agent is Ready and the rollout reports success. A critical slice’s citation correctness falls from 97% to 89%, below the release policy’s mandatory 95% floor. What should the release owner order?
A NIM runs with NIM_PEFT_SOURCE pointing at an adapter directory and NIM_PEFT_REFRESH_INTERVAL=30. The operator rejects an adapter and calls POST /v1/unload_lora_adapter for it, leaving its folder in place. Later, /v1/models lists the adapter again. The operator must preserve automatic discovery of approved adapters and prevent this rejected adapter from returning after scans or restarts. What should the operator do?
POST /v1/unload_lora_adapter again after every scanNIM_PEFT_REFRESH_INTERVAL and keep the folder in placeNIM_PEFT_SOURCE-feat_lora profile through NIM_MODEL_PROFILEA candidate’s new retrieval snapshot caused a regression, and the release owner considers restoring the predecessor bundle. Since that bundle was approved, a guideline in its retrieval snapshot has been withdrawn and must no longer be served. What should the release owner decide?
References
- NVIDIA, Agent Evaluation in NVIDIA NeMo Agent Toolkit
- Kubernetes, Images
- Kubernetes, Deployments
- NVIDIA, Fine-Tuning with LoRA