NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.2 Apply MLOps practices for continuous integration and continuous delivery (CI/CD) workflows, monitoring, and governance.
Pin the dependencies behind agent behavior
Completed by the knowledge check
Learning objective
Version prompts, agent graphs, tool contracts, models, and datasets together.
Core idea
The release unit of an agent is the combination that produced its measured behavior. A code commit does not fully identify that combination when it loads an editable prompt file, calls a mutable model endpoint, or retrieves from a changing corpus. Version those dependencies together through a release record whose entries resolve to retained artifacts.
Together does not mean putting model weights and private datasets into the application repository. It means one release identifier selects a coherent set of immutable references, access-controlled where necessary. The team can then explain what passed evaluation and restore that combination when a later candidate fails.
See 2.1.4 for prompt-model regression testing and 3.1.4 for evaluation artifacts. This lesson extends their scope to the full deployable agent and its dependencies.
How it works
Identify configuration and executable behavior
NeMo Agent Toolkit organizes a workflow through configuration sections such as functions, llms, and workflow. Component _type values select registered implementations. For example, a workflow’s tool_names and llm_name reference components elsewhere in the configuration. The YAML and the installed implementations jointly determine behavior.
Record the application source revision, dependency lockfile, runtime image digest, and workflow configuration hash in your release record. If an agent graph is built in Python, the graph-building module belongs to that source revision; if orchestration is selected through YAML, preserve that YAML too. A graph’s diagram or exported picture cannot replace its executable definition, routing conditions, and component versions.
The toolkit supports file:// references for string content, including system_prompt and function descriptions. Relative file paths resolve from the configuration file’s directory, and loading inserts plain text without rendering Jinja templates. Package and identify the referenced files with the configuration. A YAML hash alone leaves behavior ambiguous when a referenced prompt file changes at the same path.
Environment interpolation creates another dependency: ${VAR} and ${VAR:-default} resolve values outside the YAML. Record effective nonsecret settings and the identity of controlled secret bindings. Keep secret values in their secret store. A release needs the intended endpoint and authorization scope without publishing credentials in an artifact bundle.
Preserve tool compatibility and model execution
For each tool, retain the callable implementation, its advertised name and description, and its input and result schemas. Record the version of the external service interface that the adapter expects. Distinguish a compatible interface update from a semantic change: accepting the same arguments does not make a tool equivalent if it changes units, authorization rules, or the meaning of a successful result.
Treat a changed tool description as a behavioral change even when the adapter code is identical. The model uses that description to choose the tool. Treat a changed output schema as an integration change even when a prompt still generates valid inputs. These dependencies belong to the same release comparison; see 2.3.1 for tool-interface design.
For a self-hosted NIM, preserve the serving image digest, model artifact identity, and selected execution profile. Kubernetes image digests identify fixed image content; tags can move. The llms entry’s _type: nim and model_name identify the application’s selected model interface, but the deployment record must also identify the server that answers it.
Use list-model-profiles to discover profiles in the intended NIM image and retain the full 64-character ID selected through NIM_MODEL_PROFILE. Check the ID when changing image versions because a profile change can change its ID. Preserve launch arguments as well: direct container arguments, NIM_PASSTHROUGH_ARGS, backend-related environment settings, and runtime_config.json can override profile settings. A profile identifier without those settings leaves the actual execution configuration incomplete.
Identify data and the measurement that approved it
Record the evaluation dataset snapshot, labels, split definition, evaluator configuration, and judge identity. For retrieval agents, also identify the source-corpus snapshot and index-building configuration. An unchanged benchmark with a changed retrieval corpus tests a different system. Restrict access to sensitive snapshots while preserving identifiers that authorized operators can resolve.
The toolkit’s eval.general.dataset.file_path selects a dataset location, not a guarantee that its bytes stay fixed. Your release record therefore needs a retained snapshot or content hash in addition to the path. Keep the evaluated data identity separate from the production knowledge-corpus identity; one supplies cases and expected outcomes, while the other supplies information to the running agent.
nat eval saves config_original.yml, config_effective.yml, and config_metadata.json. Attach those artifacts and per-case results to the release identifier; see 3.1.4 for keeping them per run. For --endpoint evaluation, the saved configuration describes the evaluation and target URL, not the remote workflow configuration. Join it to the server’s independently recorded release identity before treating the run as approval of a deployable candidate.
Key distinctions and trade-offs
Use a release record to resolve each behavioral dependency:
| Dependency | Retained identity | Failure if only a convenient label is saved |
|---|---|---|
| Prompts | Referenced file content and configuration revision | Same file:// path loads different instructions |
| Agent graph | Graph-building source, YAML, and runtime dependencies | Same graph name selects changed branches or implementations |
| Tool contracts | Description, schemas, adapter, and expected service version | Calls remain syntactically valid but change meaning |
| Models | Model artifact, serving image digest, profile ID, and launch settings | Same endpoint name executes a different configuration |
| Datasets | Snapshot, labels, splits, and evaluator or index configuration | Same file path or index name supplies different data |
This increases artifact retention and coordination cost. It also lets teams release dependencies at different cadences: a corpus update can have its own candidate identifier and tests while retaining the same application image. Version together at approval time; rebuild only the artifacts whose content changes.
Architecture / decision scenario
A manufacturing research assistant compares maintenance bulletins. Its two-person platform team must reconstruct the tested configuration behind approved answers for 90 days. Release R17 uses prompt P8, graph G5, tool interface T3, a pinned NIM image and profile, corpus C12, and evaluation set E6. These labels are team-defined identifiers backed by retained hashes and artifact locations.
A proposed shortcut tags only the application image. In a trial, 93% of cases pass; a later run reaches 85% despite the same application tag. Investigation finds that the prompt file was edited and the bulletin index rebuilt. The image label cannot distinguish either change, so it cannot identify the approved combination or a complete rollback target.
The application owner creates R18 as one release record covering all five dependency groups. The data owner preserves C12 and its index recipe; the model-serving owner records the image, profile, and launch settings. Evaluation runs against the identified server release with E6. If the team cannot retain C12 under its data policy, it must resolve that retention decision before promising reconstruction for 90 days. The tempting image-only shortcut is cheaper to label but fails the stated operational requirement.
Common exam traps
- Equating a source commit with all inputs that determine agent behavior.
- Hashing YAML while leaving its
file://prompt contents mutable. - Treating an endpoint URL or model name as an immutable serving configuration.
- Assuming
NIM_MODEL_PROFILEoverrides every container launch setting. - Treating an unchanged input schema as proof that tool semantics are unchanged.
- Assuming remote evaluation artifacts contain the deployed server’s graph and configuration.
What to remember
Approve an identified combination of prompts, graph, tools, model execution, and data. Resolve every release entry to retained content, preserve effective settings, and join remote evaluation results to the server release. Restore the complete combination when rollback requires it; an image revision alone may cover only one part.
Knowledge check
An agent’s release record stores the source commit, the runtime image digest and a hash of the workflow YAML. The YAML sets system_prompt: file://prompts/triage.txt; that prompt file contains Jinja {{ }} placeholders and is edited outside the release process. What must the record add to identify the instructions the approved candidate ran with?
prompts/triage.txt packaged with the YAML${VAR} interpolation resolvesprompts/triage.txt resolved from the working directoryfile:// value inserts the file’s plain text at load time, so a YAML hash covers only the path, and the same path can load different instructions. Packaging the referenced file and recording its content identity closes that gap. A relative path resolves from the configuration file’s directory, and loading does not render Jinja, so neither a working-directory path nor a rendered template identifies the text the model received.A team edits only the description of a lookup_order tool so that it mentions refunds; the adapter code and the input and output schemas are unchanged. The release owner must decide whether the agent’s last approval still covers the change. How should the change be classified?
A self-hosted NIM’s release record stores the serving image digest and the 64-character NIM_MODEL_PROFILE ID. The deployment also sets NIM_PASSTHROUGH_ARGS with backend arguments. A reviewer argues that the profile ID already fixes the execution configuration. What is missing from the record?
model_name from the application’s llms entryNIM_PASSTHROUGH_ARGS , backend-mapped environment variables and runtime_config.json over the selected profile’s settings, so a profile ID without them leaves the actual execution configuration incomplete. The llms entry’s model_name names the interface the application calls, not the server that answers it, and the full ID identifies a profile more exactly than its description.A team evaluates a deployed agent with nat eval --endpoint and attaches config_effective.yml and the per-case results to release R9 as its approval evidence. What must be joined to the run before it approves a deployable candidate?
config_original.yml from a second, repeated runconfig_metadata.json that records the --endpoint URL--endpoint , the saved configuration captures the dataset, evaluators and target URL, not the workflow configuration running on the remote server. The run approves a candidate only once it is joined to that server’s own release record; a local checkout can differ from what the server runs, and a URL can serve a different release tomorrow.References
- NVIDIA, Workflow Configuration
- NVIDIA, Agent Evaluation in NVIDIA NeMo Agent Toolkit
- NVIDIA, Model Profiles and Selection
- Kubernetes, Images