NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.2 Apply MLOps practices for continuous integration and continuous delivery (CI/CD) workflows, monitoring, and governance.
Require four checks before promoting agents
Completed by the knowledge check
Learning objective
Gate delivery with automated unit, integration, safety, and evaluation checks.
Core idea
An agent release can pass deterministic tests and still choose the wrong tool, disclose restricted information, or miss the business task. Continuous integration therefore needs several kinds of results, each answering a different release question. A completed test command is insufficient when required cases were skipped or the candidate used different dependencies from production.
Treat promotion as a decision over a named candidate and a required check matrix. The matrix records the suite, required cases, acceptance rule, result location, and accountable owner. NIST AI RMF’s GOVERN 1.4 and GOVERN 2.1 support this separation between documented risk controls and assigned responsibility. Your team defines the thresholds; the framework does not prescribe a universal agent score.
See 3.1.4 for repeatable evaluation execution and 8.3.1 for golden datasets and per-release regression runs. Here the additional decision is whether the combined results authorize delivery.
How it works
Separate code correctness from connected behavior
Unit checks exercise deterministic components with controlled dependencies: argument validation, tenant filtering, result parsing, and failure classification. They answer whether the code implements its stated behavior. Keep these checks cheap enough to run on every proposed change. Mocking a model is useful here because you are testing how your code handles a response, rather than whether a model generates it.
NeMo Agent Toolkit uses pytest and documents additional flags through its test infrastructure. Integration tests marked with @pytest.mark.integration require --run_integration; slow tests marked with @pytest.mark.slow require --run_slow. The documented integration invocation is pytest --run_slow --run_integration. These conventions belong to the toolkit’s test infrastructure, including third-party workflows that install nvidia-nat-test; they are not universal pytest flags.
An integration check exercises actual boundaries: workflow configuration, the selected model endpoint, tool adapters, and required services. The toolkit’s run_workflow test utility executes a configured workflow. Its default expected-answer check is case-insensitive matching, which is suitable for a smoke assertion but does not establish semantic correctness across a business workload.
Missing services or API keys can cause integration tests to skip. The toolkit documents --fail_missing to turn missing prerequisites into failures. In a required delivery job, use that behavior and inspect collected, passed, failed, and skipped case counts. Repair the missing service or credential binding before rerunning; reducing the required suite would change the approval rule.
Test safety as an explicit policy outcome
Safety checks need adversarial and permitted interactions. An agent that refuses every request may block harmful output while failing its intended service. Specify both what must be refused and what legitimate work must remain possible.
NeMo Guardrails’ policy-based evaluation uses policies and interactions, with interaction sets associating inputs with expected_output entries describing expected behavior. nemoguardrails eval run runs interactions through the guardrails configuration. nemoguardrails eval check-compliance checks policy compliance using an LLM judge. The workflow also reports resource usage and latency impact. Keep the evaluated rails, policies, judge, and interactions associated with the candidate. This evaluates the rails configuration, not the complete toolkit workflow and its tools. Privacy cases that depend on tool access or tenant data must also exercise the full workflow, for example as required nat eval dataset cases inspected through per-case results.
A judge’s compliance result is an assessment of the supplied cases. For a release-blocking privacy violation, retain the offending interaction and have the policy owner adjudicate disputed results. Use benign cases to measure unnecessary refusal separately from violations. See 3.1.2 for judge calibration; delivery consumes that calibrated assessment rather than inventing a different judge for each release.
Require measured task outcomes
NeMo Agent Toolkit’s nat eval --config_file=EVAL_CONFIG_PATH runs the configured dataset and evaluators. EVAL_CONFIG_PATH denotes your evaluation configuration. The full CLI runtime requires nvidia-nat[eval]; the base nvidia-nat-eval package serves a different, standalone evaluation use case.
The eval.general.dataset section selects inputs, while eval.evaluators selects scoring. workflow_output.json records per-case execution results. Each configured evaluator emits its own output file, and config_effective.yml preserves configuration after command-line overrides. Retain those artifacts so the release job can distinguish an unacceptable result from a missing evaluator output.
Implement the promotion decision in your CI job: require the expected files and case counts, compare the required scores with approved thresholds, and fail if any required result is absent or unacceptable. Treat the evaluator’s numeric output as input to this decision. A console summary or an average alone cannot answer whether every mandatory privacy or high-consequence case passed.
Key distinctions and trade-offs
The four checks complement each other:
| Check | Deciding question | Mechanism | Cost and limitation |
|---|---|---|---|
| Unit | Does deterministic code behave correctly? | pytest assertions over controlled inputs | Fast, but mocks hide real integration failures |
| Integration | Do deployed components work together? | Toolkit integration markers, --run_integration, --fail_missing | Requires services and credentials; a smoke answer is shallow |
| Safety | Does the candidate follow required policies? | Guardrails policy evaluation plus full-workflow safety cases | Adds judge calls and adjudication; depends on case coverage |
| Evaluation | Does the workflow accomplish representative tasks? | nat eval, eval.evaluators, per-case artifacts | Consumes inference budget and has measurement variation |
Run independent checks concurrently where practical, but combine their outcomes before promotion. Fast unit feedback should not wait for a long benchmark. Conversely, a slow required benchmark should not silently become optional because the unit suite passed.
The promotion decision combines all four outcomes and fails if any required result is absent or unacceptable.
Architecture / decision scenario
A regional insurer is releasing an employee claims assistant. The release window is 40 minutes, and its allocation permits 500 model-backed case executions per candidate. Its required matrix contains 240 unit cases, 24 connected integration cases, 120 rail-safety interactions, and 300 full-workflow task cases, including cross-claimant privacy cases. These are scenario requirements, not product defaults.
Unit checks run on every change and take 2 minutes. After 5 minutes of release-candidate setup, the integration, rail-safety, and task suites take 6, 8, and 28 minutes respectively at their allocated concurrency. Sequential execution would take 49 minutes, exceeding the window. Running those three suites in parallel takes 35 minutes including setup and unit checks, and consumes 444 model-backed case executions. The release owner keeps every required case rather than shrinking the task suite to meet the deadline.
A developer proposes approving any candidate above 90% task success. Candidate B reaches 94%, compared with 91% for the baseline, but a required cross-claimant case in the full-workflow nat eval suite exposes another claimant’s record. Passing all 120 rail-safety interactions does not override that failure. Eight integration cases also skipped because a service credential was absent.
The application owner repairs the credential binding and reruns required integration with --fail_missing. The privacy owner requires a workflow fix before promotion. The repaired candidate runs the complete release matrix against unchanged scoring rules; its separate candidate allocation covers that run. Only complete passing results authorize delivery to the rollout approval in 4.2.3.
Common exam traps
- Treating a successful test process with skipped prerequisites as a completed integration check.
- Assuming toolkit-specific pytest flags work without its test infrastructure.
- Using a smoke test’s expected-answer match as the entire quality benchmark.
- Allowing a higher average task score to override a mandatory safety failure.
- Checking only adversarial requests and missing excessive refusals of permitted work.
- Retaining scores without the candidate identity, cases, and effective configuration.
What to remember
Unit, integration, safety, and task evaluation establish different properties. Require complete results, explicit acceptance rules, and named owners. A release decision belongs to the exact tested candidate; absent checks and mandatory policy failures block promotion even when average quality improves.
Knowledge check
A release job runs an agent’s integration suite with pytest --run_slow --run_integration and exits successfully. The summary shows 24 required integration cases collected, 16 passed and 8 skipped because a service API key was not bound in the job. What should the release owner do?
--fail_missing--run_integration alone so slow cases cannot interfere--fail_missing marks them failed so a required job cannot pass quietly. Removing the cases changes the approval rule instead of repairing the credential binding, and dropping --run_slow skips slow tests rather than the missing-key ones.A candidate claims assistant scores 94% task success in nat eval, against 91% for the baseline, and passes every rail-safety interaction. One mandatory cross-claimant privacy case in the full-workflow suite returns another claimant’s record. What should the release decision be?
An HR agent must never return another employee’s salary record, which it can reach through a payroll tool. The team’s privacy check today is a NeMo Guardrails policy evaluation run with nemoguardrails eval run and nemoguardrails eval check-compliance. Which addition makes the check cover this risk?
expected_output entries about salariesnat eval cases that call the payroll toolnemoguardrails eval check-compliancenat eval cases inspected per case. More interactions or a stronger judge still test only the rails, and a mocked unit test checks how code handles a response, not what the connected system discloses.The safety suite for a pharmacy assistant holds 150 adversarial prompts that must be refused, and the candidate refuses all of them. In a pilot, pharmacists report that it also refuses routine dosage lookups. Which change to the safety check targets this failure before release?
A CI job promotes a candidate when the average score in the nat eval console summary clears a threshold. In one run, a configured evaluator crashed and wrote no output, yet nat eval exited with status zero and the average over the remaining evaluators still cleared the threshold. What should the job verify before comparing scores?
workflow_output.json was written for every case in the runconfig_effective.yml matches the committed configurationnat eval command exited with status zero after the runworkflow_output.json records per-case execution, config_effective.yml the configuration after overrides, and an exit status that the command ended; none shows that every required score exists.