NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.1 Implement evaluation pipelines and task benchmarks to measure performance.
Preserve inputs and outputs across evaluations
Completed by the knowledge check
Learning objective
Automate repeatable evaluation runs with versioned inputs in an evaluation harness.
Core idea
An evaluation harness turns a dataset, workflow configuration, and evaluator configuration into inspectable results. Repeatability means you can identify and rerun those conditions. It does not mean a stochastic agent produces identical text every time, or that saving the YAML preserves an external search index or remote model deployment.
NeMo Agent Toolkit provides nat eval for executing a workflow against configured data and evaluators. The harness owner must preserve both what the toolkit records and the external inputs that determine the task. Bite 2.1.4 owns versioning prompt artifacts across model changes; 3.1.1–3.1.3 own dataset design, judge calibration, and RAG metric semantics. Here, focus on harness run modes, job isolation and eviction, resumption, remote state, and concurrency.
How it works
Define the run’s inputs explicitly
For the full nat eval runtime, NVIDIA documents installing nvidia-nat[eval]. The standalone nvidia-nat-eval package serves a separate ATIF evaluation mode; do not assume that installing it alone supplies the full config-driven workflow runtime and dataset readers.
Use the workflow and evaluation configuration described in 2.1.4. A harness invocation can override its dataset through --dataset, so record the resolved input rather than relying on the original configured path.
Create a run manifest alongside the results. Record content hashes or immutable revisions for the dataset, source corpus or index snapshot, workflow code, prompt artifacts, judge rubric and examples, and dependency lockfile. Record the actual model deployment revision when your serving system exposes one. These are team-owned reproducibility records, not extra keys to insert into the toolkit configuration.
A path or a model alias can stay constant while its contents change. If you cannot restore a dependency, mark that limitation in the run record. Repeating the same command against changed external state is another experiment, even when the toolkit successfully executes it.
Retain the effective configuration and results
For an overridden or resumed run, retain config_effective.yml with the run manifest and compare it with config_original.yml before combining results. The three configuration artifacts introduced in 2.1.4 identify the actual invocation; the harness must associate them with the correct job.
workflow_output.json contains per-case outputs, including the expected answer, generated_answer, and intermediate_steps. Each configured evaluator writes <evaluator-name>_output.json. A workflow output file proves generation occurred; evaluator output files establish which scoring components ran. A workflow configuration with an eval section but no evaluators can generate dataset outputs without scoring them.
Prevent one invocation from replacing another’s artifacts. Set eval.general.output.dir to the result location and enable eval.general.output.job_management.append_job_id_to_output_dir: true to give each run its own job subdirectory. Use eval.general.output.cleanup: false when retaining those runs. Cleanup is disabled by default; explicitly setting it to true can delete the output directory and its job subdirectories.
Within eval.general.output.job_management, max_jobs limits retained job directories; its default of 0 imposes no limit. eviction_policy selects TIME_CREATED (the default) or TIME_MODIFIED to identify the oldest jobs for eviction. Keep a baseline that must survive outside the evicting directory, or choose a limit that preserves it for the required comparison window.
With the LangSmith telemetry exporter configured, NVIDIA’s built-in evaluation callback records structured experiments. Associate that experiment and the job identifier with your version record. A unique output directory prevents collisions, but it does not identify the dataset contents or the model that served the requests. Both artifact separation and input identity are necessary.
Select execution, resumption, or rescoring
A fresh evaluation executes the workflow and configured evaluators. --reps repeats the evaluation so you can examine variation; it increases work rather than freezing randomness. For 300 cases and three repetitions, plan for 900 workflow executions before accounting for judge calls or retries. That arithmetic is an experiment budget, not a toolkit throughput guarantee.
--skip_workflow evaluates previously generated answers. Supply a dataset containing the expected answer and the saved generated_answer. Use this mode when testing a scoring change on fixed outputs; it cannot demonstrate that a changed workflow generates better answers because that workflow did not run.
--skip_completed_entries instead skips entries that already have a generated answer. NVIDIA recommends resuming interrupted large runs by copying saved workflow output and passing the copy through --dataset. Use the copy outside the active output location so that preparing another run does not overwrite the input you intend to resume.
A fresh evaluation executes the workflow and configured evaluators, --skip_workflow evaluates the saved generated_answer without running the workflow, and --skip_completed_entries resumes from a copy of saved workflow output passed through --dataset.
Resume only against the same intended workflow and input versions. If some completed entries came from one prompt and unfinished entries execute another, the combined file is a mixed experiment. Either preserve the original conditions or create a fresh run whose identity describes the change.
Bound concurrency and identify remote state
eval.general.max_concurrency controls parallel processing of dataset entries; NVIDIA documents a default of eight. Choose a value that the workflow services and judges can sustain. Higher concurrency can shorten batch elapsed time, but it can also produce capacity errors and different queueing behavior. Record the value so that a run under increased contention does not appear identical to a quieter run.
For a remote workflow, --endpoint selects the serving endpoint. NVIDIA explicitly notes that saved evaluation configuration does not capture the workflow configuration running on that remote server. Preserve the deployed workflow revision separately. An endpoint URL and timestamp are insufficient when the service changes between runs.
Key distinctions and trade-offs
Choose the mode according to what you want to rerun:
| Mode | Deciding condition | Mechanism | What it cannot establish |
|---|---|---|---|
| Fresh generation and scoring | Workflow behavior must be measured | nat eval --config_file=EVAL_CONFIG_PATH (replace the placeholder with the config path) | Identical output text across stochastic executions |
| Repeated evaluation | You need repeated observations under the same recorded setup | --reps | A fixed random sequence or independent external state |
| Rescore saved answers | Judge behavior changes while outputs stay fixed | --skip_workflow, --dataset | Improvement in workflow generation |
| Resume incomplete generation | An interrupted run has reusable completed entries | --skip_completed_entries, saved outputs | A coherent run if versions changed halfway through |
The automation runner should verify expected case counts and the presence of required evaluator artifacts before accepting a run as complete. Use the case identifiers and per-case results, not only a console average. Detailed comparisons between systems and statistical interpretation belong to 3.2.
Architecture / decision scenario
A retailer runs a product-research agent against 300 benchmark questions each night. The evaluation budget allows 900 workflow executions, and the team chooses three repetitions with concurrency four because the shared product API cannot sustain the concurrency used by other jobs. These are deployment choices rather than NVIDIA defaults.
A judge-rubric fix arrives after the night’s workflow outputs have been saved. The release engineer proposes rerunning generation immediately, but that would consume another 900 executions and change the answers being judged. The evaluation owner instead uses --skip_workflow on the saved answers, preserving the original job and creating a separate result directory for the revised judge. This isolates the scoring change within budget.
Later, a generation run stops with 70 cases unfinished. It can resume with --skip_completed_entries only while retaining the recorded workflow and dataset revisions. If a prompt changed after the interruption, the owner schedules a fresh run rather than combining two behaviors. For a remote endpoint, the deployment owner supplies the server’s workflow revision; the local config_effective.yml alone cannot restore it.
Common exam traps
- Equating the original YAML with the effective overridden configuration.
- Assuming
--skip_workflowtests a changed workflow. - Resuming saved answers across incompatible workflow revisions.
- Setting
cleanup: trueon a harness that relies on retained job subdirectories. - Treating an endpoint URL as a snapshot of remote workflow configuration.
Where the exam guide differs
The guide’s suggested readings say “NVIDIA Agent Intelligence Toolkit.” NVIDIA documents the harness as NeMo Agent Toolkit and invokes it through nat eval. Recognize the earlier name in the guide, then answer implementation questions using the documented configuration, run modes, and output artifacts.
What to remember
Automate the harness around identified inputs, effective configuration, per-case results, and retained job artifacts. Fresh execution, rescoring, and resumption answer different questions. Reproducibility also requires the remote deployment and external data revisions that a local evaluation configuration cannot preserve.
Knowledge check
Last night’s NeMo Agent Toolkit run saved its workflow outputs; the original benchmark contains no generated answers. This morning the team revised only the judge rubric and wants to know how the revision changes scores on those same answers, without any new workflow executions. Which invocation fits?
nat eval with --skip_workflow and the saved output as --datasetnat eval with --skip_workflow and the original benchmark as --datasetnat eval with --reps=3 and the revised evaluator configurationnat eval as a fresh run with the revised evaluator configuration--skip_workflow evaluates a dataset that already holds the expected answer and the saved generated_answer , so the scoring change is isolated on fixed outputs; pointed at the original benchmark, it has no generated_answer to score. A fresh or repeated run regenerates the answers, adding executions and a second changed variable.A 300-case generation run stopped with 70 cases unfinished. Since then the team has revised the prompt artifact, and the result must describe the revised prompt alone. What should the team run?
--skip_completed_entries under the revised prompt--skip_workflowA nightly harness writes to eval.general.output.dir with append_job_id_to_output_dir: true and max_jobs: 7, because the disk budget allows seven nightly jobs and one retained baseline. The release gate needs an older baseline job that is still present but will be evicted as nightly jobs accumulate. Which change preserves it within the disk budget?
eviction_policy to TIME_MODIFIED for the harnessmax_jobs to 0 for the nightly harnessappend_job_id_to_output_dirmax_jobs evicts the oldest job directories, so a baseline that must outlive the window belongs outside the evicting directory. TIME_MODIFIED only changes how “oldest” is judged, a max_jobs of 0 removes the limit the disk budget requires, and without job IDs every run overwrites the previous results in the same directory.A team evaluates a staging deployment with nat eval --endpoint, keeping config_effective.yml and config_metadata.json for every run. A month later it must recreate one run’s conditions. What else must the team have preserved?
config_original.yml written beside the effective configeval.general.max_concurrency value used by the run--endpoint the saved configuration captures the evaluation settings but not the workflow configuration running on the remote server, so its deployed revision must be recorded separately. The URL, timestamp and concurrency are already in the saved artifacts, the original config shows only the settings before overrides, and none of them says what the server was running.