Agent Evaluation in NVIDIA NeMo Agent Toolkit

Why it matters

An evaluation you cannot rerun under the same conditions cannot tell you whether a change helped. This guide shows how the toolkit’s harness records those conditions for you.

What to practise

Install nvidia-nat[eval] and run nat eval --config_file against the simple web query example. Read its eval section: general.dataset with _type: json and file_path, a ragas evaluator with metric: AnswerAccuracy, and the trajectory evaluator that judges the intermediate steps. Then open what the run writes: workflow_output.json with each generated_answer and its intermediate_steps, one <evaluator-name>_output.json per evaluator, and config_original.yml, config_effective.yml and config_metadata.json. Try --reps, --skip_completed_entries and --skip_workflow, and set append_job_id_to_output_dir with cleanup: false so a second run does not overwrite the first.

How it fits

This is the harness behind 3.1.4: versioned inputs, the effective configuration and a separate output per run. The dataset format it reads is where 3.1.1’s benchmark cases end up, and its judge-based evaluators still need 3.1.2’s calibration before you trust them.