NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.2 Compare agent performance across tasks and datasets.
Choose agents within quality and delivery budgets
Completed by the knowledge check
Learning objective
Compare models and configurations on quality versus cost and latency.
Core idea
The most accurate agent is not automatically the deployable agent. A candidate must satisfy the task’s minimum quality, response-time objective, and operating budget under the intended workload. Compare those dimensions together before choosing among feasible candidates.
Use the controls from 3.2.1 and the slice and repetition analysis from 3.2.2. This lesson turns their results into a deployment decision. Parameter tuning and automated search belong to 3.4; step-level bottleneck diagnosis belongs to 3.5.2. Here, you compare complete candidate configurations and make the quality–cost–latency trade-off explicit.
How it works
Define the unit of useful work
Choose a completed business task as the comparison unit: an acceptable report, a correctly resolved request, or an authorized workflow completion. Record the number of submitted tasks, accepted results, execution errors, and timeouts. A short, inexpensive answer that fails the task is not a cheap success.
Keep quality metrics from eval.evaluators alongside generated answers in workflow_output.json. Treat missing entries or unusable generated_answer values as unsuccessful executions for this comparison. With the profiler enabled, inspect per-request error flags in standardized_data_all.csv; the console summary reports the run-level Workflow Status. A judge score and a binary accepted-result rate answer different questions. If your product requires correct figures and cited sources, use the approved task rubric to determine acceptance rather than assuming that a high mean judge score makes every report usable. See 3.1.1 and 3.1.2 for defining and calibrating those criteria.
Compare model replacement separately from workflow changes when you need attribution. A configuration that uses fewer reasoning steps, fewer retrieved passages, or a different tool path may beat another bundle without proving its model is intrinsically better. Keep the candidate’s model, prompt, tool, and execution revisions together in the comparison record.
Collect quality and runtime measurements together
See 1.5.3 for profiler configuration and framework instrumentation. Install nvidia-nat[profiler] for evaluation and profiling support. The v1.8.0 example confirms eval.general.profiler.compute_llm_metrics: true for inference metrics. Before comparing totals in standardized_data_all.csv, confirm that expected model and tool calls appear: missing instrumentation can look like reduced usage.
NVIDIA’s profiler comparison example filters event_type to LLM_END before aggregating prompt_tokens and completion_tokens by example_number and llm_name. Use those fields to compare the candidate’s total generation workload, preserving run identity when combining files from multiple candidates. Do not add every event row as if each represents another billed generation.
For latency, compare elapsed time from accepting the task to returning its validated result. Keep workflow runtime distinct from one model call’s duration and from time to the first streamed token. Parallel spans overlap, so summing all span durations does not produce user-visible elapsed time. The evaluation console’s Workflow Runtime (p95) is the workflow-level comparison figure; LLM Latency (p95) measures model-call latency. If the product adds queuing or validation outside the workflow, measure that added work in the complete-task interval before comparing with the product objective.
workflow_profiling_metrics.json provides means and percentiles; workflow_profiling_report.txt includes latency at different concurrencies. Use these artifacts to compare tail latency and load conditions, since a low mean can still miss a p95 requirement. Hold eval.general.max_concurrency and the submitted task mixture fixed for the comparison, and test at concurrency representative of the deployment question. A single-request measurement does not establish behavior when requests contend for capacity. If evaluation receives HTTP 429 errors, the documented remedy is reducing concurrency; remeasure all candidates under matched conditions and distinguish that diagnostic run from the required-load comparison.
Translate usage into a cost decision
Token counts measure usage, not money. Apply the approved deployment’s input and output billing rates to the corresponding totals, and include repeated attempts and tool charges in the task’s operating-cost ledger. Track evaluation judge spend separately from the serving workflow so the cost of testing does not become the claimed cost of serving a user.
For a self-hosted deployment, use the actual allocated infrastructure cost over the measured interval divided by completed work. Include the allocation that remains idle while the service waits for traffic. Fewer tokens alone do not establish lower GPU cost if both candidates occupy the same allocation for the same time. The team’s finance and platform owners supply rates and allocation records; the profiler supplies usage measurements, not a universal price.
Calculate both cost per submitted task and cost per acceptable result. The second is total serving expenditure divided by accepted results, including expenditure on failed attempts. If 100 tasks cost a total of $3.50 and 93 are acceptable, the useful-result cost is about $0.0376. This is an illustrative accounting calculation, not a model price. It exposes a candidate that appears cheap only because it gives up early.
Key distinctions and trade-offs
Use explicit requirements before ranking candidates:
| Dimension | Deciding condition | Mechanism or artifact | Cost of optimizing it alone |
|---|---|---|---|
| Quality | Task and required slices meet acceptance thresholds | Per-case evaluator outputs and accepted-result counts | More calls or longer generation may exceed delivery budgets |
| Latency | Complete-task p95 meets the product objective at required load | Workflow timing and matched-concurrency run record | Shortened processing may reduce task success |
| Cost | Serving expense fits the budget per useful result | Token totals plus deployment cost ledger | Cheap failed answers shift work to people or retries |
Reject infeasible candidates first. Among feasible ones, remove a candidate that is no better on quality, latency, or cost and worse on at least one. Remaining choices require a business preference: extra quality can justify additional expense for one workflow and offer no useful benefit for another. See 3.4.3 for automated search and Pareto-front selection; a comparison report still needs the product owner’s acceptance rule.
Only the remaining choices require a business preference: infeasible candidates are rejected first, then a candidate that is no better on quality, latency, or cost and worse on at least one is removed.
Architecture / decision scenario
A retail analytics assistant produces daily pricing explanations. The product owner requires at least 90% acceptable reports, p95 completion within eight seconds, and at most $0.04 per submitted report. Three bundles run on the same 500 cases, repeated five times at the agreed concurrency. The following figures are hypothetical measured summaries, not vendor benchmarks or prices:
| Candidate | Acceptable reports | Complete-task p95 | Cost per submitted report |
|---|---|---|---|
| A: larger-model bundle | 92% | 12 seconds | $0.040 |
| B: smaller-model bundle | 89% | 6 seconds | $0.020 |
| C: revised workflow bundle | 93% | 7 seconds | $0.035 |
A stakeholder proposes B because it halves A’s spend. B misses the quality floor, while A misses the latency requirement. C satisfies all three and costs about $0.0376 per acceptable report. The product owner selects C after checking required slices and repeat-run stability; finance validates the cost ledger, and the platform owner verifies that the serving allocation matches the measured one.
C changes the workflow as well as its model configuration, so the report claims a bundle improvement. It does not attribute the gain solely to model size. If C’s missing calls were an instrumentation defect or its latency came from a lower-concurrency run, the team would repeat the comparison before selection.
Common exam traps
- Selecting the highest quality score before checking latency and cost constraints.
- Equating token count with a deployment’s monetary cost.
- Comparing one model call’s latency with another agent’s full task runtime.
- Summing overlapping spans to calculate elapsed time.
- Including judge calls in serving cost without separating evaluation expenditure.
- Reporting cheap successful runs while excluding spending on failures.
Where the exam guide differs
The study guide lists “NVIDIA Agent Intelligence Toolkit” readings. NeMo Agent Toolkit documents the nat eval profiler used here. Read that product name in an exam item as the toolkit family.
What to remember
Compare quality, complete-task latency, and cost on the same workload. Convert measured usage using the deployment’s actual cost basis and include failed work. Choose a configuration that meets every required floor and budget before deciding whether additional quality earns its extra expense.
Knowledge check
A claims-summary service requires at least 95% acceptable summaries, complete-task p95 within five seconds, and at most $0.10 per submitted request. Three bundles run on the same cases at the agreed concurrency: A scores 97%, 7 seconds and $0.08; B scores 94%, 3 seconds and $0.03; C scores 96%, 4.5 seconds and $0.09. Which should the team select?
The approved cost measure is total serving spend divided by acceptable reports. Bundle X spent $18 on 1,000 submitted reports, of which 600 were acceptable; its dashboard divides the $12 spent on those 600 by 600. Bundle Y spent $20 on 1,000 reports, of which 950 were acceptable. Which statement follows from the approved measure?
A self-hosted agent runs on a dedicated GPU allocation reserved around the clock. Bundle B uses 40% fewer tokens than bundle A, and at the measured traffic both complete the same work while occupying the same allocation for the same time. Finance asks whether B lowers serving cost. What is the correct answer?
A product objective sets complete-task p95 at eight seconds, measured from accepting a task to returning its validated result. The evaluation console reports Workflow Runtime (p95) of 7.7 seconds and LLM Latency (p95) of 1.6 seconds, and the product runs a validation step after the workflow returns. Which figure should be compared with the objective?
Workflow Runtime (p95) of 7.7 secondsLLM Latency (p95) of 1.6 secondsWorkflow Runtime (p95) stops when the workflow returns, LLM Latency (p95) measures single model calls, and summing spans double-counts work that runs in parallel.In standardized_data_all.csv, bundle C shows half the prompt tokens of bundle A. C’s revised tool path calls a model through a client that the team has not confirmed the profiler instruments. What should the team do before crediting the reduction?
LLM_END rowsLLM Latency (p95) with A’s to confirm it