NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.5 Analyze evaluation results to guide targeted optimization.

Find bottlenecks in step token and latency profiles

Completed by the knowledge check

Learning objective

Use per-step token and latency profiles to locate bottlenecks.

Core idea

Token totals identify where inference work accumulates. A timeline identifies which work delays the result. These are related measurements, but they answer different questions: a long parallel tool call can dominate delivery while a token-heavy model call finishes early.

Start with a failed delivery constraint, such as excessive completion time or token consumption, and inspect the steps responsible. See 3.2.3 for comparing whole-workflow quality, cost, and latency. Here you explain the measured result inside one workflow before selecting an optimization.

How it works

Collect profiles with enough coverage

NeMo Agent Toolkit provides the profiler through nvidia-nat-profiler; the documented package installation extra is nvidia-nat[profiler]. It runs through nat eval and reads settings under eval.general.profiler. Enable compute_llm_metrics: true to compute inference optimization metrics, and use the documented bottleneck_analysis.enable_nested_stack: true configuration to inspect nested activity rather than flattening every tool into an unrelated operation.

Profiler instrumentation records per-invocation usage through callbacks. The documentation lists LangChain/LangGraph, LlamaIndex, CrewAI, Google ADK, and Semantic Kernel callback support. For a registered function, specify the relevant framework_wrappers in @register_function; NVIDIA recommends explicit selection even though the toolkit can infer frameworks. For a custom inner function that needs its own timing boundary, the documented @track_function decorator records starts, ends, arguments, and output. Confirm that the operations you intend to compare actually appear in the trace.

Use four artifacts together. Start with inference_optimization.json, which summarizes workflow metrics with 90%, 95%, and 99% confidence intervals for latency, throughput, and workflow runtime. standardized_data_all.csv holds standardized usage data, including prompt and completion tokens and framework metadata. all_requests_profiler_traces.json contains raw LLM and tool traces, runtimes, and other metadata. workflow_profiling_report.txt describes bottlenecks and latency at different concurrencies. The nested-stack analysis also produces Gantt charts for examining where time is spent.

Enable eval.general.profiler.prompt_caching_prefixes.enable: true to identify repeated prompt prefixes that could be pre-populated in KV caches. Treat the identified prefixes as caching candidates to test in the serving configuration. A table is easier to aggregate; the raw trace preserves execution context. If an unexpected gap appears between operations, investigate it rather than assigning that unobserved time to the nearest model call. Missing inner instrumentation can make a parent look expensive without revealing its cause.

Separate token volume from repeated work

NVIDIA’s analysis filters the CSV to event_type == LLM_END before aggregating prompt_tokens and completion_tokens. Group by example_number and llm_name for per-case totals; retain function_name when locating the responsible function. Avoid summing start and end records as though both report independent usage.

Examine both the size and count of calls. Ten modest requests can consume more tokens than one large request. Repeated prompts, accumulated tool observations, and repair loops can make a workflow expensive even when no individual response is unusually long. Compare the invocation sequence in the raw trace before concluding that one verbose answer is the only problem.

Treat unreported usage as unknown until you verify instrumentation. A missing token value is not a measured zero. Likewise, token count is a workload measure, not a monetary bill: translating it to money requires the applicable serving arrangement or pricing. See 0.2 for the distinction between prompt processing and generation; do not infer exact duration from a token count alone.

Reconstruct elapsed time without double-counting

The profiler CSV exposes event_type and event_timestamp; its examples use LLM_START and LLM_END to inspect elapsed LLM activity. Match the corresponding invocation in the trace before subtracting timestamps. Grouping all starts and ends by function name alone merges repeated calls and hides retry or loop behavior.

Distinguish three measurements. A call’s elapsed duration covers its start-to-end interval. A parent interval can include child intervals, so adding parent and child durations counts overlapping time twice. The final-answer path depends on the ordering and joins of the workflow; optimizing a parallel branch that already finishes early might save tokens without shortening delivery.

NVIDIA’s example takes the first LLM start and last LLM end per case and model. That interval is useful for comparing the envelope of LLM activity, but you should not substitute it for complete request latency: it excludes work before the first model call or after the last. Retain the workflow boundaries when the constraint concerns the user’s full wait.

Key distinctions and trade-offs

Use the profile pattern to choose the next investigation:

Observed pattern	Deciding measurement	Mechanism or artifact	Optimization hypothesis to test
Large inputs on one function	`prompt_tokens` by `function_name`	`standardized_data_all.csv`, recorded model input, and `prompt_caching_prefixes` analysis	Distinguish reusable prefixes to cache from irrelevant context to remove; preserve answer quality
Repeated generation	Call count and `completion_tokens` per case	Raw invocation sequence	Repair the loop or failed intermediate result causing extra calls
A long tool shadows a model call	Overlapping start/end intervals	Nested-stack Gantt chart	Repair the tool path before buying faster model inference
A parent looks slower than its visible children	Unexplained time within its interval	Raw traces and `@track_function` instrumentation	Add the missing boundary before assigning a remedy
Slow cases coincide with more active functions	Tail cases and overlapping activity	`concurrency_spike_analysis`	Test resource contention with controlled evaluation concurrency

The profiler config supports concurrency_spike_analysis.enable and spike_threshold; NVIDIA’s example marks activity at or above seven concurrent functions when that threshold is seven. This is a configured detection threshold, not a supported-concurrency limit. The docs describe these analysis outputs as advanced or experimental techniques. Use the timeline to investigate them rather than treating a flagged spike as proof of overload.

For a controlled comparison, keep eval.general.max_concurrency explicit. Reducing evaluation concurrency can help test contention, but a faster low-concurrency run does not establish performance at the required workload. Production load behavior belongs to 4.3.3; serving changes belong to 7.3.3.

Architecture / decision scenario

A manufacturer’s incident-summary assistant has a 7-second completion target and a fixed on-premises inference allocation. In a representative trace, planning occupies seconds 0–1. A diagnostics tool then runs from 1–5 while a classification model runs from 1–3. Final synthesis waits for both and runs from 5–8.

The classification call consumes 7,000 prompt tokens and 400 completion tokens, while synthesis consumes 2,000 prompt tokens and 600 completion tokens. The team proposes optimizing classification because it has the largest input. Halving its 2-second duration makes it finish at second 2, but the diagnostics tool still finishes at second 5 and the final answer still arrives at second 8.

The performance owner uses the nested trace to choose a diagnostics-path experiment instead. If that path falls from 4 seconds to 2 seconds, both parallel branches finish at second 3 and synthesis ends at second 6. This meets the target if the tool results and answer quality remain acceptable. The classification input remains a separate token-efficiency candidate; it is not the first latency fix. The owner verifies the same ordering and quality across the slow-case slice before adopting either change.

Common exam traps

Ranking latency fixes by token totals alone.
Adding parent and child intervals as independent elapsed time.
Merging repeated invocations because they share a function name.
Treating missing token telemetry as zero usage.
Reading an LLM-activity envelope as complete request latency.
Treating a configured concurrency-spike threshold as a capacity guarantee.

Where the exam guide differs

The study guide’s readings use “NVIDIA Agent Intelligence Toolkit” and “AIQ Toolkit.” The profiling documentation uses NeMo Agent Toolkit, nat eval, and eval.general.profiler. Interpret the guide’s name as the workflow-profiling role, and use the documented artifacts to distinguish token consumption from elapsed time.

What to remember

Count tokens at completed model calls, preserve invocation identity, and inspect overlapping execution before choosing a latency fix. A useful profile identifies both where work accumulates and which dependency delays the final result. Confirm the improvement with the same quality and workload constraints.