NVIDIANVIDIA-Certified Professional: Agentic AI2026-107.3 Optimize workflows with the NVIDIA NeMo Agent Toolkit.
Feed profiler predictions into served workflows
Completed by the knowledge check
Learning objective
Profile and serve toolkit workflows to cut token and latency cost.
Core idea
A profiler identifies where a workflow spends resources. Reducing those resources requires a change to the workflow or its configured components, followed by measurement through the interface clients use. Starting a server does not make the workflow cheaper, and collecting a profile does not apply an optimization.
See 3.5.2 for interpreting per-step tokens and timelines. This lesson owns the delivery decision: which profiler finding becomes a configuration or implementation change, what quality constraint it must preserve, and how the measured candidate becomes the workflow served to clients.
How it works
Reuse the measured profile
See 3.5.2 for profiler installation, nat eval, analysis settings, and output artifacts. Begin with that measured profile and a specific proposed change; bottleneck and concurrency analyses belong to the profiler documentation’s advanced or experimental techniques, so confirm a suspected delay against the underlying traces.
Translate an output artifact into an action
Use the profile to choose between reducing the work itself and changing how the inference backend schedules it. The former changes the registered workflow or its component settings; the latter can consume the profiler’s prediction trie at serving time.
The following decisions connect analysis to implementation:
| Finding | Candidate change | Toolkit mechanism | Trade-off to verify |
|---|---|---|---|
| Repeated prompt prefixes | Arrange stable reusable prompt content and assess backend cache reuse | prompt_caching_prefixes.enable identifies candidates | A reported prefix does not establish a backend cache hit |
| Excess generation in one model call | Reduce that client’s output allowance | llms instance max_tokens | Truncated or incomplete results can erase the savings |
| Unnecessary repeated model work | Remove or combine the redundant operation | Edit the registered workflow or function implementation | Fewer calls can reduce verification or recovery quality |
| Delayed nested dependency | Repair that dependency before changing the model | bottleneck_analysis.enable_nested_stack attributes nesting | Faster generation might leave total duration unchanged |
Token counts are usage measurements, not currency amounts. Apply the deployment’s actual pricing or resource accounting to the measured calls before declaring a monetary saving. With a fixed on-premises GPU allocation, fewer tokens can release capacity without changing the monthly hardware bill. Compare useful completed work within the latency constraint as well as tokens per request; see 3.2.3 for quality-versus-cost selection.
Carry predictions into Dynamo requests
The prediction trie connects offline profiling to runtime routing. Under eval.general.profiler, set prediction_trie.enable: true; after nat eval, the profiler writes prediction_trie.json in the configured output directory. NVIDIA labels the Dynamo integration experimental and requires Dynamo 1.1.0 or later.
The trie identifies each LLM call position by its function path and call index. It records expected remaining calls, interarrival time in milliseconds, and output-token p50, p90, and p95. With auto_sensitivity enabled, which is the default, it also estimates latency sensitivity for each position. A planning call that delays several dependent calls can therefore carry different hints from a leaf call.
The documented sensitivity weights are w_critical: 0.5 for critical-path contribution, w_fanout: 0.3 for downstream fan-out, and w_position: 0.2 for user-facing position. w_parallel: 0.0 disables the optional penalty for calls hidden behind longer parallel siblings. A positive w_parallel reduces sensitivity for those calls, which can tolerate more delay without changing the parent’s completion time.
At serving time, configure the selected llms entry with _type: dynamo and prediction_trie_path pointing to the generated file. The Dynamo transport looks up the active call position, replaces static remaining-call, interarrival-time, and output-token hints with the predictions, and injects the hints into nvext.agent_hints in the request body. The backend can use that information for KV-cache management and request scheduling.
A manual @latency_sensitive(n) annotation takes precedence over the trie’s automatic sensitivity. Use it when domain knowledge changes a call’s importance, and inspect existing annotations before attributing served behavior to an automatic score. Enabling the trie does not override a deliberately assigned manual priority.
This path changes inference scheduling rather than removing model work. Workflow edits in the table can reduce tokens on other backends; trie-based routing requires the compatible Dynamo deployment and its generated artifact. After changing workflow structure, re-profile the candidate before deploying its hints: the file describes call positions observed in that workflow, not an arbitrary replacement graph.
The prediction trie connects offline profiling to the hints in the Dynamo request body, and a change to workflow structure sends the candidate back to profiling.
Serve the selected configuration
Start the toolkit API server with nat serve --config_file EVAL_CONFIG_PATH, using the same configuration file evaluated by nat eval --config_file EVAL_CONFIG_PATH. Replace EVAL_CONFIG_PATH with that file’s path. If deployment requires a changed value, nat serve --override KEY_PATH VALUE uses a dot-separated key path; record and evaluate those changes before claiming the served candidate matches the measured one. The server’s Generate interface uses the workflow’s transaction schema; the running server exposes interface documentation at http://localhost:8000/docs in the documented example.
The endpoint reference lists /v1/workflow for a complete non-streaming result and /v1/workflow/stream for streaming. Their legacy paths are /generate and /generate/stream. NVIDIA marks the versioned paths experimental because of HTTP human-in-the-loop and OAuth support, and documents legacy paths as registered by default unless explicitly disabled. Select and verify the path your client actually uses rather than assuming a route from another deployment.
A non-streaming transaction waits until workflow data is available before returning the result. Streaming can expose partial progress, but it does not demonstrate reduced total tokens or completion latency. Keep time to initial output and time to accepted final output separate when judging an interface change. The serving owner must verify that the chosen workflow supports the response mode the client expects.
Preserve observability without widening data exposure
The profiler stores inputs and outputs in its raw traces. csv_exclude_io_text: true avoids placing large input and output text in the CSV; it is not a declaration that the other artifacts contain no sensitive data. Treat raw trace access and retention as part of the measurement design, especially for employee or customer content.
Exercise representative requests through the selected served route. Record the evaluated configuration, any deployment overrides, and the prediction-trie artifact together; see 3.1.4 for repeatable evaluation artifacts.
Key distinctions and trade-offs
The agent owner changes workflow execution and generation settings. The platform owner deploys the selected configuration and prediction artifact; the data owner approves the measurement dataset and trace access.
A prompt-prefix report identifies potential reuse, a prediction trie supplies runtime routing hints, and a streaming route changes result delivery. None establishes a lower cost per acceptable answer without a measured before-and-after comparison.
Architecture / decision scenario
A retail merchandising assistant must deliver accepted product summaries within eight seconds at its planned load. In a hypothetical evaluation of 400 requests, the baseline consumes 6,000 generated tokens per request and takes 11 seconds at the 95th percentile. The quality floor is 95% of summaries passing the team’s product-fact checks.
The approved environment serves NIM endpoints and has no Dynamo deployment; introducing an experimental backend requires a separate platform review. One proposal generates a prediction trie and expects the existing backend to consume its hints. The platform owner rejects it because _type: dynamo and a compatible Dynamo service are prerequisites. Another proposal enables streaming and calls the deadline problem solved. The platform owner rejects it because the deadline concerns the accepted complete summary. Nested profiles instead expose a duplicate summarization call. Removing that call produces 3,800 generated tokens per request, a seven-second 95th percentile, and 96% passing summaries on the same evaluation set.
The agent owner selects this candidate over a lower output allowance that reaches six seconds but only 91% passing summaries. The serving owner loads its configuration with nat serve, verifies the chosen API route, and measures complete responses at planned load. If serving exceeds eight seconds, the team investigates that discrepancy rather than citing the offline result as deployment success. The figures are scenario measurements, not toolkit performance guarantees.
Common exam traps
- Assuming profiler configuration applies an optimization automatically.
- Generating
prediction_trie.jsonwithout configuring a Dynamo client to load it. - Expecting automatic sensitivity to override
@latency_sensitive(n). - Calling partial output a completed workflow or a token saving.
- Assuming
csv_exclude_io_textremoves sensitive text from every artifact. - Presenting experimental versioned routes or Dynamo integration as universal deployment behavior.
What to remember
Reduce workflow work or feed prediction_trie.json into a compatible Dynamo client through prediction_trie_path; the latter injects nvext.agent_hints and remains experimental. Preserve task quality and serve the evaluated configuration through a verified interface. Token savings, final-result latency, partial-result delivery, and data exposure require separate checks; a favorable result in one does not establish the others.
Knowledge check
A NeMo Agent Toolkit workflow must cut generated tokens per request in this release. The approved environment serves NIM endpoints only; adding a Dynamo deployment needs a separate platform review that will not happen this release. Nested profiles show that the workflow calls the same summarization step twice with identical input. Which change should the team make?
prediction_trie.json and set prediction_trie_path on the NIM clientprediction_trie.enable so the profiler applies its predictions/v1/workflow/stream route to deliver output earlierllms entry of _type: dynamo and a compatible Dynamo service, which this release does not have, and streaming changes delivery rather than the tokens generated.A workflow served through the experimental Dynamo integration loads a prediction trie generated with auto_sensitivity enabled and the sensitivity weights left at their documented defaults. For one planning function, the trie’s automatic sensitivity score is 4, yet its requests carry a sensitivity of 1. What is the most likely cause?
@latency_sensitive(1) annotation that takes precedencew_parallel weight lowers the score of calls with parallel siblings@latency_sensitive(n) annotation always overrides the trie’s automatic score, so a @latency_sensitive(1) annotation explains the fixed value of 1. The trie scores every LLM call position, planning calls included; w_parallel defaults to 0.0, which disables the parallel-sibling penalty; and the transport replaces static hints with the trie’s predictions.A merchandising assistant must return an accepted, complete product summary within 8 seconds at the 95th percentile; it currently takes 11 seconds. A proposal moves clients from /v1/workflow to /v1/workflow/stream, and in testing the first partial output arrives after 2 seconds. How should the serving owner assess the proposal?
A data owner approves profiling over customer support conversations only if raw customer text stays under the restricted trace policy. An engineer sets csv_exclude_io_text: true in the profiler configuration and proposes sharing the whole profiler output directory with an external analytics team. What should the data owner decide?
csv_exclude_io_text keeps large input and output text out of the CSV so its structure stays readable; it is not a redaction of the other artifacts. The raw trace file records LLM and tool inputs with other metadata, so the output directory stays under the policy that governs raw traces however the CSV was produced.