NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.3 Profile performance and reliability under distributed system loads.
Trace the path that limits loaded workflows
Completed by the knowledge check
Learning objective
Trace critical paths, fan-out bottlenecks, and failure behavior under load.
Core idea
The operation consuming the most cumulative time is not necessarily the operation delaying the response. A workflow can run many calls concurrently, wait for one slow branch, then generate an answer. Increasing load can change which branch determines completion and where failures spread.
Use a distributed trace to reconstruct the causal path for an individual request, then compare many such paths across load levels. See 3.5.1 for per-case failure attribution and 3.5.2 for step token and latency profiles. This lesson adds contention, fan-out, and failure propagation: what changes when many workflows share the same dependencies.
How it works
Preserve causality across service boundaries
OpenTelemetry spans carry a trace identifier, a span identifier, parent relationships, timestamps, attributes, and events. Root and child intervals describe the complete operation and its constituent work. For HTTP boundaries, context propagation injects and extracts the W3C traceparent header so downstream work remains connected to its caller.
Configure NeMo Agent Toolkit tracing under general.telemetry.tracing using an exporter with _type: otelcollector and its endpoint. The OpenTelemetry integration requires the nvidia-nat[opentelemetry] extra. Instrument downstream services too: exporting local events does not reveal remote execution.
For NIM’s vLLM backend, send traceparent from the caller and set NIM_PASSTHROUGH_ARGS="--otlp-traces-endpoint COLLECTOR_URL", replacing COLLECTOR_URL with the collector’s tracing endpoint. Forwarding the header alone does not produce inference spans. This environment setting replaces container defaults, so include any arguments the model container already sets. Supply X-Request-Id to correlate backend logs with the request. NIM forwards tracing headers only on inference endpoints; management endpoints such as /v1/metrics and /v1/models, and health endpoints, do not propagate them.
Verify the trace shape at low load before trusting the stressed run. A downstream root with no caller relationship can mean missing propagation. Unexplained parent time can mean queueing or an uninstrumented adapter. Add explicit spans around the adapter’s wait-for-capacity interval and actual remote call, with names you define in the application, so those intervals are distinguishable. Investigate missing instrumentation before diagnosing a blank interval.
The profiler package provides @track_function for custom Python functions outside the toolkit’s built-in functions. It records function start, output chunks for generators, and completion through intermediate-step events. Control serialized inputs and outputs to limit trace volume and sensitive content while retaining diagnostic identifiers and timing.
Follow the dependency chain to completion
For sequential work, the critical path includes each required operation in order. For parallel work followed by a join that requires every branch, the parent waits for the slowest branch. Summing overlapping child durations overstates elapsed time; subtracting all child durations from the parent can even produce nonsense.
Suppose planning takes 200 ms, three simultaneous tool calls take 300, 500, and 1,800 ms, and answer generation takes 800 ms. Ignoring other overhead, completion takes 2,800 ms: 200 + 1,800 + 800. Reducing the 300 ms tool call to 100 ms does not shorten that path. Reducing the slowest branch can help until another branch becomes the longest.
The join waits for the slowest branch, so the critical path runs through the 1,800 ms tool call.
The profiler supplied by nvidia-nat-profiler, installed through nvidia-nat[profiler], runs with nat eval. Recall the controls from 3.5.2: eval.general.profiler.compute_llm_metrics, bottleneck_analysis.enable_nested_stack, and the resulting standardized_data_all.csv. Nested-stack analysis exposes tool calls inside other calls; its Gantt charts help you inspect overlap rather than rank cumulative duration alone.
The profiling page groups concurrency and bottleneck outputs with advanced or experimental techniques. Treat those analyses as diagnostic aids and confirm conclusions against trace intervals and live-load measurements. Confirm production capacity separately from inferred priorities.
Find the fan-out that amplifies load
Count actual child operations per workflow, including retries and repeated agent turns. At 15 workflows/second, four calls to one service per workflow imply 60 calls/second before retries. An average across all tools hides concentration on a single database or retrieval endpoint.
Compare fan-out counts, wait intervals, and the longest required branch at successive load levels from 4.3.2. If only one dependency’s wait-for-capacity span expands while its execution time stays stable, its admission capacity is a stronger suspect than model generation. If the remote execution interval itself grows, investigate that service with its owner. Trace both sides before assigning the delay.
NeMo Agent Toolkit exposes eval.general.profiler.concurrency_spike_analysis.enable and spike_threshold to surface overlapping function activity. This identifies concurrency spikes within the profiled workflow data. Supplement it with service-wide traces and request counts: separate application replicas can contend on the same dependency without one local workflow showing the entire demand.
Observe failure propagation under sustained traffic
In an isolated test environment, repeat the workload with a controlled dependency delay, explicit error response, and caller cancellation. Define expected outcomes before injection: which branches must stop, which partial results remain usable, and what terminal response the client receives. Trace successful, slow, and failed requests from each test, rather than inspecting only the successful percentile population.
OpenTelemetry span status distinguishes Unset, Error, and Ok; timestamped span events provide additional detail. Record a failing dependency with its error status and capture separate attempt spans for retries. Preserve the relationship to the parent even when recovery returns a usable response. Parent success must not erase the failed work and extra dependency traffic.
Inspect whether retry attempts raise the downstream call rate, whether sibling spans continue after caller cancellation, and whether a fallback adds a serial delay after the primary path. These observations tell you which existing control to repair: retry ownership in 2.4.2, circuit-breaker admission in 2.4.3, or cancellation and isolation in 1.8.2. Reapply the same injected fault at the same offered load after the repair.
Key distinctions and trade-offs
The trace patterns lead to different decisions:
| Observation | Interpretation to test | Mechanism to inspect | Trade-off |
|---|---|---|---|
| Longest required branch dominates | Join waits for one dependency | Parent-child timestamps and nested-stack Gantt chart | Faster noncritical branches may save resources without reducing latency |
| Child counts grow with traffic | Fan-out multiplies dependency demand | Attempt spans and service request counts | More parallel work can reduce one request’s latency while overloading shared services |
| Parent returns but children continue | Cancellation or cleanup is incomplete | Child end times relative to caller cancellation | Early return may hide continuing resource consumption |
| Successful parent contains errors | Recovery masks extra work | Child Error status and retry spans | Availability can improve while latency and load worsen |
Architecture / decision scenario
A legal research assistant searches three independent repositories before drafting a response. The product requires p95 completion below 5 seconds at 15 research requests/second. The workflow owner compares traces from that load with a 3/second baseline; all three search branches are required for the promised coverage.
In illustrative loaded traces, planning takes 200 ms, the searches take 400 ms, 600 ms, and 4 seconds, and drafting takes 1 second. The required path is 5.2 seconds. The third repository receives 15 calls/second before recovery, and its adapter’s capacity-wait span expands. The tempting proposal is a faster generator that saves 200 ms; it reaches the limit with no allowance and leaves the overloaded dependency unchanged.
A controlled error on the third repository also reveals three attempt spans per workflow and siblings continuing after client cancellation. During that failure test, repository demand can reach 45 attempts/second. The service owner and workflow owner therefore address dependency admission, bounded retry ownership, and cancellation before reconsidering generation speed.
They rerun both normal and injected-failure traffic, checking the root latency, attempt counts, child end times, and explicit incomplete outcomes. Dropping the third repository would change required coverage, so it needs a product decision rather than being counted as a performance improvement. The passing result must preserve the task as well as its deadline.
Common exam traps
- Adding overlapping child durations to estimate complete-request latency.
- Optimizing the largest token consumer without checking the required completion path.
- Assuming an exporter automatically instruments every downstream service.
- Counting only workflows when one dependency receives several calls per workflow.
- Treating a recovered parent as proof that no dependency failed.
- Assuming caller timeout means all child work stopped.
What to remember
Preserve trace context, inspect required branches and overlap, and compare the path across load levels. Count fan-out and attempts at the shared dependency. Verify recovery and cancellation through child outcomes and end times, then rerun the fault at the original offered load.
Knowledge check
A due-diligence workflow plans for 150 ms, then calls three tools in parallel taking 250 ms, 700 ms, and 2,100 ms, and generates the answer in 900 ms once all three return. Ignoring other overhead, which single change most reduces completion time?
A workflow calls NIM for LLMs with the vLLM backend and sends a W3C traceparent header on each inference request. The workflow exporter samples every request, but the NIM backend has no tracing export destination configured. The collector shows the workflow’s spans but no inference spans from NIM. What should the platform engineer change?
traceparent on the /v1/metrics and /v1/models requests tooX-Request-Id header to every inference request sent to NIMNIM_PASSTHROUGH_ARGS with --otlp-traces-endpoint for the collector--otlp-traces-endpoint passed through NIM_PASSTHROUGH_ARGS , together with any arguments the container already sets, since the variable replaces its defaults. X-Request-Id correlates backend logs, not spans, and management endpoints do not propagate tracing headers.A legal research workflow is traced at 3 and at 15 workflows/second, with every span connected under its workflow’s trace. At both levels the remote search call itself takes about 600 ms. At 15/second, the adapter’s wait-for-capacity span before that call grows from 20 ms to 3 seconds. Where should the investigation focus first?
During a fault-injection test, a retrieval dependency returns errors and the workflow recovers through retries. Every root span ends with status Ok, and p95 completion time stays within target. What should the engineer inspect before concluding that the fault had no operational cost?
Error status and the downstream call rategeneral.telemetry.tracingError status show how many attempts failed and how much extra traffic recovery generated. Root status and successful-request latency report only the outcome the caller saw.Six application replicas share one vector database. NeMo Agent Toolkit’s concurrency spike analysis, run on one replica’s profiled workflows, reports no spike, yet the database saturates at peak. Which evidence locates the demand on the database?
spike_threshold on the same replica’s profileReferences
- NVIDIA, Observe workflows
- NVIDIA, Logging and observability
- NVIDIA, Profiling and performance monitoring of NVIDIA NeMo Agent Toolkit workflows
- OpenTelemetry, Traces
- OpenTelemetry, Context propagation