NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.2 Track logs, errors, and anomalies for root cause diagnosis.
Isolate causes before changing failing components
Completed by the knowledge check
Learning objective
Perform root cause analysis of tool, retrieval, and inference failures from traces and logs.
Core idea
The component reporting the final error is not necessarily the component that introduced the defect. A tool can reject an argument because retrieval supplied no usable context. An inference timeout can follow repeated tool recovery rather than slow generation. Diagnose the causal chain before changing the component at its end.
Use the correlated execution from 8.2.1 and the affected population from 8.2.2. Unlike the offline evaluation attribution in 3.5.1, production diagnosis must also establish what was running, whether requests reached the dependency, and which intervention is safe during the incident. Google SRE’s troubleshooting method is hypothesis-driven: collect observations, choose a discriminating test, record the result, and revise the hypothesis.
How it works
Reconstruct the failed execution and a healthy comparison
Select a failing request and a comparable successful request: the same task class, processing path, and relevant input conditions. Record their trace and request identifiers, effective workflow configuration, deployment version, model/profile, and retrieval collection. Use the application log attributes defined in 8.2.1 for these associations rather than inferring configuration from a release’s intended settings.
Read parent-child spans to establish which operation invoked which dependency. OpenTelemetry span status has three values, Unset, Error, and Ok; status describes the instrumented operation, not business correctness. A completed workflow does not establish that every tool attempt succeeded or that retrieved content supported the answer. Inspect child outcomes and the application result check.
Separate absence of telemetry from evidence that an operation did not run. If a tool span is missing, first check whether the client recorded dispatch and the service recorded receipt under the request ID. A broken propagation boundary can otherwise look like a skipped operation. Once the path is complete, identify the earliest relevant divergence from the healthy comparison and test whether it explains later events.
Distinguish tool selection, execution, and recovery
For a tool incident, ask three separate questions: did the agent choose the intended tool with valid arguments, did the service execute it successfully, and did the workflow represent its outcome correctly? Compare the recorded tool name and argument-validation result with the service log. A rejected request never reaching business execution needs a different repair from a service that receives valid input and fails internally.
For workflow._type: react_agent, NeMo Agent Toolkit makes recovery behavior explicit. pass_tool_call_errors_to_agent defaults to True, so tool errors can become observations for another reasoning step rather than terminating execution. tool_call_max_retries defaults to 1; max_tool_calls defaults to 15. Inspect the effective values and the sequence of attempts before interpreting a parent workflow’s successful termination.
An agent-response parsing failure has separate controls: retry_agent_response_parsing_errors, parse_agent_response_max_retries, and raise_on_parsing_failure. With raise_on_parsing_failure=True, exhausted parsing failures raise ReActAgentParsingFailedError. Do not classify that as a remote tool outage merely because it occurs near a tool step.
If invalid arguments recur, test the description, schema, or upstream context with the same case; see 2.6.3. If valid calls fail inside the dependency, the tool-service owner investigates the corresponding service error. Raising retries without distinguishing these paths can repeat the same defect and obscure its origin.
Distinguish retrieval failure from empty usable context
A retrieval span can complete successfully and still return nothing useful. Record application diagnostic fields for collection, filter, candidate count, retained count, and the source identifiers admitted to the prompt. These are application-defined observations. Their purpose is to locate the stage that lost the relevant material, not to log the entire corpus.
The NVIDIA RAG Blueprint troubleshooting page identifies overly restrictive confidence filtering as a cause of no returned documents. Its request field is confidence_threshold; the server default can be set through RERANKER_SCORE_THRESHOLD. Check that reranking is enabled to provide meaningful relevance scores. In a controlled replay, compare candidates before filtering with retained results, then lower the threshold only if the quality checks still accept the admitted passages. Empty results alone do not justify weakening filtering.
An embedding-dimension error is a different branch. The Blueprint documents APP_EMBEDDINGS_DIMENSIONS and embeddings.dimensions as configuration mechanisms. Match the model’s actual output dimensions rather than retrying a rejected request unchanged. Changing the embedding model or dimensions after ingestion requires re-ingesting documents. Coordinate that repair with the retrieval owner; a query-only configuration change does not repair an incompatible stored representation.
Distinguish inference delay from a failed worker
NIM’s request-troubleshooting guidance first checks /v1/health/ready and container logs for errors or worker exits, then compares a short request on the same model, profile, and endpoint. Readiness does not establish progress on the incident request. Check both client and proxy deadlines against the observed inference interval.
For a healthy server generating slowly, max_tokens bounds output on /v1/chat/completions and /v1/completions; max_output_tokens does so on /v1/responses. These are length limits, not timeouts. Increasing the client timeout is appropriate only if the expected generation time fits the user requirement; it does not fix a worker crash. An invalid HTTP-success response follows the output-check path in 8.2.2.
For GPU OOM, identify the failing allocation in logs. Weight-loading failure points toward a supported profile with suitable precision or parallelism. A KV-capacity failure after weights load can require reducing --max-model-len, equivalently NIM_MAX_MODEL_LEN, which limits input plus output length. Lowering --gpu-memory-utilization instead reduces the KV budget and can worsen a KV-capacity failure. An illegal-memory-access error is not by itself proof of OOM; check model-specific release notes before treating a configuration experiment as a validated remedy.
Readiness, container logs and the failing allocation decide which inference repair applies.
Key distinctions and trade-offs
| Observed path | Discriminating observation | Targeted response |
|---|---|---|
| Tool failure | Invalid arguments versus valid execution failing in the service | Repair caller context/schema versus dependency behavior |
| Retrieval failure | No candidates versus candidates removed by confidence filtering | Repair ingestion/query compatibility versus validate threshold adjustment |
| Inference failure | Slow live processing versus logged worker or allocation failure | Bound generation/deadlines versus repair the supported serving configuration |
Capture the hypothesis, test, predicted distinguishing result, actual result, and changed setting in an incident record. Google SRE warns that active tests and verbose logging can themselves change system behavior. Prefer a controlled replay when a production experiment could alter external state, and record how to restore any setting you change.
Architecture / decision scenario
A manufacturer’s purchasing assistant runs a react_agent workflow that researches supplier terms before calling a quote tool. After a configuration rollout, 18% of 500 requests fail the task check, compared with 1% before it. Operations proposes raising tool_call_max_retries from 1 to 4 because quote-tool errors dominate the logs. The purchasing workflow owner has a four-second response target and rejects that change until the upstream path is checked.
Correlated traces show eight retrieval candidates but zero retained passages in failing cases, followed by quote-tool argument rejection for a missing supplier identifier. Comparable healthy cases retain three passages. Inference spans remain near 0.9 seconds in both groups, and serving logs show no worker exits. These observations favor retrieval filtering over tool capacity or inference failure.
The retrieval owner replays the same requests with the prior confidence_threshold, preserving the collection and model configuration. Relevant passages return and quote-tool arguments validate; replayed task failures fall from 90 of 500 requests to 6 of 500 (1.2%). The team restores that setting and checks result quality and latency before closing the incident. The incident record names restrictive filtering as the initiating defect, repeated tool errors as propagation, and inadequate retained-count monitoring as a detection gap.
Common exam traps
- Assigning root cause to the last error or longest span without testing alternatives.
- Treating a recovered tool error as invisible because the workflow completed.
- Retrying argument or embedding-dimension defects unchanged.
- Assuming empty retrieval proves the vector service was unavailable.
- Increasing inference timeouts when the worker has exited.
- Applying one memory adjustment to every startup failure.
What to remember
Follow the causal path from the first relevant divergence to the user-visible failure. Separate tool arguments from execution, retrieval candidates from admitted context, and slow inference from failed serving. Verify one targeted change against the incident case and a healthy comparison before declaring the cause resolved.
Knowledge check
A procurement agent built on the NVIDIA RAG Blueprint changes confidence_threshold and several agent settings in one rollout, and task failures rise from 2% to 15%. The logs are dominated by a pricing tool rejecting requests for a missing part number. Correlated traces of failing requests show twelve retrieval candidates and zero retained passages; healthy requests retain four. Inference spans and serving logs look the same in both groups. What should the team test first?
tool_call_max_retries and compare the failure rateconfidence_thresholdA team switches its RAG Blueprint deployment to a different embedding model after document ingestion. In this deployment, the new model emits 1,024 dimensions, the stored embeddings have 2,048 dimensions, and the query configuration still requests 2,048. Queries fail with a dimension-configuration error. What restores accurate retrieval?
APP_EMBEDDINGS_DIMENSIONS=1024 and re-ingest the documentsAPP_EMBEDDINGS_DIMENSIONS=1024 and keep the existing collectionconfidence_threshold on the failing query requestsA NIM-backed step times out at the client’s 30-second deadline. /v1/health/ready reports ready, but the container logs show a worker exit shortly before the timeouts began, and a short request to the same model, profile and endpoint also fails. What should the inference owner do?
max_tokens on the step’s requeststool_call_max_retries in the workflowmax_tokens is a length limit, not a remedy for a worker crash.A NIM for LLMs deployment of a model with a 128K native context crashes at startup after its weights load. The logs report insufficient memory during KV-cache allocation. The workload needs at most 16K tokens of input plus output per request. What change addresses the failure?
--gpu-memory-utilization to leave more headroommax_tokens in each client requestNIM_MAX_MODEL_LEN to fit the workloadNIM_MAX_MODEL_LEN , equivalent to --max-model-len , limits input plus output length per request. Lowering --gpu-memory-utilization shrinks the KV budget and can worsen a KV-capacity failure, and a per-request output limit does not change what the server sizes at startup.A NeMo Agent Toolkit react_agent workflow fails with ReActAgentParsingFailedError immediately after a step that called an inventory tool. The on-call engineer proposes paging the inventory service owner. What should the investigation examine first?
tool_call_max_retries settingraise_on_parsing_failure=True , this error means the agent’s response still failed to parse after its parsing retries, a path controlled separately from tool-call errors. Its position next to a tool step does not make it a tool outage, so inspect the unparseable response before attributing the failure to the tool service or changing tool retry limits.References
- OpenTelemetry, Traces
- NVIDIA, Configure the ReAct Agent
- NVIDIA, Troubleshoot NVIDIA RAG Blueprint
- NVIDIA, Troubleshooting Request Timeouts and Responses
- NVIDIA, Troubleshooting GPU Memory Out-of-Memory Errors
- Google, Site Reliability Engineering: Effective Troubleshooting