NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.2 Track logs, errors, and anomalies for root cause diagnosis.
Scope anomaly detectors by population and denominator
Completed by the knowledge check
Learning objective
Detect anomalies in latency, token use, error rates, and outputs.
Core idea
An anomaly is a departure from expected behavior for a defined population. It is a reason to investigate, not a root-cause label. A token spike can reflect longer legitimate inputs, repeated model calls, or a prompt defect; an error-rate improvement can conceal invalid answers returned with HTTP 200.
Monitor the complete workflow alongside its dependencies. Use the reliability targets from 8.1.1 to decide whether a deviation needs immediate attention, and the correlation from 8.2.1 to select concrete executions. This lesson defines detection conditions. Release benchmarking belongs to 8.3, and the controlled profiler analysis of individual steps belongs to 3.5.2.
How it works
Define the observation before the threshold
For each detector, write a rule record containing the measured event, population, time window, expected range, minimum observation count, and response owner. Separate completed workflows, individual model calls, and tool attempts: their denominators differ. Count a workflow that ends at its deadline as a failed terminal outcome rather than silently dropping it because no model response arrived.
NIM exposes backend-native Prometheus metrics at /v1/metrics; its logging and observability documentation says those metrics pass through without renaming. Inspect the deployed endpoint before binding a detector to a metric name. A backend-serving metric describes its own requests, not automatically the user workflow that may call it several times.
Add application measurements where the user outcome exists. In the examples below, agent_requests_total, agent_output_checks_total, and the classic histogram agent_request_duration_seconds are application-defined metric names, not built-in NVIDIA metrics. Define their labels and terminal outcomes in the instrumentation code. The application owner must reconcile counter totals with request logs before trusting an alert.
Detect latency shifts without hiding the tail
Measure duration from workflow admission to terminal result, including tool waits, retries, and validation. Separate successful and failed-request latency: Google SRE explains that fast errors can make an aggregate latency look better while service deteriorates.
For the illustrative classic histogram, calculate a fleet p95 with histogram_quantile over the sum of bucket rates, retaining the le label. For example, the expression is histogram_quantile(0.95, sum by (le) (rate(agent_request_duration_seconds_bucket[5m]))). To retain task classes, include the task-class label in that aggregation. Never average replica p95 values to obtain a fleet p95; aggregate the histogram observations instead.
Histogram quantiles are estimates. Classic histogram error depends on bucket width near the quantile. Place useful buckets around the response target so a broad bucket does not turn a small latency movement into an apparent large regression. Use representative traffic counts alongside percentiles; a window with three requests should not inherit the confidence of a window with 3,000.
Normalize tokens and distinguish attempts from outcomes
Track prompt and completion tokens separately, together with model-call count per workflow. NeMo Agent Toolkit’s profiler documents prompt_tokens and completion_tokens on LLM_END events in standardized_data_all.csv, grouped by example_number and llm_name. Those fields support investigation of the call-level source of an observed token increase; they are profiler artifacts, not a continuously running production anomaly detector.
Build the live workflow token measure from observed call usage in your serving integration and retain which calls contributed. Treat missing usage as unknown, with a separate missing-usage count; substituting zero makes timeouts and failed calls appear cheap. Compare token distributions within comparable task and input-length classes. A rise in total tokens per minute can merely reflect more traffic, whereas tokens per completed task exposes a different change.
For errors, record both failed attempts and failed terminal workflows. A retry can raise dependency failures while preserving user success, but still consume extra tokens and latency. Compute an error fraction from the relevant failed count divided by the matching total, over the same interval. With application counters, use rate for both numerator and denominator rather than comparing accumulated totals from processes with different lifetimes. Keep the observation count visible, especially for low-volume classes.
Detect invalid outputs even when transport succeeds
NIM’s request-troubleshooting documentation distinguishes HTTP success from a required tool call being present. For /v1/chat/completions with tool_choice=required or a named function, verify choices[].message.tool_calls; assemble streamed tool-call deltas before validation. Count an absent required call as an invalid result rather than a successful task.
Other output detectors need equally explicit definitions. For a /v1/completions result that must contain an entire record, inspect final text and finish_reason; finish_reason=length means the output limit was reached and requires a completeness check. A transport-success counter alone cannot detect this failure.
For factual answers, use the task-specific human or evaluator labels established in 3.1.1–3.1.3. Track the reviewed population and its denominator separately from automatic structural checks. A low rejection rate on a handpicked review sample is not the fleet’s factual-error rate. Preserve the detector version with each result so changing a validator does not masquerade as changing model quality.
Key distinctions and trade-offs
| Signal | Useful detector | Misleading shortcut | Investigation owner |
|---|---|---|---|
| Latency | Task-specific tail duration over an adequate window | Fleet mean or average of replica percentiles | Workflow operations |
| Tokens | Prompt/completion distributions and calls per comparable task | Total tokens without traffic normalization | Agent implementation owner |
| Errors | Attempt failures and terminal failure fractions | HTTP failures alone | Dependency and workflow owners |
| Outputs | Required-call, completeness, and reviewed-quality failures | Treating HTTP 200 as an acceptable answer | Product quality owner |
Combine an absolute user-impact threshold with a baseline comparison. A relative change can expose a regression below the service target, while an absolute threshold catches a chronically poor baseline. Reserve paging for actionable user symptoms; use lower-urgency investigations for unexplained cost or distribution shifts without immediate user harm. This follows Google SRE’s distinction between symptom alerting and cause-oriented debugging.
Architecture / decision scenario
A media agency operates a meeting companion that summarizes calls and creates follow-up tasks. It requires a complete summary and a task-tool call within six seconds. In comparable five-minute windows of 2,000 workflows, a prompt rollout moves p95 latency from 2.4 to 7.8 seconds and mean tokens per workflow from 1,800 to 7,200. Transport errors move only from 0.4% to 0.5%, while output-check failures rise from 1% to 12%.
The serving team proposes waiting for HTTP errors before acting. The workflow owner rejects that alternative: it misses both the six-second breach and invalid successful responses. For this service, the owner defines an investigation threshold of p95 above six seconds for ten minutes, or output-check failures above 5% with at least 100 checked results. These are local policy values, not NVIDIA limits.
The quality owner validates assembled tool_calls and completeness. Operations groups latency and token observations by task and release, then opens correlated traces from failing and healthy requests. A fourfold token increase is not enough to diagnose a slower GPU: call counts and prompt lengths must distinguish repeated calls from expanded context. The owner stops the rollout on user-impact conditions while the trace investigation tests those causes.
Common exam traps
- Treating a statistical deviation as a proven cause.
- Averaging per-replica percentiles or ignoring histogram resolution.
- Comparing token totals without normalizing for traffic and task mix.
- Counting recovered attempts as identical to failed workflows.
- Replacing missing usage with zero.
- Measuring output quality through HTTP status or a biased review sample.
What to remember
Define the population and denominator before the threshold. Combine latency tails, normalized token use, attempt and workflow errors, and explicit output checks. A detector should identify a consequential deviation and the executions to inspect; diagnosis determines the cause.
Knowledge check
A marketing campaign triples traffic to a support assistant for a week, and a detector on total tokens per minute fires. The release owner suspects that a prompt shipped the same week expanded each request. Within the same task and input-length classes, which measure distinguishes increased traffic from increased consumption per workflow?
In a workflow token detector, model calls that hit the client deadline return no usage data. How should the detector treat these calls?
A meeting companion calls /v1/chat/completions with streaming and tool_choice=required, because every response must create follow-up tasks. After a prompt change, HTTP error rates are flat, but users report missing tasks. Which detector catches a missing required call whatever its cause?
tool_calls contain the required callfinish_reason set to lengthchoices[].message.tool_calls and counts an absent call as an invalid result. Streamed tool calls arrive as deltas and must be assembled before validation, and finish_reason=length checks completeness against the output limit, a neighbouring failure.A contract-review assistant has a six-second p95 target. Its p95 has been about nine seconds since launch, and its only latency detector compares each hour with the previous week. The detector has never fired. What should the owner add?
After a tool-service release, failed tool attempts double. Retries recover them: terminal workflow success and p95 latency stay within their targets, while tokens per completed task rise 15%. No other user-impact or budget target is breached. How should the team respond?