NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.1 Define monitoring dashboards and reliability metrics.
Join GPU health to agent outcomes
Completed by the knowledge check
Learning objective
Build GPU, inference, and agent health dashboards and alerts with DCGM Exporter, Prometheus, and Grafana.
Core idea
Build a DCGM Exporter, Prometheus, and Grafana dashboard that answers three questions: are GPU resources healthy, is inference meeting demand, and are agents completing useful tasks? A GPU utilization panel answers only part of the first question. An agent can fail because of retrieval or tool errors while its inference server remains healthy.
DCGM Exporter supplies GPU telemetry, Prometheus collects metrics, and Grafana presents them. In the design here, Grafana-managed alert rules query Prometheus and notification policies route alerts to owners. Keep thresholds, pending periods, and notification ownership explicit.
See 8.1.1 for SLI definitions. Here you connect those measurements to component signals and actionable alerts; distributed trace correlation and detailed root cause analysis belong to 8.2.
How it works
Collect GPU and inference signals separately
DCGM Exporter exposes GPU metrics at /metrics, with the documented listener on port 9400. You can deploy it as a DaemonSet on GPU nodes. DCGM_EXPORTER_COLLECTORS, also available as -f, selects the field collection file; DCGM_EXPORTER_KUBERNETES, also -k, enables mapping metrics to Kubernetes pods. Verify the actual collection file before importing a dashboard: selecting a panel does not enable its underlying metric.
NVIDIA’s collector file defines DCGM_FI_DEV_GPU_UTIL as GPU utilization in percent, DCGM_FI_DEV_FB_USED as framebuffer memory used in MiB, and DCGM_FI_DEV_GPU_TEMP as GPU temperature in Celsius. It defines DCGM_FI_DEV_XID_ERRORS as a gauge containing the last XID error value. An unchanged nonzero value does not count fresh failures. For new-XID alerts, enable DCGM_EXP_XID_ERRORS_TOTAL in the custom collection file (it is commented out in the default file) and query increase() over a window.
Preserve GPU identity and workload association. Under Multi-Instance GPU (MIG), the exporter documentation requires GPU-instance-level reporting for pod attribution. If the host already runs nv-hostengine, the documented -r option connects the exporter to that instance; check the documented DCGM client/hostengine compatibility guidance before combining versions.
NIM for LLM and VLM uses /v1/metrics, a different path from DCGM Exporter. Configure the Prometheus job’s metrics_path accordingly. NVIDIA documents pass-through of the vLLM backend’s native metrics; inspect the deployed endpoint before choosing queries rather than importing names from another backend or NIM release.
For a vLLM-backed endpoint that exposes them, useful metrics include vllm:num_requests_waiting, a queue gauge; vllm:time_to_first_token_seconds, a histogram; and vllm:e2e_request_latency_seconds, another histogram. These measure inference rather than the complete agent task; use 8.1.1’s histogram aggregation rules. vLLM pre-allocates GPU cache according to gpu_memory_utilization, so DCGM_FI_DEV_FB_USED describes allocation, not live serving pressure. Use vllm:kv_cache_usage_perc for KV-cache usage; this gauge reports 1 for 100% usage.
Add deployment and agent health
For NIM service telemetry, set spec.metrics.enabled: true on a NIMService and configure spec.metrics.serviceMonitor.additionalLabels to match the monitoring installation; NVIDIA’s example uses a release label. This makes the NIM’s own /v1/metrics endpoint discoverable through a ServiceMonitor.
Scrape the Operator’s separate metrics service for nimService_status_total, which reports resource counts by state, including Ready, NotReady, and Failed. NVIDIA’s example annotates k8s-nim-operator-metrics-service in namespace nim-operator with prometheus.io/scrape=true. Despite its _total suffix, NVIDIA documents it as a gauge. Plot state counts directly; readiness, inference completion, and task success remain separate measurements.
At the agent layer, expose the team-defined eligible-request, terminal-outcome, completion-duration, and assessment-coverage measurements specified in 8.1.1. NeMo Agent Toolkit provides diagnostic workflow traces through general.telemetry.tracing with _type: otelcollector, an endpoint, and a project; the documented installation extra is nvidia-nat[opentelemetry]. This tracing configuration sends traces to a collector; it does not by itself define the business-success metric or a Prometheus scrape endpoint for that metric.
Keep bounded dimensions such as service, deployment version, and outcome category in metric series. Keep individual request identifiers in diagnostic records. This avoids a separate time series for every task.
Organize panels by the decision they support
Use a shared time range and workload selection so panels describe the same deployment. The following layers serve different owners:
| Dashboard layer | Signals | Decision | Owner |
|---|---|---|---|
| GPU | DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, temperature, XID value | Inspect allocation, utilization, and device health | GPU platform team |
| Inference | vllm:kv_cache_usage_perc, waiting requests, latency histograms | Investigate serving queue pressure and latency | Inference service team |
| Deployment | nimService_status_total by resource state | Investigate unavailable or failed NIM resources | Kubernetes platform team |
| Agent | User-boundary latency, terminal outcomes, task success and assessment coverage | Investigate workflow or result-quality degradation | Agent service owner |
| Collection | Prometheus up for each discovered target | Restore missing telemetry | Monitoring owner |
Prometheus sets up to 1 for a successful scrape and 0 for a failed scrape. It measures collection reachability, not whether an assistant can complete a task. Check the expected target inventory too: a target missing from discovery is different from a discovered target whose scrape fails.
Make alerts actionable and observable
For Grafana-managed alert rules, choose an evaluation interval and a pending period. The pending period requires the condition to persist before the rule fires, filtering short spikes at the cost of detection delay. Configure No Data and Error handling deliberately; Grafana documents these states for Grafana-managed rules. An empty panel must not silently become proof of health.
Use alert labels to select a notification policy and its contact point. Grafana notification policies can group related alert instances into one notification. Attach the affected service and a runbook reference so the receiving owner has a concrete next action. Choose one place to own each alert; configuring independent copies in Grafana and Prometheus creates competing pending periods and duplicate notifications.
Prioritize sustained user impact. GPU pressure helps explain a latency alert, but high utilization alone can mean productive work. Device health and missing-telemetry alerts need their own response because a task SLI can remain normal briefly while redundancy or visibility disappears. Test collection failure and a controlled threshold breach before relying on notification delivery.
Key distinctions and trade-offs
A low GPU utilization value can accompany a slow tool dependency; a high value can accompany healthy throughput. Correlate utilization with queue depth and user-boundary latency before choosing a capacity response.
A longer pending period reduces noise but delays intervention. DCGM Exporter collects at DCGM_EXPORTER_INTERVAL (-c), documented in milliseconds with a 30,000 ms default. Keep that job’s scrape interval at least as long as its collection interval; faster scrapes repeat collected values at extra storage cost.
A dashboard visualizes measurements, an alert rule evaluates a condition, and a notification policy chooses delivery. A working chart does not verify that an on-call owner receives an alert.
A dashboard and an alert rule read the same Prometheus metrics, but only the alert rule’s labels select a notification policy and its contact point.
Architecture / decision scenario
A manufacturer runs a maintenance retrieval assistant on a fixed four-GPU cluster with no budget for more GPUs this quarter. Its illustrative completion p95 target is ten seconds at 40 requests per minute. The GPU dashboard shows 95% utilization during a busy shift, which tempts the platform team to page on every interval above 90%.
The service owner instead combines the layers. During healthy operation, queue depth stays below two requests, completion p95 is seven seconds, task success is 96%, and all scrape targets report up=1. High utilization alone produces a capacity-review item, not a repeated user-impact page.
After a workload change, the inference queue exceeds 20 waiting requests, vllm:kv_cache_usage_perc reaches 0.95, and completion p95 reaches 18 seconds while GPU utilization remains near 95%. A Grafana-managed user-latency rule evaluates every 30 seconds with a two-minute pending period. Its service label routes the sustained breach to the inference owner, who uses the queue, KV-cache, and GPU panels to investigate serving pressure before changing admission or workload settings.
If task success instead falls to 80% while inference latency stays normal, the agent owner investigates workflow results rather than requesting GPUs. If the DCGM target reports up=0, the monitoring owner repairs collection and checks discovery and reachability; the team does not celebrate the missing utilization line as recovered capacity. Each path assigns a different repair owner.
Common exam traps
- Scraping NIM at DCGM Exporter’s
/metricspath. - Treating
nimService_status_totalor the last XID error value as a counter. - Assuming high GPU utilization proves overload or task success.
- Expecting trace export to create application success metrics automatically.
- Treating no data as zero failures.
- Verifying a panel but never testing the notification path.
What to remember
Join GPU, inference, deployment, agent, and collection health without collapsing their meanings. Verify exposed metrics and selectors, assign alert evaluation and delivery explicitly, and route each failure to the owner who can repair it.
Knowledge check
A team runs a NIM for LLMs container with docker run on a GPU host, without the NIM Operator, and adds a static Prometheus job that targets the container’s port 8000 with default scrape settings. The target reports up = 0 with HTTP 404, and no vllm: series appear. What fixes the job?
DCGM_EXPORTER_KUBERNETES on the exporterspec.metrics.enabled: true on a NIMServicemetrics_path to /v1/metrics/v1/metrics , not at the /metrics path a default job scrapes. DCGM Exporter on port 9400 supplies GPU telemetry rather than the inference metrics, pod mapping does not change the NIM’s path, and the NIMService setting applies only to a deployment the Operator manages.A Prometheus alert fires when DCGM_FI_DEV_XID_ERRORS > 0. After one XID error on a node three days ago, it is still firing, although no new errors have occurred. The GPU platform team wants an alert for new XID errors. What should it change?
increase() to DCGM_FI_DEV_XID_ERRORS over a ten-minute windowDCGM_EXP_XID_ERRORS_TOTAL and apply increase() over a windowDCGM_FI_DEV_GPU_TEMP also exceeds its thresholdDCGM_FI_DEV_XID_ERRORS is a gauge holding the last XID value, so an unchanged nonzero value does not mean fresh failures, a repeat of the same code leaves it unchanged, and increase() over it reads a code rather than a count. The counter DCGM_EXP_XID_ERRORS_TOTAL is commented out in the default collection file; enabling it gives increase() real events to count.A maintenance assistant runs on a vLLM-backed NIM on a fixed GPU pool with no budget for more GPUs. During busy shifts GPU utilization sits near 95% while completion p95 stays inside its ten-second target and task success is normal. The platform team proposes paging whenever utilization exceeds 90%. Which alert should page the inference owner instead?
DCGM_FI_DEV_FB_USED value near the framebuffer capacityup = 0 for a nodeAfter a node-pool upgrade, the GPU utilization line for one node disappears from the dashboard, and a Grafana-managed rule on that metric, whose No Data handling keeps it Normal, stays quiet. What does the empty line establish, and what follows?
up exists only for discovered targets. Leaving No Data handling to keep the rule Normal is how an empty panel turns into apparent health.A Kubernetes platform team scrapes the NIM Operator’s metrics service and alerts on increase(nimService_status_total{state="Failed"}[10m]) > 0. The rule fired briefly when a NIMService failed and then resolved, although the resource is still Failed an hour later. Which condition matches the metric?
rate(nimService_status_total{state="Failed"}[1h]) > 0increase(nimService_status_total{state="Ready"}[10m]) < 0sum(nimService_status_total) > 0nimService_status_total{state="Failed"} > 0_total suffix, NVIDIA documents nimService_status_total as a gauge of resource counts by state, so the current Failed count is the condition. rate() and increase() see only the transition, which is why the rule resolved, and a sum across all states is nonzero whenever any NIMService exists.References
- NVIDIA, DCGM Exporter
- NVIDIA, DCGM Exporter default collectors
- NVIDIA, Logging and Observability
- vLLM, Production Metrics
- NVIDIA, NVIDIA NIM Operator Observability
- NVIDIA, Observe Workflows
- Prometheus, Jobs and instances
- Grafana Labs, Alert rule evaluation
- Grafana Labs, Notification policies
- vLLM, Optimization and Tuning
- Prometheus, Query functions: increase()