Setting up Prometheus

Why it matters

An agent’s latency and failure targets mean little until GPU pressure is measured beside them. This tutorial builds the GPU telemetry layer that the agent and inference views are joined to.

What to practise

Install prometheus-community/kube-prometheus-stack with Helm, setting serviceMonitorSelectorNilUsesHelmValues to false and adding a gpu-metrics job under additionalScrapeConfigs. Deploy the DCGM Exporter Helm chart and note how its arguments value sets the collection interval. Confirm the scrape by querying DCGM_FI_DEV_GPU_UTIL in Prometheus, then expose Grafana and import dashboard 12239. Finish by running the DeepStream demo and watching the GPU panels respond to a real workload.

How it fits

This is the DCGM Exporter, Prometheus and Grafana layer of 8.1.2. GPU utilisation alone does not tell you whether agent tasks are completing; 8.1.1 sets those targets, and 8.1.2 decides which dashboard owns each signal.