NVIDIA NIM Operator Observability

Why it matters

A NIM that Kubernetes reports as running can still be unready or failed as a managed resource. Operator metrics give the platform team that deployment signal, which is separate from inference health.

What to practise

Install kube-prometheus-stack and the Prometheus Adapter with Helm. Add spec.metrics to a NIMService, with enabled: true and a serviceMonitor.additionalLabels entry that matches the Prometheus release. Annotate k8s-nim-operator-metrics-service so Prometheus scrapes it, then read nimService_status_total, nimCache_status_total and nimPipeline_status_total by state. Keep three sources apart: these resource-state gauges, the Operator’s own controller-runtime and workqueue metrics, and the Triton metrics a non-LLM NIM serves once you set spec.expose.service.metricsPort.

How it fits

This is the deployment row of 8.1.2’s dashboard: resource state per NIM, owned by the Kubernetes platform team. Compare it with the inference latency and token metrics that 8.1.1 builds its targets from — a Ready count does not prove that requests complete.