Validate NIM from health to completion

Learning objective

Run a NIM container, validate health, and call its OpenAI-compatible API.

Core idea

A running NIM container is one step toward usable inference. Its proxy can answer a liveness request before the model is loaded. Your deployment acceptance therefore needs three observations: the container runs, the model reports ready, and a request to the intended model produces the expected response structure.

For NIM for LLM and VLM, these observations use GET /v1/health/live, GET /v1/health/ready, and POST /v1/chat/completions. Keeping them separate tells you which owner must act: the runtime administrator, the inference operator, or the application integrator. See 4.4.1 for Kubernetes admission; this lesson establishes what the underlying endpoint proves.

How it works

Establish the runtime and artifact path

Start with the image and backend appropriate to the model and GPU. The NIM prerequisites page lists Docker 24.0 or later, NVIDIA Container Toolkit 1.14.0 or later, and GPU driver 580 or later. The toolkit enables GPU access from Docker; installing a container image does not provide the host driver. Model-specific memory and hardware eligibility still need the selected profile, covered in 7.2.2.

Separate registry access from model-source access. The quickstart permits Docker login and NGC_API_KEY to be omitted for eligible public-catalog NIMs. Production Branch models and releases before 2.0.10 require an NGC key. Also check credentials required by your image, tag, or model source. Model-source credentials such as HF_TOKEN can remain necessary even when pulling the public image is keyless. Check the selected assets rather than turning either rule into a universal prerequisite.

Persist the artifact cache so restarting a container does not mean downloading the model again. NIM_CACHE_PATH defaults to /opt/nim/.cache inside the container. A bind mount connects that directory to a writable host directory. The cache changes startup work and storage consumption; it does not make the GPU’s model memory persistent.

The deployment shape is short enough to inspect directly:

docker run --gpus=all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  "$NIM_LLM_IMAGE"

Set NIM_LLM_IMAGE to the supported image and explicit tag, LOCAL_NIM_CACHE to an existing writable host directory, and NGC_API_KEY in the environment when the selected artifacts require it. Omit the key line for eligible keyless assets. This shows the deployment interface, not a hardware recommendation: --gpus=all exposes available GPUs, while the selected profile determines the inference arrangement.

A cache can also fail before inference starts. The environment-variable reference documents NIM_CACHE_PROBE_TIMEOUT, a 60-second default deadline for the initial artifact-cache reachability probe. An unreachable NFS, CIFS, or FUSE mount causes an exit within that deadline. Repair the mount or connectivity; increasing an application request timeout does not make the cache reachable.

Test readiness separately from liveness

GET /v1/health/live returns HTTP 200 when the container is running. The API reference identifies this as an nginx-served check that does not require a loaded model. GET /v1/health/ready returns 200 when the model is loaded and inference is available. These are different assertions even when both eventually succeed.

Both endpoints use the server port (8000 in the quickstart) unless NIM_HEALTH_PORT is set. If you set NIM_HEALTH_PORT, direct health requests to that port instead. A check aimed at the wrong port cannot diagnose model initialization. Likewise, a working proxy does not establish that artifact download, profile selection, and backend loading all succeeded.

Keep an endpoint out of the application’s ready pool until the readiness check succeeds. If live succeeds while ready does not, examine the selected profile, cache reachability, download credentials, and backend startup rather than declaring the service healthy from its process status alone. The purpose is to resolve the failed stage before sending ordinary requests through it.

Discover and call the served model

Query GET /v1/models to find the model name actually served. NIM_SERVED_MODEL_NAME, when explicitly set, determines that ID; otherwise NIM derives it. A registry repository name or a filename is not a substitute for this discovery step.

For a minimal nonstreaming check, send the discovered ID as model, with messages and a bounded max_tokens value:

{
  "model": "SERVED_MODEL_ID",
  "messages": [{ "role": "user", "content": "Reply with a short greeting." }],
  "max_tokens": 32
}

Replace SERVED_MODEL_ID with the returned ID and send the body as JSON to http://localhost:8000/v1/chat/completions. The 32-token value is a chosen smoke-test budget, not a NIM limit. Inspect the completion’s model, choices, choices[0].message, and finish_reason, as illustrated in NVIDIA’s quickstart response. A successful HTTP exchange alone does not establish that the application parsed the intended answer.

OpenAI compatibility describes an interface. It does not establish identical model capabilities, response quality, or supported input modalities across images. Keep this first check text-only and nonstreaming so you can isolate endpoint and parsing defects before adding the application’s tool or multimodal workload.

Key distinctions and trade-offs

Use the observation that answers the deployment question:

A persistent cache consumes disk but avoids repeated artifact downloads. Keyless access removes a credential requirement only for eligible assets. A short inference check consumes GPU time, yet catches a wrong model ID or incompatible parser before production traffic discovers it.

Architecture / decision scenario

A hospital’s employee assistant must keep prompts on-premises and bring a replacement node into service within 180 seconds. In its rehearsal, a shared network cache gives every replica access to the downloaded artifacts: live succeeds after 8 seconds and ready after 94 seconds. With node-local storage on a fresh node, the model download extends readiness to 214 seconds. These are measured scenario timings, not NIM guarantees.

The infrastructure team proposes node-local caches to remove the shared mount from the startup path. The inference owner instead chooses the shared cache: a fresh node misses the 180-second target by 34 seconds with the measured local download. Existing local caches help restarts on those nodes but do not populate a fresh node. The shared cache saves that download while making mount availability an explicit infrastructure responsibility.

If the shared mount becomes unreachable, the default NIM_CACHE_PROBE_TIMEOUT=60 makes the container exit within 60 seconds. Setting the documented example value of 120 permits a longer probe but does not repair the mount or guarantee recovery within 180 seconds. The infrastructure owner restores connectivity; the inference operator checks live, waits for ready, discovers the served ID through /v1/models, and sends the bounded chat request. The application integrator checks the completion fields before enabling traffic. Record the image, profile, cache path, and acceptance timings so a cache-placement change can be tested against the same target.

Common exam traps

• Treating a liveness HTTP 200 as proof the model can accept inference.
• Sending a registry image name as the API’s model without checking /v1/models.
• Assuming every NIM requires an NGC key, or that keyless image access removes all model-source credentials.
• Treating a persistent artifact cache as persistent GPU memory.
• Checking port 8000 after moving health endpoints with NIM_HEALTH_PORT.
• Equating API compatibility with application quality under load.

What to remember

Validate runtime access, artifact access, model readiness, and the application request as separate stages. The health endpoints identify different states; model discovery and a bounded chat completion establish that your integration reaches the intended serving path.

Knowledge check

References

• NVIDIA, Prerequisites
• NVIDIA, Quickstart
• NVIDIA, API Reference
• NVIDIA, Environment Variables