Fit visual detail within serving budgets

Learning objective

Balance VLM resolution, frame count, and batch size against GPU memory and latency.

Core idea

A vision-language model (VLM) spends resources on the visual content you retain, the requests you admit together, and the text it generates. Resolution preserves spatial detail; frame count preserves temporal observations; batch size controls how much work competes for serving capacity. Raising all three can exhaust GPU memory or violate latency targets before it improves useful answers.

Choose the smallest visual representation that passes task-specific quality tests, then find a serving operating point under realistic concurrency. See 7.5.1 for decoding routes and 4.3.1 for serving latency terminology. Here the decision is which visual information earns its cost and which limit controls the failing allocation.

How it works

Control the frames the VLM processes

Video Search and Summarization (VSS) Real-Time VLM exposes VLM_INPUT_WIDTH and VLM_INPUT_HEIGHT for input frame dimensions. VLM_DEFAULT_NUM_FRAMES_PER_SECOND_OR_FIXED_FRAMES_CHUNK controls sampling, with a documented default of 30. In the default fixed-frame mode, it selects that many equally spaced frames per chunk. With --use-fps-for-chunking, the value instead means frames per second. Thirty frames per chunk and thirty frames per second can create very different workloads as chunk duration changes. Chunk duration itself is set per request with chunk_duration (and chunk_overlap_duration), so it is the third sampling lever beside frames per chunk and FPS mode.

Reducing resolution can erase small text within every frame. Reducing frame count can omit the only frame containing an event. For a fixed frame count, longer chunks spread observations further apart. Test spatial and temporal losses independently: a model that reads a stationary label correctly can still miss a brief warning between samples.

The VSS Known Issues section gives a model- and platform-specific example. With Cosmos Reason3 on Jetson Thor or DGX Spark, more than two concurrent live streams at 448×448 and 80 frames, approximately 8K vision tokens, can crash the microservice or reboot the device through memory pressure. NVIDIA recommends two or fewer streams at that budget, or VLM_INPUT_WIDTH=372, VLM_INPUT_HEIGHT=372, and 30 fixed frames per chunk, approximately 2K vision tokens, to support higher concurrency. Those figures describe that documented configuration, not a universal pixel-to-token conversion.

Separate processing controls from memory hints

On a vLLM-backed NIM, media enters through a mixed Chat Completions content array, including image_url or video_url for supported models. A DALI preprocessing sequence does not by itself set the number of frames the model processes from a video_url. Configure the model’s processor and verify its effective inputs.

vLLM’s limit_mm_per_prompt, exposed as --limit-mm-per-prompt, caps media items per prompt by modality. A modality limit of 0 disables that modality and avoids allocating memory for it. Its configurable video form accepts count, num_frames, width, and height. The count limits video items; the size hints affect activation-memory profiling only. Setting a num_frames hint does not resample the actual video, and the encoder cache still depends on actual runtime inputs.

Use mm_processor_kwargs, exposed as --mm-processor-kwargs, to change processed input size where the selected model supports it. vLLM documents max_pixels for Qwen2-VL-series processing, with an example of 768 * 768, and max_dynamic_patch for InternVL. These are model-specific processor options, not interchangeable settings. Check support in the backend version shipped in the NIM image. Pass these as direct container arguments; if only NIM_PASSTHROUGH_ARGS is available, append to the image’s value — see 4.5.2.

Tune batching and diagnose the failing allocation

See 0.9 for memory beyond model weights. Here, distinguish visual processing allocations from key-value (KV) cache capacity and CUDA graph headroom. A smaller compressed upload does not necessarily reduce those allocations; fitting weights alone does not establish that the intended visual load fits.

VSS VLM_BATCH_SIZE controls the inference batch size and is auto-calculated by default. NVIDIA’s out-of-memory (OOM) guidance suggests trying VLM_BATCH_SIZE=32, lowering VLLM_MAX_NUM_SEQS, or reducing NUM_VLM_PROCS. Batch size, concurrent sequences, inference processes, and frames within one request are separate controls. The suggested batch value is a troubleshooting candidate, not a safe universal maximum.

VSS advises lowering VLLM_GPU_MEMORY_UTILIZATION for startup OOMs, and increasing it only when KV-cache capacity is too low and free GPU memory remains. For NIM’s documented vLLM backend, --gpu-memory-utilization sizes the KV-cache budget after other measured allocations; NIM_KVCACHE_PERCENT configures the same parameter. Lowering it can leave graph headroom but worsen a KV-capacity shortage.

If weights fail to load, select a supported hardware/profile combination. If insufficient KV capacity is the issue, --max-model-len or NIM_MAX_MODEL_LEN bounds input-plus-output sequence length. Context reduction alone might not resolve multimodal allocations that exhaust memory before KV sizing. vLLM’s optimization guidance also recommends reducing max_num_seqs or max_num_batched_tokens when preemption and recomputation harm latency. Choose the remedy from the failing phase, then measure quality and loaded end-to-end latency again.

Key distinctions and trade-offs

Use a separate quality test for each control:

Quality is a constraint alongside memory and latency. Reject an apparently fast setting if it loses required observations. Keep an evaluation record with source interval, preprocessing dimensions, frame sampling, serving configuration, answer correctness, peak memory, and end-to-end latency. That record lets the team attribute a regression to a changed input or a changed serving limit.

Architecture / decision scenario

A manufacturing research assistant reviews equipment demonstrations using VSS Real-Time VLM on a fixed supported deployment. It must answer within eight seconds at p95 and detect at least 95% of annotated two-second warning-light events. Every request uses 10-second chunks (chunk_duration=10 on POST /v1/generate_captions), so in fixed-frame mode 30 frames per chunk means one frame about every 0.33 seconds and eight frames one about every 1.25 seconds. The following trial figures are hypothetical local measurements, separate from NVIDIA’s documented edge-platform issue.

The tempting proposal uses 80 fixed frames per chunk, 448×448 input, and VLM_BATCH_SIZE=64 to maximize detail and throughput. It exhausts memory under the intended load. Reducing only the input dimensions to 256×256 avoids that failure but drops event-answer accuracy to 89% because labels become unreadable. Keeping 448×448 and reducing to eight frames per chunk reaches 91% because warnings fall between observations.

The vision owner instead sets VLM_INPUT_WIDTH=372, VLM_INPUT_HEIGHT=372, and VLM_DEFAULT_NUM_FRAMES_PER_SECOND_OR_FIXED_FRAMES_CHUNK=30, leaving FPS mode disabled. On the unchanged evaluation segments, this candidate scores 97%. With VLM_BATCH_SIZE=32, loaded p95 latency is 10.4 seconds; reducing the batch to 16 yields 7.2 seconds and retains that quality result. Queueing time is included in both measurements; process count and sequence limits remain fixed during the comparison.

The serving owner accepts the batch size of 16 because it clears both constraints. The quality owner retains short-event and small-label cases for regression tests. A smaller profiling hint would not substitute for the tested processor settings. Rising demand requires another capacity or admission decision rather than silently reducing validated visual coverage.

Common exam traps

• Treating frames per chunk, frames per second, batch size, and concurrent sequences as interchangeable.
• Using limit_mm_per_prompt size hints as actual video processing limits.
• Inferring model visual-token counts from uploaded image dimensions alone.
• Fixing a weight-loading OOM by changing client-side frame sampling.
• Lowering --gpu-memory-utilization when the failure is insufficient KV capacity.
• Calling downsampling successful without checking small-object and short-event accuracy.
• Replacing NIM_PASSTHROUGH_ARGS and discarding image-provided startup defaults.

What to remember

Preserve the spatial and temporal information the task needs, then tune concurrent inference around measured capacity. Identify the allocation that fails before changing memory controls. The acceptable operating point meets answer quality, GPU memory, and end-to-end latency constraints together.

Knowledge check

References

• NVIDIA, Real-Time VLM Microservice
• NVIDIA, Image, Audio, and Video Input
• NVIDIA, Troubleshooting GPU Memory Out-of-Memory Errors
• NVIDIA, Advanced Configuration
• vLLM, Optimization and Tuning
• vLLM, Conserving Memory
• vLLM, serve