NVIDIANVIDIA-Certified Professional: Agentic AI2026-100 Prerequisites
Budget GPU memory beyond model weights
Completed by the knowledge check
Learning objective
Explain GPU inference basics: VRAM, precision, quantization, and batching.
Core idea
Video random-access memory (VRAM) holds more than model weights. Serving also needs attention state, activations, and runtime workspace. A model that loads successfully can still lack memory for the intended context length and concurrent requests.
Precision changes numerical representation. Quantization reduces the representation used for selected tensors. Batching schedules work from multiple requests together. These choices interact, but none substitutes for measuring task quality and latency under a representative workload.
How it works
Weights and working memory: A dense model’s weight storage can be approximated as parameter count multiplied by bits per stored parameter, divided by eight. This is a lower-level storage calculation, not a complete deployment estimate: metadata, scales, caches, and runtime allocations add to it. NVIDIA’s NIM profile selector estimates weights, key-value (KV) cache, activations, and overhead when checking available GPU memory.
Precision: FP32 uses 32 bits per value; FP16 and BF16 use 16. TensorRT LLM documents FP4/FP8 quantization support in its default PyTorch backend for Blackwell and Hopper GPU families, with recipe-specific limits. Its hardware matrix lists NVFP4 for Blackwell but not Hopper; consult both hardware and model matrices for the intended recipe. Equal bit widths do not make numerical formats interchangeable, and a model, kernel, and GPU must support the chosen combination.
Quantization: TensorRT LLM lists W4A16 recipes for AWQ and GPTQ: weights use four bits while activations remain at 16 bits. Thus, four-bit weights do not imply four-bit activations or a fourfold reduction in total serving memory. See 3.4.2 for model-size and quantization choices and 7.4.3 for implementation and accuracy evaluation.
Batching: TensorRT LLM’s in-flight batching interleaves requests in context and generation phases; it requires packed input tensors without padding. Its documented scheduling controls distinguish max_batch_size, the request-count bound, from max_num_tokens, the batched token budget. Larger work batches can improve utilization, but exceeding the useful operating point can hurt latency. See 7.4.2 for batching, paged KV cache, and chunked-context configuration.
Key distinctions and trade-offs
Weight precision, KV-cache storage, and request scheduling are separate memory consumers and controls. More simultaneous long requests need more attention state even when weights remain unchanged. Reducing weight storage may release memory for that state; it does not establish the resulting quality or latency.
For NIM, use list-model-profiles to inspect profiles for the selected image and NIM_MODEL_PROFILE to select a documented profile. The backend must match the container’s inference engine. A profile classified as low memory means weights fit but full context does not; the documentation points to reduced --max-model-len. An incompatible profile needs a different memory arrangement, such as supported quantization or higher tensor parallelism. See 7.2.2 for profile selection rather than reconstructing a profile name from memory.
Architecture / decision scenario
A financial research assistant has one GPU with a hypothetical 48 GB memory budget. Its measured 16-bit configuration uses 32 GB for weights and 20 GB for cache, activations, and workspace at the target load: 52 GB total. A proposal to increase batching cannot repair that deficit.
A supported eight-bit candidate measures 16 GB of weights plus 2 GB of quantization metadata and scales. With the same 20 GB of working memory, it needs 38 GB total. The inference owner chooses it only if the team’s evaluation passes and its measured p95 latency stays below the three-second requirement. If batching raises that latency to five seconds, extra throughput is insufficient. These are scenario measurements, not specifications for a named GPU or model. The owner retains memory headroom and tests the intended context distribution before admitting the target load.
Common exam traps
- Comparing weight size alone with GPU capacity.
- Assuming quantization preserves every task’s accuracy.
- Treating batch size as the number of tokens.
- Assuming a supported precision works on every GPU and backend.
What to remember
Budget weights and working memory together. Quantization changes selected representations; batching changes scheduled work. Choose a supported combination that fits memory and passes both quality and latency requirements.
Knowledge check
A support assistant must serve its target load on one GPU with an 80 GB memory budget. Measured at that load, its 16-bit configuration needs 56 GB for weights and 30 GB for KV cache, activations and workspace. Release also requires the team’s evaluation to pass and p95 latency to stay under two seconds. Which plan should the inference owner pursue?
A team plans a W4A16 AWQ recipe for a model whose 16-bit deployment measures 28 GB of weights plus 20 GB of KV cache, activations and workspace at the target load. A reviewer predicts the quantized deployment will need about 12 GB, a quarter of today’s 48 GB. Which estimation approach corrects the reviewer’s mistake?
W4A16 recipe the weights use four bits while activations stay at 16. The raw weight estimate falls from 28 GB to about 7 GB, but quantization metadata, KV cache, activations and workspace still need a budget; the recipe does not quarter the whole footprint or specify four-bit KV storage. Measure those allocations under the target load rather than dividing the previous total by four.In a vLLM-based NIM image, list-model-profiles lists the team’s preferred profile as compatible but low memory and suggests a reduced context length that fits the available GPU memory. That length covers the workload’s input plus output tokens. Which adjustment follows the listing’s recommendation?
trtllm profile of the model inside the same vLLM image--max-model-len that covers the workload--max-model-len , which limits input plus output tokens per request and, in this scenario, still covers the workload. Quantization or higher tensor parallelism is the remedy for an incompatible profile, whose weights alone do not fit, and a profile’s backend must match the engine in the container image.An engineer uses packed inputs with TensorRT LLM and sets max_batch_size to 8, expecting a 4,000-token budget per scheduled batch. Profiling shows that the token count varies with prompt length. Which change directly expresses the token budget rather than a request-count limit?
max_batch_size until batches stay under 4,000 tokensmax_num_tokens to bound the batched input tokensmax_batch_size bounds how many requests a batch holds, while max_num_tokens bounds the batched input tokens after padding is removed, so only the second expresses a token budget. Lowering the request count or scheduling requests alone controls the number of requests rather than expressing a token budget. In-flight batching requires packed inputs without padding.References
- NVIDIA, Quantization
- NVIDIA, Paged Attention, IFB, and Request Scheduling
- NVIDIA, Model Profiles and Selection