NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.4 Tune model parameters (e.g., accuracy, latency-efficiency trade-offs).
Trade model size, precision, and reasoning effort
Completed by the knowledge check
Learning objective
Choose model size, quantization, and reasoning budget for accuracy–latency trade-offs.
Core idea
Choose a complete inference configuration rather than a model name in isolation. Model capability, deployment precision, and reasoning behavior interact: a larger model might meet the quality target only with an expensive reasoning mode, while a smaller model might satisfy the task with shorter generation and more replica capacity.
The deciding artifact is a measured candidate comparison under the target workload. It includes the model, NIM image, selected profile, request controls, quality result, latency distribution, and GPU allocation. See 0.9 for the memory baseline and 3.2.3 for comparing task quality, complete-request latency, and cost per acceptable result. This lesson uses those constraints to select between viable candidates rather than estimate memory from weight count alone.
How it works
Select capability before spending capacity
Model size is a candidate-selection variable, not a quality guarantee. Compare models on the task benchmark from 3.1.1, including cases that distinguish retrieval, tool selection, and multi-step inference. Keep the task’s completion requirements identical. A smaller candidate that answers only the easy cases cannot claim the same service level as a candidate that completes the full workload.
In NeMo Agent Toolkit, llms.<name>.model_name identifies an LLM candidate, and the NIM provider uses _type: nim. The toolkit optimizer supports categorical model choices; see 3.4.3. That configuration change chooses the requested model, but it does not provision a compatible self-hosted NIM or select its startup precision. The platform owner must supply the corresponding serving deployment.
Use more capacity only when it produces an outcome your application needs. A higher average answer score might be irrelevant if the smaller model already exceeds the required threshold and the larger model violates the deadline. Conversely, saving GPU resources is not an improvement if the resulting tool arguments fail the task’s checks.
Treat precision as a supported deployment choice
NIM model profiles are prevalidated inference configurations selected at startup. A model-specific container’s manifest describes backend, precision, tensor parallelism, pipeline parallelism, and LoRA support. Use list-model-profiles for the exact container image, then select a printed full profile ID or complete description with NIM_MODEL_PROFILE. Do not reconstruct a selector from fragments of a naming convention.
The backend must match the image: an SGLang container loads sglang-* profiles. Precision labels such as bf16, fp8, or nvfp4 describe deployment choices; their appearance in the general documentation does not mean every model and GPU offers each one. A profile can also carry a latency or throughput performance variant. Benchmark the variant against the service requirement instead of assuming the throughput-oriented candidate minimizes per-request latency.
NIM distinguishes compatible, low-memory, and incompatible profiles. For low-memory profiles, the weights fit but the full context does not; the listing suggests a reduced --max-model-len. For incompatible profiles, weights alone exceed available memory, and NVIDIA suggests higher tensor parallelism or quantized precision. Shortening an answer’s max_tokens is not a substitute for repairing an incompatible deployment.
Quantization also has separate targets. TensorRT LLM’s quantization documentation covers weights, activations, and KV cache. Its KvCacheConfig(dtype='fp8') controls FP8 KV cache in the LLM API; it is not a NIM environment variable or a request-level precision switch. Use that interface when operating TensorRT LLM directly. With NIM, use the container’s supported profile and documented runtime configuration.
Check both the model and hardware support matrices for a quantization recipe. For example, the TensorRT LLM hardware table lists FP8 per-tensor support for Hopper, but not NVFP4; Blackwell SM100/103 lists both. A recipe’s existence does not establish support on a different architecture. Re-evaluate task quality after precision changes rather than treating reduced memory as a free optimization.
Budget reasoning without starving the answer
Reasoning controls depend on the model. The GLM-5.2 NIM uses chat_template_kwargs.enable_thinking to enable or disable thinking per request. Its documentation separates parsed choices[0].message.reasoning_content from final choices[0].message.content. Allow room for both when reasoning is enabled.
The same page gives a concrete failure and remedy: when finish_reason: "length" arrives before a final answer, increase max_tokens or disable thinking. A smaller cap may reduce generated work while also eliminating the answer. See 3.4.1 for termination checks. Do not transfer GLM’s control names or recommended behavior to another model without its documentation.
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B provides a graded alternative: reasoning_effort accepts "low", "medium", or "high". Low and medium select shorter, efficient reasoning; high or an omitted field selects full reasoning. Reasoning is on by default, its tokens count toward max_tokens, and the message returns reasoning in reasoning rather than GLM’s reasoning_content.
For this Nemotron NIM, do not use thinking_token_budget as a hard cap. When that budget is reached, generation continues to max_tokens, ending with finish_reason: "length" and empty content. Use reasoning_effort or chat_template_kwargs.enable_thinking instead. A shorter reasoning mode changes the inference behavior; a token ceiling can terminate it before the answer.
A reasoning budget is therefore a service decision as well as a request setting. Compare disabled, shorter, and full reasoning on the hard task subset, not only on short questions. Reserve the extra generation for a measurable quality gain that still fits the deadline. Increasing the budget without inspecting complete-answer rates can spend more tokens without improving delivery.
Key distinctions and trade-offs
Compare these choices as separate axes of one candidate:
| Axis | Deciding condition | Mechanism | Cost or failure risk |
|---|---|---|---|
| Model size | A candidate lacks the capability needed on hard cases | llms.<name>.model_name and a matching serving deployment | More capacity consumed without a guaranteed task-quality gain |
| Quantization | A supported lower-precision candidate frees needed capacity | NIM list-model-profiles and NIM_MODEL_PROFILE | Hardware or model incompatibility; possible quality regression |
| Reasoning budget | Additional inference improves the task enough to justify latency | GLM-5.2 chat_template_kwargs.enable_thinking; Nemotron reasoning_effort; max_tokens | Longer generation or no final answer when output space runs out |
A quantized larger model and a full-precision smaller model are distinct candidates. A model with thinking enabled is another candidate even if its weights and profile are unchanged. Hold the task set and target load fixed when comparing them.
Architecture / decision scenario
A manufacturing maintenance assistant has an on-premises pool of eight H100 80GB GPUs and must leave at least four GPUs available for other workloads. It requires at least 96% correct diagnostic recommendations and p95 below 4 seconds at 12 concurrent requests. The following illustrative deployment measurements compare candidates whose supported profiles have been confirmed.
A smaller BF16 candidate with thinking disabled uses two GPUs and reaches 93% accuracy with p95 at 1.4 seconds. A larger BF16 candidate, NVIDIA-Nemotron-3.5-Super-VL-120B-A12B, with full reasoning uses all eight GPUs, as its BF16 profile requires on H100 80GB, and reaches 98% with p95 at 6.2 seconds. The tempting proposal is to accept the largest model because it wins the accuracy column; that violates both the response target and the four-GPU reserve.
The same model’s NVFP4 profile uses two H100 80GB GPUs. With reasoning disabled, it reaches 96.5% and p95 at 3.1 seconds. Full reasoning raises accuracy to 97.4% but p95 to 4.8 seconds. A shorter-reasoning trial using the Nemotron reasoning_effort: "low" control reaches 97% with p95 at 3.7 seconds on the same two-GPU allocation. All three deliver complete answers on this benchmark. Reducing the full-reasoning trial’s token ceiling lowers p95 to 3.4 seconds but leaves 7% of answers incomplete, so that apparent latency fix is rejected.
The application owner selects the NVFP4 candidate with low reasoning effort: it improves accuracy over disabled reasoning while meeting the latency and capacity requirements. The platform owner records the exact NIM_MODEL_PROFILE and validates the two-GPU allocation under the same load. Full reasoning needs a separate latency allowance or more serving capacity; its higher score does not override the existing requirements.
Common exam traps
- Selecting the largest model without checking the task threshold and response deadline.
- Assuming every quantization recipe works on every GPU and model.
- Treating request-level
max_tokensas a startup precision or memory setting. - Confusing weight quantization with KV-cache quantization.
- Assuming reasoning text means a complete final answer was delivered.
- Treating Nemotron’s
thinking_token_budgetas a safe hard cap instead of using its documented reasoning controls.
What to remember
Select the smallest viable resource commitment that meets the complete task requirement under load. Verify the model, profile, hardware, and reasoning controls together. Quality, compatible deployment, and timely final answers must all survive the comparison.
Knowledge check
A contract-review assistant needs at least 94% correct findings and p95 at or below 3 seconds, and may use at most two GPUs. Every profile below is confirmed as supported on the target GPUs, and these are the team’s measurements under target load:
- Model X, FP8, thinking disabled, one GPU: 91%, 1.2 seconds
- Model Y, BF16, full reasoning, four GPUs: 97%, 3.8 seconds
- Model Y, FP8, low reasoning effort, two GPUs: 95%, 2.6 seconds
- Model Y, FP8, full reasoning, two GPUs: 96%, 3.4 seconds
Which candidate should the team select?
A team runs a model-specific LLM NIM on H100 GPUs and wants to free memory for more replicas. list-model-profiles for the exact image prints BF16 and FP8 profiles and no NVFP4 profile. Which step selects the lower precision?
NIM_MODEL_PROFILE to the full FP8 profile ID the listing printsNIM_MODEL_PROFILE to the BF16 profile ID with bf16 renamed nvfp4KvCacheConfig(dtype='fp8') in the NIM container’s environmentmax_tokens on each request sent to the endpointKvCacheConfig belongs to TensorRT LLM’s LLM API rather than NIM, and max_tokens is a request ceiling, not a precision. Re-evaluate task quality on the FP8 candidate before adopting it.On the current GPU count, list-model-profiles marks the validated BF16 profile as low memory and suggests --max-model-len=8192. The workload’s longest request, input plus output, is 6,000 tokens, and the team must keep both the validated profile and the current GPU count. Which change lets the profile run?
max_tokens on each request to fit the available memory--max-model-len at the suggested value--max-model-len that still covers the longest request is the listing’s own remedy. Higher tensor parallelism or quantized precision is NVIDIA’s suggestion for an incompatible profile whose weights alone exceed memory, and both would replace the validated profile; a request’s max_tokens is not a memory setting.A team serves the NVIDIA-Nemotron-3.5-Super-VL-120B-A12B NIM and needs shorter reasoning to meet a latency target. They set thinking_token_budget: 512, and responses now end with finish_reason: "length" and an empty content. Which change follows the model’s documentation?
thinking_token_budget until content returns againreasoning_effort to "low"max_tokens to stop generation within the deadlinereasoning fieldthinking_token_budget does not finish the answer: generation continues to max_tokens and content stays empty, so the documentation says to use reasoning_effort or enable_thinking instead, and "low" selects a shorter reasoning mode. A lower ceiling ends generation before the answer, and the reasoning field holds the model’s reasoning, not its final answer.A team serves a model through TensorRT LLM’s LLM API directly on H100 GPUs. FP8 KV cache is supported for this model and hardware. Long-context requests are limited by KV-cache memory; their input and output allowances must stay unchanged, as must the validated BF16 weights. Which change targets the constraint?
NIM_MODEL_PROFILE to an FP8 profile at container startupKvCacheConfig(dtype='fp8') when constructing the LLMmax_tokens on each request sent to the modelKvCacheConfig(dtype='fp8') , which leaves the validated weights alone; task quality is still re-evaluated afterwards. An NVFP4 weight recipe changes the weights and is not listed for Hopper, NIM_MODEL_PROFILE belongs to NIM rather than direct TensorRT LLM, and an output ceiling does not change how the cache is stored.References
- NVIDIA, Model Profiles and Selection
- NVIDIA, Get Started with NVIDIA-Nemotron-3.5-Super-VL-120B-A12B
- NVIDIA, Get Started with GLM-5.2
- NVIDIA, Quantization
- NVIDIA, NVIDIA NeMo Agent Toolkit Optimizer