Test precision and drafting changes separately

Learning objective

Apply quantization and speculative decoding, and evaluate their accuracy impact.

Core idea

Quantization changes the numeric representation of weights, activations, or cached attention state. Speculative decoding changes how candidate output tokens are proposed and verified. They address different costs and need separate experiments before you combine them.

TensorRT LLM can load Model Optimizer quantized checkpoints and configure speculation through speculative_config. Neither a supported precision nor a successful draft acceptance proves that an agent still chooses the correct tool or produces an acceptable answer. Preserve the model quality comparisons from 3.4.2; this lesson adds the runtime mechanisms, compatibility checks, and experiments needed to attribute an optimization’s effects.

How it works

Apply a supported numeric representation

TensorRT LLM’s quantization documentation lists recipes including FP8, NVFP4, and weight-only formats such as W4A16 AWQ. Read both the model support matrix and the hardware support matrix before selecting one. For example, the hardware matrix lists NVFP4 for Blackwell but not Hopper, while FP8 per-tensor support includes Hopper. A format appearing in the overall recipe list is not a promise for every GPU and architecture.

One application path loads a pre-quantized Model Optimizer checkpoint through LLM(model=...), or the corresponding model key in Triton’s model.yaml. Keep the selected checkpoint and its quantization metadata with the release. Setting a generic floating-point dtype is not a substitute for producing a calibrated quantized checkpoint.

For your own checkpoint, Model Optimizer’s post-training quantization example imports modelopt.torch.quantization as mtq and invokes mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG, forward_loop). The forward_loop runs calibration data to determine quantization scaling factors. NVIDIA describes a typical calibration set of 128–512 samples; that is an example range, not a quality guarantee. Export it with modelopt.torch.export.export_hf_checkpoint(model, export_dir) to a Unified Hugging Face checkpoint, then load that directory through model in the PyTorch path. This supplies the checkpoint for low-precision deployment; the legacy TensorRT-LLM checkpoint format supports the C++ backend only. Model Optimizer distinguishes simulated quantization used during optimization from the deployed low-precision representation.

Use representative calibration inputs, then evaluate on held-out cases. Reusing the calibration examples as the sole acceptance set gives you little information about behavior on unseen requests. Model Optimizer warns that post-training quantization accuracy loss depends on the model and method. Its documented remedies include disabling KV-cache quantization, using quantization-aware training, or restricting NVFP4 to selected layers. For example, mtq.NVFP4_MLP_ONLY_CFG leaves attention layers unquantized while applying NVFP4 to MLP layers.

Separate KV precision from weight precision

kv_cache_config.dtype: "fp8" selects FP8 KV storage through KvCacheConfig. TensorRT LLM documents enabling it even when a checkpoint does not enable it by default. This changes the cache representation, rather than declaring the checkpoint’s weights to be FP8.

NVFP4 KV cache has a different prerequisite: TensorRT LLM requires offline Model Optimizer quantization, and its quantization page limits this path to FP8 weight/activation quantization. The hardware matrix also distinguishes Blackwell SM100/103 from SM120: NVFP4 KV cache is listed for the former, not the latter. A generic claim that “Blackwell supports FP4” is too broad for this choice.

Test weight/activation precision and KV precision independently. If long-context accuracy falls only after changing cache precision, restoring the cache’s prior type is a more targeted response than abandoning every weight-quantization option. Keep the cache capacity experiment from 7.4.2 separate from this numeric comparison.

Propose drafts and verify them

Speculative decoding uses a lightweight drafting mechanism to propose tokens, then verifies candidates with the target model in one forward pass. Accepted tokens reduce the number of sequential target passes. Extra draft work and verification overhead still consume resources.

For draft/target speculation, configure DraftTargetDecodingConfig with speculative_model and max_draft_len, then pass it as LLM(..., speculative_config=...). The documented example also sets disable_overlap_scheduler=True, a beta LLM argument; follow that example when reproducing it, without treating the setting as a universal requirement. NVIDIA warns that draft and target must have been trained with the same tokenizer; otherwise acceptance is extremely low and performance regresses. A small drafter with incompatible tokenization is therefore a poor choice even if it fits easily in memory.

EAGLE 3 uses Eagle3DecodingConfig, also with speculative_model and max_draft_len, and needs a compatible draft checkpoint. The documentation accepts decoding_type: Eagle as an alias for Eagle3, but EAGLE v1/v2 checkpoints are incompatible with that PyTorch path. Do not interpret this alias as checkpoint conversion.

The speculation page states that speedups are only observable at low batch sizes in the described implementation. Speculation generates a draft sequence for every request and cannot be dynamically disabled per request. Compare low and high concurrency explicitly before enabling it for a shared traffic mix. More draft tokens can mean more accepted progress, but also more wasted work when acceptance is poor. Observe per-request acceptance_rate through the Triton performance metrics described in 7.4.1 to distinguish poor drafting from load-related overhead.

Acceptance rules matter for accuracy. MTPDecodingConfig.use_relaxed_acceptance_for_thinking permits a draft token in the thinking phase to be accepted from a candidate set controlled by relaxed_topk and relaxed_delta. This differs from a strict match. Do not claim universal output equivalence for every configuration called speculative decoding; retain quality tests and record whether relaxed acceptance was enabled.

Key distinctions and trade-offs

Compare each option against its own baseline:

For a general accuracy smoke test, NVIDIA provides trtllm-eval --model CHECKPOINT_PATH mmlu; replace CHECKPOINT_PATH with the checkpoint under test. The tool uses the offline LLM API and is intended for development, not production use. Its result supplements your agent task tests; a multiple-choice score does not validate tool arguments or authorized business actions.

Architecture / decision scenario

A retail company generates product descriptions and proposes catalog updates through an agent. Its fixed Blackwell (SM100/103) deployment must handle both four concurrent editorial sessions and a nightly workload of 64 concurrent jobs. The product owner requires at least 98% valid structured results, no more than a one-percentage-point loss in reviewed factual correctness, and a p95 editorial response below 3 seconds. These are project acceptance criteria.

The inference team proposes a combined quantized-and-speculative release. The evaluation owner instead requests four matched runs: the baseline checkpoint without speculation, a supported quantized checkpoint alone, the baseline with compatible draft/target speculation, and their combination. Within the quantized trial, KV-cache precision is a separately recorded factor; speculation runs also record acceptance_rate (see 7.4.1). Prompts, sampling, output ceilings, and task cases remain fixed; generated tool proposals are checked using the evaluation approach in 2.6.1.

Suppose the quantized candidate reduces editorial latency from 3.4 to 2.8 seconds but loses two percentage points of factual correctness. It fails quality despite meeting latency. The team tries a less aggressive recipe or restores cache precision according to the failing slice. If speculation alone reaches 2.9 seconds at concurrency four but regresses nightly throughput at 64, the serving owner rejects enabling it indiscriminately for the shared workload. Separate deployments are a possible capacity decision, requiring their own memory budget and validation; per-request dynamic disabling cannot be assumed.

Common exam traps

• Treating a supported precision as universal across GPU families and model architectures.
• Confusing FP8 KV storage with FP8 model weights.
• Assuming simulated quantization alone delivers deployment speedups.
• Choosing a draft model without tokenizer compatibility.
• Equating higher acceptance or faster decoding with unchanged task accuracy.
• Accepting a combined optimization without identifying which change caused a regression.

What to remember

Check model and hardware support, retain calibration and checkpoint details, and measure quantization separately from speculation. Evaluate strict versus relaxed acceptance deliberately, and accept an optimization only when task quality and the relevant concurrency regime both pass.

Knowledge check

References

• NVIDIA, Quantization
• NVIDIA, Speculative Decoding
• NVIDIA, Model Optimizer quantization
• NVIDIA, Model Optimizer post-training quantization
• NVIDIA, trtllm-eval
• NVIDIA, End to end workflow to use the pytorch LLMAPI workflow
• NVIDIA, LLM