Speculative Decoding with TensorRT-LLM

Why it matters

Speculative decoding can multiply token throughput or slow serving down, depending on how often the target accepts the draft. This tutorial makes that acceptance visible.

What to practise

Configure speculative_config in the LLM API model.yaml with decoding_type: Eagle3, max_draft_len and an EAGLE-3 speculative_model, serve it through Triton, and request sampling_param_return_perf_metrics to read acceptance_rate and the draft-token counts. Benchmark the base and speculative configurations with GenAI-Perf on HumanEval at --concurrency 1. Then read the Draft_Target variant and its warning: a draft model with a different tokenizer drives acceptance down and performance below the baseline. MEDUSA works only with the legacy engine backend.

How it fits

This is 7.4.3’s drafting trial: measure acceptance and speedup on your own workload rather than the tutorial’s single-GPU numbers. Keep any quantization change in a separate run.