NVIDIANVIDIA-Certified Professional: Agentic AI2026-107.4 Leverage NVIDIA TensorRT-LLM and Triton Inference Server for latency reduction.
Balance prefill work against active decoding
Completed by the knowledge check
Learning objective
Explain in-flight batching, paged KV cache, and chunked context, and configure them for latency.
Core idea
A request waiting to process a long prompt competes with requests already generating answers. TensorRT LLM coordinates that competition through iteration-level scheduling, a bounded token budget, and a pool of key-value (KV) cache blocks. Chunked context, also called chunked prefill, lets the scheduler process part of a prompt alongside ongoing generation.
These mechanisms solve different problems. In-flight batching keeps execution useful as requests arrive and finish. Paged KV cache assigns memory as sequences need it. Chunked prefill divides prompt work across iterations. Their combination can improve latency under mixed traffic, but a larger batch or cache allocation does not guarantee a better tail. See 4.3.1 for metric definitions and 7.4.1 for the Triton wrapper that exposes this runtime.
How it works
Admit work at each iteration
In-flight batching is also called continuous or iteration-level batching. TensorRT LLM can process sequences in the context phase together with sequences in the generation phase. Requests do not all need to reach their final output token before the runtime can schedule other work.
Three runtime limits answer different questions. max_batch_size bounds the requests scheduled together. max_num_tokens bounds the packed tokens processed in a batch. max_seq_len bounds one sequence. A large value for one does not remove the others. NVIDIA’s scheduler walkthrough prioritizes generation-phase requests first, each consuming one token of the iteration’s max_num_tokens budget in its illustrated single-token decode path. Prompt work then uses the remaining budget, so request count alone conceals a mixed batch’s cost.
In the LLM API/PyTorch path, these are top-level LLM() arguments and therefore top-level keys in model.yaml. They are separate from triton_config.max_batch_size, which the shipped Triton wrapper sets to zero. Keep that setting’s wrapper support boundary from 7.4.1; do not try to enable runtime in-flight batching by editing a Triton dynamic-batching policy.
Increasing max_num_tokens can improve GPU utilization by making larger matrix operations possible. It also affects workspace allocation: an excessively large token budget takes memory away from the KV cache needed by concurrent sequences. NVIDIA documents that excessive values can harm first-token and end-to-end latency after utilization saturates. Choose an operating point from the workload’s latency measurements rather than interpreting the maximum supported value as a recommendation.
Allocate KV state in pages
The KV cache retains attention keys and values computed for preceding tokens, avoiding repeated computation during generation. A paged cache divides that state into blocks assigned to requests as needed. It avoids reserving each request’s full possible sequence length in one monolithic allocation.
kv_cache_config.free_gpu_memory_fraction controls the fraction of free GPU memory allocated to the cache. The documented default is 0.9; it is a fraction of free memory, not 90% of the GPU’s entire installed memory. If you also set kv_cache_config.max_tokens, the runtime uses the smaller allocation implied by the two limits. This max_tokens is cache capacity, distinct from max_num_tokens for one execution batch and from a request’s output ceiling.
A cache allocation that crowds out other runtime needs is not useful capacity. For multiple runtime instances sharing a GPU, NVIDIA’s Triton LLM API guide explicitly recommends reducing kv_cache_config.free_gpu_memory_fraction so their allocations fit together. Start with a measured allocation for the intended deployment, then test the longest allowed sequences under concurrency. A single short request cannot demonstrate adequate KV capacity.
Paging and prefix reuse are separate mechanisms. Paging assigns blocks; kv_cache_config.enable_block_reuse controls reuse across requests and is enabled by default in the KV-cache documentation. Reuse finds matching token prefixes and can save their computation. Unrelated prompts still need their own state, and reusable blocks can be evicted. Budget for a cold or low-reuse traffic mix rather than assuming every request receives a cache hit. Observe kv_cache_hit_rate using the request-level performance metrics described in 7.4.1. KV-aware routing between workers belongs to 4.4.2.
Divide prefill instead of truncating it
NVIDIA recommends always enabling chunked context: it prevents large prompts from being excluded by the remaining iteration budget, mitigates queuing and worst-case first-token latency, and permits smaller batch token budgets that leave more memory for KV cache. Set enable_chunked_prefill: true deliberately: the LLM API reference lists its default as False. Chunking processes a prompt in portions while preserving the prompt’s contents. It does not summarize, discard, or shorten the allowed context.
Without chunking, a prompt larger than the token budget remaining in an iteration can wait even when some budget is unused. Splitting it lets a portion fit with decoding requests. NVIDIA’s scheduler explanation shows how this can reduce queuing and worst-case time to first token, while allowing a smaller max_num_tokens than the longest supported prompt.
Within one iteration, decoding requests take their tokens first, and a prompt larger than the remaining budget can wait unless chunked context lets a portion fit.
The recommendation does not replace workload validation. NVIDIA describes minimal worst-case performance impact and potential significant benefits. Keep chunking enabled while testing both traffic classes and choosing their token and cache budgets. The documented paged-attention execution mechanism requires non-final context chunks to be multiples of the cache block size; the runtime manages that execution constraint.
Key distinctions and trade-offs
Choose the mechanism by the pressure you observe:
| Mechanism | Deciding condition | Configuration or identifier | Cost or limit |
|---|---|---|---|
| In-flight batching | Different requests finish and arrive at different times | Runtime max_batch_size, max_num_tokens | More simultaneous work can raise per-request latency |
| Paged KV cache | Concurrent sequences need growing attention state | kv_cache_config.free_gpu_memory_fraction, max_tokens | Cache capacity consumes memory available to other work |
| Prefix reuse | Requests share identical token prefixes | enable_block_reuse | Benefit depends on matching prefixes and eviction |
| Chunked context | Long prefill delays other schedulable work | enable_chunked_prefill | Non-final chunks must align to the cache block size |
Do not reduce max_seq_len to imitate chunking: that changes the accepted sequence budget. An oversized execution token budget consumes workspace that could otherwise support KV cache. These are coupled allocations with different purposes.
Architecture / decision scenario
A manufacturing analytics assistant shares an inference deployment between engineers asking short equipment questions and research jobs supplying long maintenance histories. Its project workload has 80% 600-token prompts and 20% 12,000-token prompts, with up to 32 active requests. Short requests require p95 first-token latency below 800 ms and p95 inter-token latency below 60 ms. The inference team owns these requirements and the serving configuration.
The team enables enable_chunked_prefill: true as its baseline and tests runtime max_batch_size: 32. The tempting proposal raises max_num_tokens above its 8192 default to at least 12,000 plus active decode tokens, enough to process a long prompt in one iteration. That workspace budget takes memory from the KV cache needed by 32 sequences. The team compares it with a trial max_num_tokens: 4096. These values are experiment settings, not guaranteed capacity. It selects a supported max_seq_len that covers each prompt plus its permitted output and tests the chosen checkpoint on its actual hardware.
For this experiment, the team also compares cache fractions of 0.7 and 0.9 under the same request mix, recording peak memory, latency, and kv_cache_hit_rate (see 7.4.1) with prefix reuse enabled and with reuse disabled. If the larger execution budget lowers long-prompt first-token time but pushes short-request inter-token latency from 45 ms to 90 ms, it fails the interactive requirement. A chunked candidate that remains below both interactive thresholds earns further testing only if it also fits the cache allocation under long sequences. This decision exercises scheduling, prompt partitioning, and memory capacity together.
Common exam traps
- Equating request count, per-iteration tokens, sequence length, and cache token capacity.
- Reading
free_gpu_memory_fractionas a fraction of total installed GPU memory. - Treating paging as a promise of prefix reuse for unrelated prompts.
- Assuming chunked context removes prompt tokens or is enabled by default because NVIDIA recommends it.
- Raising the batch token budget without checking decode latency and workspace allocation.
What to remember
Use in-flight batching to interleave requests, paged KV cache to allocate attention state, and chunked prefill to share execution time with long prompts. Tune request, token, sequence, and cache limits together against the latency targets of each traffic class.
Knowledge check
A TensorRT LLM deployment serves short interactive prompts alongside 6,000-token research prompts. Under load, research requests wait many iterations before their first token while short requests keep decoding, even though part of each iteration’s token budget goes unused. max_num_tokens is at its 8,192 default, the KV cache already uses the memory the team can give it, and long prompts must still be accepted in full. model.yaml does not mention enable_chunked_prefill. Which change fits?
max_num_tokens to cover the longest prompt plus active decode tokensenable_chunked_prefill: true in the runtime configurationmax_seq_len so that long prompts fit the remaining iteration budgetFalse , so it must be set. A larger token budget takes memory from a KV cache that has none to spare, and a lower max_seq_len changes what the deployment accepts.Two TensorRT LLM runtime instances must share one GPU through the Triton LLM API path’s instance_group, and the launcher already uses --no-mpi. The first instance starts, but the second cannot allocate its KV cache. Both use the default kv_cache_config. Model and workspace allocations must stay unchanged. Which change directly reduces each instance’s KV-cache memory allocation?
max_num_tokens on both instances and keep the default cache fractionkv_cache_config.max_tokens above each instance’s current cache capacitytriton_config.max_batch_size: 1 on each instance to cap its cache demandkv_cache_config.free_gpu_memory_fraction on each instancemax_tokens does not lower the allocation capped by the fraction, and the wrapper’s batching setting does not size the runtime’s cache.A team running mixed traffic with chunked prefill enabled raised max_num_tokens from 8,192 to 16,384. Long-prompt first-token latency improved, but p95 inter-token latency for short requests rose from 45 ms to 90 ms, above its 60 ms limit. Both traffic classes must meet their targets. What should the team try next?
max_num_tokens again, with chunked prefill on, and re-measure both classeskv_cache_config.free_gpu_memory_fraction to restore the lost cache capacitymax_batch_size so more decoding requests share each iterationenable_block_reuse so the larger token budget gets more GPU memoryA deployment sized its KV cache assuming every request reuses a 3,000-token shared prefix, with enable_block_reuse left at its default. A new nightly workload sends unrelated documents through the same deployment, and the team must keep concurrent sequences from running out of cache. How should the team size the cache?
kv_cache_hit_ratekv_cache_hit_rate keeps capacity honest; the same number of active requests can require more cache when their prefixes differ.References
- NVIDIA, Paged Attention, IFB, and Request Scheduling
- NVIDIA, KV Cache System
- NVIDIA, LLM
- NVIDIA, KvCacheConfig
- NVIDIA, End to end workflow to use the pytorch LLMAPI workflow