NVIDIANVIDIA-Certified Professional: Agentic AI2026-107.4 Leverage NVIDIA TensorRT-LLM and Triton Inference Server for latency reduction.
Separate Triton serving from model execution
Completed by the knowledge check
Learning objective
Serve TensorRT-LLM through Triton with the LLM API (PyTorch) backend, separating model execution from serving and scheduling.
Core idea
Triton Inference Server supplies the serving interface; TensorRT LLM supplies the model runtime behind it. For the LLM API integration, a Triton Python backend wrapper constructs tensorrt_llm.LLM with backend: "pytorch". The word backend therefore describes two different interfaces: Triton’s config.pbtxt says backend: "python", while the model’s 1/model.yaml selects TensorRT LLM’s PyTorch execution path.
This separation matters when a service responds slowly. Changing an HTTP response mode does not change how the runtime schedules tokens. Changing runtime concurrency does not supply the model’s chat template. You need to identify the layer that owns the observed delay before choosing its control. See 7.4.2 for token scheduling and cache allocation, and 7.4.3 for changes to numeric execution and drafting.
How it works
Load the runtime through a model repository
The documented repository layout is TensorRT-LLM/triton_backend/all_models/llmapi/. Its tensorrt_llm/config.pbtxt declares Triton’s model interface, including inputs such as text_input and optional sampling_param_* fields. Its tensorrt_llm/1/model.yaml selects the checkpoint through model, chooses backend: "pytorch", and carries runtime settings.
Keys outside triton_config map to LLM() constructor arguments. For example, tensor_parallel_size, max_seq_len, max_num_tokens, and kv_cache_config configure TensorRT LLM. The nested triton_config controls the Triton wrapper, including max_batch_size and decoupled. Keep these scopes explicit: the same identifier can mean different things. The LLM reference marks top-level max_batch_size and max_num_tokens stable, with defaults of 2048 requests and 8192 tokens respectively.
Use a repository release tag matching the installed tensorrt_llm version in the serving container. NVIDIA’s deployment guide obtains that version with python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" before selecting the repository tag. A working example from another release is insufficient proof that the wrapper and installed library agree on their constructor arguments.
The launcher is triton_backend/scripts/launch_triton_server.py, with --model_repo pointing to the llmapi/ directory. Run it from the parent of the cloned TensorRT-LLM/ directory. NVIDIA documents that launching inside the clone can import the source tree instead of the installed package and produce ModuleNotFoundError: No module named 'tensorrt_llm.bindings'. Correct the working directory before treating that failure as a GPU or checkpoint problem.
The PyTorch path serves supported Hugging Face checkpoints directly without a separate TensorRT engine compilation step. Model compatibility still matters; the serving guide qualifies this as supported models. A repository path does not establish that every architecture, precision, and parallel configuration works on the target hardware.
Translate the request without changing its meaning
The HTTP completion endpoint is /v2/models/tensorrt_llm/generate. It accepts text_input; sampling_param_max_tokens limits generated output, and sampling_param_exclude_input_from_output controls whether the returned text includes the prompt. These are request inputs, separate from runtime capacity settings such as max_num_tokens.
generate is a raw completion endpoint. It does not apply a chat template. The application adapter must format the prompt using the checkpoint’s tokenizer_config.json template. Sending a chat application’s internal message list as arbitrary text can produce a completed inference request whose answer has the wrong conversational behavior. Assign template selection to the adapter owner and test it alongside the checkpoint revision.
Streaming requires triton_config.decoupled: True. The client then calls /v2/models/tensorrt_llm/generate_stream with streaming: true. Decoupled mode permits multiple responses for one request; it does not establish faster model execution or a higher concurrency budget. Measure first-token delivery and complete-answer time separately, using the metric distinctions in 4.3.1.
Request sampling_param_return_perf_metrics: true to retrieve per-request runtime metrics. arrival_time_ns, first_token_time_ns, and last_token_time_ns expose runtime timing; compare their differences with client-observed first-token and completion times to distinguish execution from delivery delays. The same input returns cache and speculative-decoding statistics used in 7.4.2 and 7.4.3.
Keep serving batches separate from runtime scheduling
The shipped wrapper configuration sets triton_config.max_batch_size: 0; NVIDIA’s guide describes this as serving without batching. Do not add Triton dynamic batching settings to this template and assume they enable TensorRT LLM’s iteration scheduler.
The wrapper implementation schedules each incoming request asynchronously and passes its prompt to LLM.generate_async. Its source explicitly leaves batched prompts within one request unimplemented. That source-level distinction explains why a Triton request batching setting must not be treated as the runtime’s scheduling capacity: independent asynchronous requests and an input containing multiple batched prompts are different interfaces.
A streamed request crosses three layers: the application adapter formats its prompt, the Triton wrapper passes it to LLM.generate_async, and TensorRT LLM schedules its tokens.
Keep the shipped wrapper batching setting when adopting this path. Tune the runtime controls described in 7.4.2 against concurrent independent requests, and inspect the implementation shipped with your selected release before extending the wrapper. Multiple instances use instance_group and gpu_device_ids in config.pbtxt: semicolons separate per-instance GPU sets, while commas separate GPUs within a set. The number of sets must match the instance count. The launcher requires --no-mpi for multi-instance deployments. Each instance has its own runtime; reduce kv_cache_config.free_gpu_memory_fraction when instances share a GPU.
Key distinctions and trade-offs
Use the layer that owns the change:
| Decision | Mechanism | Benefit | Cost or boundary |
|---|---|---|---|
| Expose inference over Triton | Model repository and config.pbtxt | Stable serving interface around execution | Wrapper must match the runtime release |
| Select checkpoint and execution | model, backend: "pytorch" | Load a supported checkpoint directly | Model and hardware compatibility still constrain it |
| Deliver partial output | triton_config.decoupled, generate_stream, streaming | Earlier visible progress | Multiple response handling in the client |
| Schedule runtime work | Top-level max_batch_size, max_num_tokens | Control concurrent model execution | GPU memory and latency trade-offs; see 7.4.2 |
A wrapper configured without Triton batching can still accept independent requests that enter an asynchronous model runtime. Conversely, a client that submits a large batched tensor needs explicit wrapper support. Neither property follows from the word batching alone.
Architecture / decision scenario
An insurance company operates a claims assistant on an existing GPU pool. It targets a p95 first visible token below 900 ms and a complete response below 5 seconds at 24 concurrent requests. These are project requirements, not product benchmarks. Its application team proposes enabling streaming; its platform team proposes increasing Triton’s batching setting to 24.
The serving owner keeps triton_config.max_batch_size: 0 for the documented wrapper and enables decoupled: True for the streaming endpoint. The application owner applies the checkpoint’s chat template and handles multiple streamed responses. The 24-request target needs no batching change at either layer: it is below the runtime’s default request limit of 2048. The inference owner tests the token budget and cache capacity described in 7.4.2; that request ceiling alone does not prove the latency target is achievable.
If streaming reduces visible delay from 1,100 ms to 700 ms but full completion remains 6 seconds in the team’s test, only the first requirement passes. The team keeps the interface improvement and investigates runtime scheduling separately. It rejects a release justified solely by the first-token measurement because the complete-response constraint still fails.
Common exam traps
- Confusing Triton’s Python backend with TensorRT LLM’s PyTorch execution backend.
- Treating
sampling_param_max_tokensas the runtime’s batch token budget. - Assuming the raw
generateendpoint supplies a chat template. - Enabling streaming without decoupled mode.
- Reading
triton_config.max_batch_size: 0as proof that the runtime cannot schedule independent requests.
Where the exam guide differs
The study guide suggests “Batchers—NVIDIA Triton Inference Server.” In the LLM API path, the shipped wrapper sets triton_config.max_batch_size: 0, described as serving requests without batching; request interleaving comes from TensorRT LLM’s in-flight batching (7.4.2). If an exam option proposes enabling Triton dynamic batching for this path, distinguish it from the runtime levers max_batch_size and max_num_tokens. NVIDIA’s deployment guide also marks the convert_checkpoint.py plus trtllm-build engine-build workflow and inflight_batcher_llm ensemble layout as deprecated and being removed. Do not transfer their settings into model.yaml.
What to remember
Separate the Triton contract, the wrapper, and TensorRT LLM execution. Match releases, format raw completion prompts correctly, enable decoupled mode for streaming, and tune runtime scheduling in its own configuration scope.
Knowledge check
An engineer deploys a model through Triton’s TensorRT LLM API integration and must run it on TensorRT LLM’s PyTorch execution path. A colleague proposes changing backend: "python" in the model’s config.pbtxt to "pytorch". Which configuration is correct?
backend: "python" in config.pbtxt , and top-level backend: "pytorch" in 1/model.yamlbackend: "pytorch" in config.pbtxt , and no backend key in 1/model.yamlbackend: "pytorch" in config.pbtxt , and top-level backend: "pytorch" in 1/model.yamlbackend: "python" in config.pbtxt , and backend: "pytorch" under triton_config in 1/model.yamlconfig.pbtxt declares; the wrapper then constructs tensorrt_llm.LLM , and the top-level backend: "pytorch" in model.yaml selects TensorRT LLM’s execution path. Keys under triton_config configure the wrapper, not LLM() .A client calls /v2/models/tensorrt_llm/generate_stream with streaming: true against the Triton LLM API model, but the deployment does not stream. Its model.yaml contains triton_config.decoupled: False. Which change enables streaming?
triton_config.max_batch_size above 0 so Triton can batch responsesstreaming: true to the generate endpoint instead of generate_streamtriton_config.decoupled: True and keep the generate_stream endpointdecoupled: True as a top-level key beside the LLM() argumentstriton_config . Top-level keys in model.yaml map to LLM() constructor arguments, so the same name there configures the wrong layer; batching and the endpoint choice do not enable multiple responses.A chat application serializes its message list as JSON and sends it as text_input to /v2/models/tensorrt_llm/generate. Requests complete successfully, but the answers ignore the system instructions and often continue the JSON. Which change addresses the cause?
sampling_param_max_tokens so the model can finish its answergenerate_streamsampling_param_exclude_input_from_output so the prompt is not echoed backgenerate is a raw completion endpoint and applies no chat template, so the model sees arbitrary text rather than a conversation. The adapter must format the messages with the template from the checkpoint’s tokenizer_config.json . Output length, streaming and echo control change how the answer is returned, not how the model reads the prompt.An engineer clones TensorRT-LLM/ at the release tag that matches the installed tensorrt_llm version, then runs python3 triton_backend/scripts/launch_triton_server.py from inside the clone. The launch fails with ModuleNotFoundError: No module named 'tensorrt_llm.bindings', although import tensorrt_llm succeeds elsewhere in the same container. What should the engineer do first?
TensorRT-LLM/ clonetensorrt_llm.bindings . Launching from the parent directory resolves the installed package. The release tag already matches, and the error names an import, not a GPU or checkpoint fault.A claims assistant served through the Triton LLM API path must handle 24 concurrent independent requests through one runtime instance using the shipped wrapper. The platform team proposes adding Triton dynamic batching and raising triton_config.max_batch_size from 0 to 24 so that requests share GPU execution. What should the serving owner do?
triton_config.max_batch_size to 24 and add dynamic batching to config.pbtxttriton_config.max_batch_size: 0 and tune the runtime’s top-level limitsinstance_group , each serving one requestmax_batch_size and max_num_tokens . The wrapper does not implement batched prompts within one request, and every extra instance carries its own runtime and KV cache.