NVIDIANVIDIA-Certified Professional: Agentic AI2026-107.4 Leverage NVIDIA TensorRT-LLM and Triton Inference Server for latency reduction.
TensorRT-LLM User Guide
Completed by opening the official lab
Why it matters
Triton and TensorRT-LLM are separate layers with separate settings. Running them together once shows which file owns which decision.
What to practise
Start the tritonserver trtllm-python-py3 container, clone TensorRT-LLM, and set model: in triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml to a Hugging Face model ID. Launch with launch_triton_server.py and call /v2/models/tensorrt_llm/generate. Note that the PyTorch backend serves the checkpoint without compiling an engine, and that every key in model.yaml maps to an LLM API argument — KV cache, quantization, parallelism. The guide then links onward to scheduling, KV cache reuse, chunked context and the metrics endpoint.
How it fits
The split between the Triton model repository and model.yaml is 7.4.1’s division of serving from execution. The KV cache settings in model.yaml, and the scheduling and chunked-context pages the guide links to, are 7.4.2’s prefill and cache controls.