NVIDIA released TensorRT-LLM 1.0, making its PyTorch-native backend and stable high-level Python LLM API the default workflow. Removing the mandatory offline engine compilation step lowered operational friction while retaining Blackwell NVFP4 and disaggregated serving optimizations.
TensorRT-LLM
Also known as: TRT-LLM · NVIDIA TensorRT-LLM
Plain English
NVIDIA's open-source LLM serving library tuned specifically for NVIDIA GPUs, using custom low-level kernels and low-precision math to squeeze maximum token speed out of Hopper and Blackwell hardware.
Technical Overview
TensorRT-LLM pairs NVIDIA's fused CUDA/CUTLASS kernels, in-flight batching, paged KV caching, Speculative Decoding, and native FP8/NVFP4 execution with tensor and pipeline parallelism. Initially centered on ahead-of-time compiled TensorRT engine files, its 1.0 architecture adopted a modular PyTorch-native backend and high-level Python LLM API to simplify deployment alongside vLLM and SGLang.
Chronological Entries (2)
NVIDIA publicly open-sourced TensorRT-LLM on GitHub, providing a Python API to compile Transformer models into optimized TensorRT engines with custom FlashAttention kernels, in-flight batching, tensor parallelism, and FP8/INT4 Quantization for Hopper and Ampere GPUs.