IndexMapAbout
Tool / RuntimeLLMCross-Modal

TensorRT-LLM

Also known as: TRT-LLM · NVIDIA TensorRT-LLM

Plain English

NVIDIA's open-source LLM serving library tuned specifically for NVIDIA GPUs, using custom low-level kernels and low-precision math to squeeze maximum token speed out of Hopper and Blackwell hardware.

Technical Overview

TensorRT-LLM pairs NVIDIA's fused CUDA/CUTLASS kernels, in-flight batching, paged KV caching, Speculative Decoding, and native FP8/NVFP4 execution with tensor and pipeline parallelism. Initially centered on ahead-of-time compiled TensorRT engine files, its 1.0 architecture adopted a modular PyTorch-native backend and high-level Python LLM API to simplify deployment alongside vLLM and SGLang.

Chronological Entries (2)

Direct Relations (6)