NVIDIA /TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

★ 15,079⑂ 2,936👁 119⑃ 17,811● 4,252▤ 96
PythonNOASSERTIONCreated 2023-08-16nvidia.github.io/TensorRT-LLM
下载:v1.2.1v1.2.1 ↓1.2.12026-04-20
+4
日增长
+77
周增长
+277
月增长
15,079
stars
2,936
forks
119
watchers
17,811
prs
4,252
issues
96
releases

README

TensorRT LLM
===========================
TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework that enables you to customize and extend the system.










Architecture   |   Performance   |   Examples   |   Documentation   |   Roadmap

---


 Tech Blogs



 [11/11] Dense DWDP: Extending Distributed Weight Data Parallelism to Dense Weights on Vera Rubin NVL72
✨ ➡️ link

 [11/03] ISER-MoE: In-Switch Expert Replication for Load-Balanced MoE Prefill
✨ ➡️ link

 [10/14] KV Cache Compression in TensorRT LLM
✨ ➡️ link

 [09/22] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K
✨ ➡️ link

 [09/02] Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM
✨ ➡️ link

 [08/20] Evaluating Agentic Serving with Trace Replay and Job-Level Metrics
✨ ➡️ link

 [07/17] DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM
✨ ➡️ link

 [07/01] Scaling Video Generation Across NVL72 Rack with TensorRT-LLM
✨ ➡️ link

 [05/15] Joint Optimization of Agent Applications and TensorRT-LLM
✨ ➡️ link

 [04/03] Tuning CUDA Graph Batch Sizes for Higher Output Throughput
✨ ➡️ link

 [04/03] DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
✨ ➡️ link


 [03/16] Optimizing MoE Communication with One-Sided AlltoAll Over NVLink
✨ ➡️ link

 [03/04] Sparse Attention in TensorRT LLM
✨ ➡️ link

 [02/06] Accelerating Long-Context Inference with Skip Softmax Attention
✨ ➡️ link

 [01/09] Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs
✨ ➡️ link


Previous Blogs
 [10/13] Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary)
✨ ➡️ link

 [09/26] Inference Time Compute Implementation in TensorRT LLM
✨ ➡️ link

 [09/19] Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly
✨ ➡️ link

 [08/29] ADP Balance Strategy
✨ ➡️ link

 [08/05] Running a High-Performance GPT-OSS-120B Inference Server with TensorRT LLM
✨ ➡️ link

 [08/01] Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization)
✨ ➡️ link

 [07/26] N-Gram Speculative Decoding in TensorRT LLM
✨ ➡️ link

 [06/19] Disaggregated Serving in TensorRT LLM
✨ ➡️ link

 [06/05] Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP)
✨ ➡️ link

 [05/30] Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers
✨ ➡️ link

 [05/23] DeepSeek R1 MTP Implementation and Optimization
✨ ➡️ link

 [05/16] Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs
✨ ➡️ link


 Latest News
 [04/03] 🎨 TensorRT LLM now supports diffusion models for visual generation ➡️ link


Previous News

Top issues

Discussions

Latest release

v1.2.1 v1.2.1 2026-04-20
Highlights

- Fixed Issue
  - Fixed an issue that caused KV cache corruption (12770)

- Infrastructure Changes
  - Upgraded xgrammar and flashinfer (12811)

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON