TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
TensorRT-LLM scores 8.3/10 (Strong) in Inference & Serving.
Language Python License Custom (see repo) Stars 14k★ Type library Self-host Yes Repo NVIDIA/TensorRT-LLM Homepage https://nvidia.github.io/TensorRT-LLM
Quality
Strong 8.3
Adoption 54
Activity 97
Maturity 96
Community 61
Capability 100
Show the math
Overall 8.3/10
value = 8.3/10
score = value
Adoption 5.8/10
value = 53.7/100
score = 1 + 9 * value / 100
Activity 9.8/10
value = 97.4/100
score = 1 + 9 * value / 100
Maturity 9.6/10
value = 95.5/100
score = 1 + 9 * value / 100
Community 6.4/10
value = 60.5/100
score = 1 + 9 * value / 100
Capability 10/10
value = 100/100
score = 1 + 9 * value / 100
Key metrics
14k★ GitHub stars
1.7k/90d Recent commits
55 devs/90d Recent contributors
~2400 tok/s Decode throughput (Llama 3 8B, A100 80GB, 100 users) BentoML benchmark (Jun 2024, TRT-LLM v0.9.0): ~2300-2500 tok/s at 100 users on Llama 3 8B / A100 80GB (grouped with vLLM/TGI; on Llama 3 70B Q4 it matched LMDeploy's top throughput). LMSYS separately measured TRT-LLM up to ~5000 tok/s on short-input Llama3. Source: https://www.bentoml.com/blog/benchmarking-llm-inference-backends
2.9y Project age
active Last commit
Gotchas
No gotchas documented yet. Contribute one if you know a constraint we missed.
Labels
- No license
No OSS license
No recognized open-source license is declared; using it commercially is legally ambiguous.
- Self-hosted
Self-hosted
You deploy and operate it yourself; there is no hosted option here.
nvidia gpu optimization tensorrt production