vLLM
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM scores 9/10 (Exceptional) in Inference & Serving.
Language Python License Apache-2.0 Stars 85.3k★ Type service Self-host Yes Repo vllm-project/vllm Homepage https://docs.vllm.ai
Quality
Exceptional 9.0
Adoption 73
Activity 99
Maturity 98
Community 82
Capability 100
Show the math
Overall 9/10
value = 9/10
score = value
Adoption 7.6/10
value = 73.3/100
score = 1 + 9 * value / 100
Activity 9.9/10
value = 98.6/100
score = 1 + 9 * value / 100
Maturity 9.8/10
value = 97.8/100
score = 1 + 9 * value / 100
Community 8.4/10
value = 82.2/100
score = 1 + 9 * value / 100
Capability 10/10
value = 100/100
score = 1 + 9 * value / 100
Key metrics
85.3k★ GitHub stars
2.9k/90d Recent commits
67 devs/90d Recent contributors
~2400 tok/s Decode throughput (Llama 3 8B, A100 80GB, 100 users) BentoML benchmark (Jun 2024): vLLM decoding rate ~2300-2500 tok/s at 100 users on Llama 3 8B / A100 80GB; best-in-class TTFT across all loads. (vLLM's own blog separately claims up to 24x HF / 3.5x TGI throughput.) Source: https://www.bentoml.com/blog/benchmarking-llm-inference-backends
3.4y Project age
active Last commit
Gotchas
No gotchas documented yet. Contribute one if you know a constraint we missed.
Labels
- Self-hosted
Self-hosted
You deploy and operate it yourself; there is no hosted option here.
- Managed option
Managed option
Available as a hosted/managed service in addition to self-hosting.
serving gpu openai-compatible throughput pagedattention