LMDeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
LMDeploy scores 7.3/10 (Strong) in Inference & Serving.
Language Python License Apache-2.0 Stars 7.9k★ Type service Self-host Yes Repo InternLM/lmdeploy Homepage https://lmdeploy.readthedocs.io
Quality
Strong 7.3
Adoption 48
Activity 77
Maturity 96
Community 47
Capability 100
Show the math
Overall 7.3/10
value = 7.3/10
score = value
Adoption 5.3/10
value = 47.5/100
score = 1 + 9 * value / 100
Activity 7.9/10
value = 76.7/100
score = 1 + 9 * value / 100
Maturity 9.7/10
value = 96.3/100
score = 1 + 9 * value / 100
Community 5.2/10
value = 47/100
score = 1 + 9 * value / 100
Capability 10/10
value = 100/100
score = 1 + 9 * value / 100
Key metrics
7.9k★ GitHub stars
160/90d Recent commits
12 devs/90d Recent contributors
4000 tok/s Decode throughput (Llama 3 8B, A100 80GB, 100 users) BentoML cross-engine benchmark (Jun 2024): LMDeploy TurboMind delivered the best decoding token-generation rate, up to ~4000 tok/s at 100 concurrent users on Llama 3 8B / A100 80GB. Source: https://www.bentoml.com/blog/benchmarking-llm-inference-backends
3.1y Project age
active Last commit
Gotchas
No gotchas documented yet. Contribute one if you know a constraint we missed.
Labels
- Self-hosted
Self-hosted
You deploy and operate it yourself; there is no hosted option here.
- Managed option
Managed option
Available as a hosted/managed service in addition to self-hosting.
serving quantization turbomind vlm gpu