MLC LLM
Universal LLM Deployment Engine with ML Compilation
MLC LLM scores 6.9/10 (Solid) in Inference & Serving.
Language Python License Apache-2.0 Stars 22.9k★ Type library Self-host Yes Repo mlc-ai/mlc-llm Homepage https://llm.mlc.ai
Quality
Solid 6.9
Adoption 59
Activity 51
Maturity 97
Community 59
Capability 100
Show the math
Overall 6.9/10
value = 6.9/10
score = value
Adoption 6.3/10
value = 59/100
score = 1 + 9 * value / 100
Activity 5.6/10
value = 50.9/100
score = 1 + 9 * value / 100
Maturity 9.7/10
value = 96.9/100
score = 1 + 9 * value / 100
Community 6.3/10
value = 58.9/100
score = 1 + 9 * value / 100
Capability 10/10
value = 100/100
score = 1 + 9 * value / 100
Key metrics
22.9k★ GitHub stars
12/90d Recent commits
6 devs/90d Recent contributors
3500 tok/s Decode throughput (Llama 3 8B, A100 80GB, 100 users) BentoML benchmark (Jun 2024): MLC-LLM reached ~3500 tok/s for 100 users on Llama 3 8B / A100 80GB (matches LMDeploy at low concurrency but trails it at very high load). Sources: https://www.bentoml.com/blog/benchmarking-llm-inference-backends and Reddit recap https://www.reddit.com/r/LocalLLaMA/comments/1da76ql/
3.2y Project age
active Last commit
Gotchas
No gotchas documented yet. Contribute one if you know a constraint we missed.
Labels
- Self-hosted
Self-hosted
You deploy and operate it yourself; there is no hosted option here.
compiler edge webgpu mobile cross-platform