Text Generation Inference
Large Language Model Text Generation Inference
Text Generation Inference scores 5.3/10 (Solid) in Inference & Serving.
Language Python License Apache-2.0 Stars 10.9k★ Type service Self-host Yes Repo huggingface/text-generation-inference Homepage https://huggingface.co/docs/text-generation-inference
Quality
Solid 5.3
Adoption 51
Activity 11
Maturity 91
Community 53
Capability 100
Show the math
Overall 5.3/10
value = 5.3/10
score = value
Adoption 5.6/10
value = 50.9/100
score = 1 + 9 * value / 100
Activity 2/10
value = 11/100
score = 1 + 9 * value / 100
Maturity 9.2/10
value = 90.9/100
score = 1 + 9 * value / 100
Community 5.8/10
value = 53.4/100
score = 1 + 9 * value / 100
Capability 10/10
value = 100/100
score = 1 + 9 * value / 100
Key metrics
10.9k★ GitHub stars
0/90d Recent commits
0 devs/90d Recent contributors
~2400 tok/s Decode throughput (Llama 3 8B, A100 80GB, 100 users) BentoML benchmark (Jun 2024, TGI v2.0.4): TGI decoding rate in the ~2300-2500 tok/s band at 100 users on Llama 3 8B / A100 80GB (explicitly grouped with vLLM and TRT-LLM). Source: https://www.bentoml.com/blog/benchmarking-llm-inference-backends
3.7y Project age
3mo ago Last commit
Gotchas
No gotchas documented yet. Contribute one if you know a constraint we missed.
Labels
- Archived
Archived
The repository is archived/read-only — no longer maintained.
- Self-hosted
Self-hosted
You deploy and operate it yourself; there is no hosted option here.
- Managed option
Managed option
Available as a hosted/managed service in addition to self-hosting.
serving huggingface gpu production tensor-parallel