2,002 offers
Free Inference & Serving — open-source & self-hostable
Run & serve LLMs locally or on your own hardware
Scores rate free-tier value on a 1–10 scale. How scores work →
-
vLLM Inference & Serving 9.0 A high-throughput and memory-efficient inference and serving engine for LLMs 85.3k★2.9k/90d67 devs/90d -
llama.cpp Inference & Serving 8.6 LLM inference in C/C++ 119.2k★1.2k/90d49 devs/90d -
Ray Serve Inference & Serving 8.3 Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads. 43.1k★1.1k/90d47 devs/90d -
TensorRT-LLM Inference & Serving 8.3 TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way. 14k★1.7k/90d55 devs/90d -
LocalAI Inference & Serving 8.0 LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required. 47.3k★1k/90d10 devs/90d -
Ollama Inference & Serving 7.8 Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. 175.4k★246/90d16 devs/90d -
KoboldCpp Inference & Serving 7.6 Run GGUF models easily with a KoboldAI UI. One File. Zero Install. 10.9k★1.6k/90d43 devs/90d -
Jan Inference & Serving 7.5 Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. 43.4k★276/90d7 devs/90d -
LMDeploy Inference & Serving 7.3 LMDeploy is a toolkit for compressing, deploying, and serving LLMs. 7.9k★160/90d12 devs/90d -
Xinference Inference & Serving 7.1 Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API. 9.4k★239/90d16 devs/90d -
KServe Inference & Serving 7.1 Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes 5.6k★176/90d33 devs/90d -
llamafile Inference & Serving 6.9 Distribute and run LLMs with a single file. 25.2k★40/90d8 devs/90d -
MLC LLM Inference & Serving 6.9 Universal LLM Deployment Engine with ML Compilation 22.9k★12/90d6 devs/90d -
KTransformers Inference & Serving 6.9 A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations 17.4k★62/90d21 devs/90d -
Outlines Inference & Serving 6.9 Structured Outputs 14.4k★55/90d20 devs/90d -
GPUStack Inference & Serving 6.9 A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment. 5.3k★462/90d9 devs/90d -
llm-d Inference & Serving 6.9 Achieve state of the art inference performance with modern accelerators on Kubernetes 3.7k★518/90d45 devs/90d -
Triton Inference Server Inference & Serving 6.8 The Triton Inference Server provides an optimized cloud and edge inferencing solution. 10.8k★66/90d9 devs/90d -
RamaLama Inference & Serving 6.5 RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers. 2.9k★298/90d10 devs/90d -
LightLLM Inference & Serving 6.3 LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. 4.2k★68/90d8 devs/90d -
CTranslate2 Inference & Serving 6.2 Fast inference engine for Transformer models 4.6k★21/90d6 devs/90d -
nano-vLLM Inference & Serving 6.0 Nano vLLM 14.3k★17/90d5 devs/90d -
Guidance Inference & Serving 5.9 A guidance language for controlling large language models. 21.5k★5/90d2 devs/90d -
XGrammar Inference & Serving 5.9 Fast, Flexible and Portable Structured Generation 1.8k★61/90d12 devs/90d -
Text Embeddings Inference Inference & Serving 5.6 A blazing fast inference solution for text embeddings models 4.9k★5/90d5 devs/90d -
Aphrodite Engine Inference & Serving 5.6 Large-scale LLM inference engine 1.8k★52/90d1 devs/90d -
LLGuidance Inference & Serving 5.4 Super-fast Structured Outputs 804★37/90d11 devs/90d -
Text Generation Inference Inference & Serving 5.3 Large Language Model Text Generation Inference 10.9k★0/90d0 devs/90d -
GPT4All Inference & Serving 5.0 GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use. 77.4k★0/90d0 devs/90d -
LoRAX Inference & Serving 5.0 Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs 3.8k★2/90d1 devs/90d -
ExLlamaV2 Inference & Serving 4.4 A fast inference library for running LLMs locally on modern consumer-class GPUs 4.6k★0/90d0 devs/90d -
LM Format Enforcer Inference & Serving 4.4 Enforce the output format (JSON Schema, Regex etc) of a language model 2k★0/90d0 devs/90d -
Cortex Inference & Serving 3.9 Local AI API Platform 2.8k★0/90d0 devs/90d