Free Observability & Evals — open-source & self-hostable

Tracing, metrics, evaluation, prompt testing

Scores rate free-tier value on a 1–10 scale. How scores work →

Grafana Observability & Evals 8.5 The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more. 75.3k★2.9k/90d46 devs/90d
Prometheus Observability & Evals 8.2 The Prometheus monitoring system and time series database. 64.9k★668/90d19 devs/90d
Langfuse Observability & Evals 7.9 🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 30.4k★1k/90d15 devs/90d
SigNoz Observability & Evals 7.8 SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool 27.6k★589/90d21 devs/90d
Promptfoo Observability & Evals 7.7 Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. 22.9k★1.2k/90d17 devs/90d
Opik Observability & Evals 7.7 Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards. 20.3k★1.1k/90d25 devs/90d
DeepEval Observability & Evals 7.5 The LLM Evaluation Framework 16.6k★601/90d18 devs/90d
Arize Phoenix Observability & Evals 7.4 AI Observability & Evaluation 10.4k★826/90d17 devs/90d
OpenTelemetry Collector Observability & Evals 7.2 OpenTelemetry Collector 7.2k★252/90d34 devs/90d
Garak Observability & Evals 7.1 the LLM vulnerability scanner 8.3k★310/90d20 devs/90d
Inspect AI Observability & Evals 7.0 Inspect: A framework for large language model evaluations 2.3k★1.2k/90d24 devs/90d
Latitude Observability & Evals 6.9 Latitude is the open-source ai monitoring platform. 4.4k★1.3k/90d9 devs/90d
Giskard Observability & Evals 6.8 🐢 Open-Source Evaluation & Testing library for LLM Agents 5.5k★116/90d15 devs/90d
Agenta Observability & Evals 6.8 The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place. 4.3k★2.9k/90d3 devs/90d
LangWatch Observability & Evals 6.8 The platform for LLM evaluations and AI agent testing 3.3k★1.1k/90d9 devs/90d
OpenLLMetry Observability & Evals 6.5 Open-source observability for your GenAI or LLM application, based on OpenTelemetry 7.3k★61/90d12 devs/90d
Guardrails AI Observability & Evals 6.5 Adding guardrails to large language models. 7.1k★53/90d6 devs/90d
Pydantic Logfire Observability & Evals 6.5 AI observability platform for production LLM and agent systems. 4.3k★127/90d22 devs/90d
Weave Observability & Evals 6.5 Weave is a toolkit for developing AI-powered applications, built by Weights & Biases. 1.1k★582/90d14 devs/90d
Laminar Observability & Evals 6.4 Laminar - open-source observability platform purpose-built for AI agents. YC S24. 3.1k★320/90d9 devs/90d
OpenLIT Observability & Evals 6.4 Open source platform for AI Engineering: OpenTelemetry-native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground. 🚀💻 Integrates with 50+ LLM Providers, VectorDBs, Agent Frameworks and GPUs. 2.6k★127/90d22 devs/90d
OpenInference Observability & Evals 6.3 OpenTelemetry Instrumentation for AI Observability 1.1k★224/90d24 devs/90d
TruLens Observability & Evals 6.2 Evaluation and Tracking for LLM Experiments and AI Agents 3.4k★52/90d16 devs/90d
DeepTeam Observability & Evals 5.9 DeepTeam is a framework to red team LLMs and AI agents. 1.9k★117/90d4 devs/90d
Helicone Observability & Evals 5.8 🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓 5.9k★8/90d3 devs/90d
Evidently Observability & Evals 5.7 Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics. 7.7k★5/90d4 devs/90d
Rhesis Observability & Evals 5.7 The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests, simulate (adversarial) conversations, and trace every failure to its root cause. 375★393/90d8 devs/90d
Monocle Observability & Evals 5.5 Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python. 265★126/90d15 devs/90d
AgentOps Observability & Evals 5.4 Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI 5.7k★1/90d1 devs/90d
Ragas Observability & Evals 4.9 Supercharge Your LLM Application Evaluations 🚀 14.6k★0/90d0 devs/90d
UpTrain Observability & Evals 3.6 UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them. 2.4k★0/90d0 devs/90d
Langtrace Observability & Evals 3.6 Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊 1.2k★0/90d0 devs/90d
Lunary Observability & Evals 3.5 Lunary AI JS Client (analytics and logging for AI apps) 📈 9★1/90d1 devs/90d
Athina Evals Observability & Evals 2.7 Python SDK for running evaluations on LLM generated responses 301★0/90d0 devs/90d
Zeno Observability & Evals 2.6 AI Data Management & Evaluation Platform 214★0/90d0 devs/90d
Literal AI Observability & Evals 2.3 Observability, evaluation, and analytics platform (from the Chainlit team) for production-grade LLM apps. 23★0/90d0 devs/90d