Free Crawl & Search — open-source & self-hostable

Crawlers, scrapers, and search for agents

Scores rate free-tier value on a 1–10 scale. How scores work →

Scrapy Crawl & Search 8.4 Scrapy, a fast high-level web crawling & scraping framework for Python. 62.9k★145/90d28 devs/90d
Browser Use Crawl & Search 8.0 🌐 Make websites accessible for AI agents. Automate tasks online with ease. 102.5k★665/90d7 devs/90d
Firecrawl Crawl & Search 7.9 The API to search, scrape, and interact with the web at scale. 🔥 143.7k★669/90d11 devs/90d
Meilisearch Crawl & Search 7.8 A lightning-fast search engine API bringing AI-powered hybrid search to your sites and applications. 58.4k★528/90d8 devs/90d
Maigret Crawl & Search 7.6 🕵️‍♂️ Collect a dossier on a person by username from 3000+ sites 34.9k★208/90d20 devs/90d
SearXNG Crawl & Search 7.6 SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled. 33.2k★257/90d9 devs/90d
Skyvern Crawl & Search 7.6 Automate browser based workflows with AI 22.1k★1.2k/90d11 devs/90d
Docling Crawl & Search 7.5 Get your documents ready for gen AI 62.6k★241/90d34 devs/90d
Scrapling Crawl & Search 7.4 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! 67.9k★145/90d9 devs/90d
Crawlee Crawl & Search 7.4 Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation. 24.4k★120/90d22 devs/90d
GPT Researcher Crawl & Search 7.3 An autonomous agent that conducts deep research on any data using any LLM providers 28.1k★78/90d27 devs/90d
Stagehand Crawl & Search 7.2 The SDK For Browser Agents 23.3k★167/90d11 devs/90d
Crawl4AI Crawl & Search 6.9 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN 70.8k★65/90d9 devs/90d
Katana Crawl & Search 6.9 A next-generation crawling and spidering framework. 17.1k★96/90d9 devs/90d
Quickwit Crawl & Search 6.9 Cloud-native OSS search engine for observability 11.4k★177/90d19 devs/90d
AgenticSeek Crawl & Search 6.8 Fully Local Manus AI. No APIs, No $200 monthly bills. Enjoy an autonomous agent that thinks, browses the web, and code for the sole cost of electricity. 🔔 Official updates only via twitter @Martin993886460 (Beware of fake account) 26.6k★60/90d9 devs/90d
MarkItDown Crawl & Search 6.7 Python tool for converting files and office documents to Markdown. 162.7k★4/90d3 devs/90d
Khoj Crawl & Search 6.7 Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free. 35.5k★10/90d4 devs/90d
ScrapeGraphAI Crawl & Search 6.7 Python scraper based on AI 28k★32/90d5 devs/90d
Colly Crawl & Search 6.7 Elegant Scraper and Crawler Framework for Golang 25.4k★23/90d8 devs/90d
Morphic Crawl & Search 6.7 An AI-powered search engine with a generative UI 9k★102/90d5 devs/90d
Trafilatura Crawl & Search 6.7 Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML 6.2k★27/90d9 devs/90d
Spider Crawl & Search 6.6 Low latency web data collector 2.6k★391/90d5 devs/90d
Perplexica (Vane) Crawl & Search 6.3 Vane is an AI-powered answering engine. 35.5k★20/90d2 devs/90d
Steel Browser Crawl & Search 6.1 🔥 Open Source Browser API for AI Agents & Apps. Steel Browser is a batteries-included browser sandbox that lets you automate the web without worrying about infrastructure. 7.3k★16/90d7 devs/90d
Open Deep Research Crawl & Search 6.0 Configurable open deep-research agent from LangChain that brings your own models, search tools and MCP servers for multi-step research. 11.9k★10/90d1 devs/90d
Jina Reader Crawl & Search 5.9 Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/ 11.5k★22/90d2 devs/90d
YaCy Crawl & Search 5.9 Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance 4k★16/90d4 devs/90d
Tavily Python Crawl & Search 5.8 The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features. 1.3k★10/90d5 devs/90d
Browsertrix Crawl & Search 5.6 Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all! 434★86/90d8 devs/90d
llm-scraper Crawl & Search 5.5 Turn any webpage into structured data using LLMs 6.8k★2/90d1 devs/90d
AgentQL Crawl & Search 5.4 AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger. 1.4k★16/90d3 devs/90d
Exa (exa-py) Crawl & Search 5.4 The Official Exa Python Package 216★28/90d9 devs/90d
Mozilla Readability Crawl & Search 5.3 A standalone version of the readability lib 11.3k★0/90d0 devs/90d
newspaper4k Crawl & Search 4.9 📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites. 1.1k★14/90d2 devs/90d
LaVague Crawl & Search 3.9 Large Action Model framework to develop AI Web Agents 6.4k★0/90d0 devs/90d
Farfalle Crawl & Search 3.7 🔍 AI search engine - self-host with local or cloud LLMs 3.5k★0/90d0 devs/90d
OpenDeepSearch Crawl & Search 3.6 SOTA search powered LLM 3.8k★0/90d0 devs/90d
go-readability Crawl & Search 3.6 Go package that cleans a HTML page for better readability. 940★0/90d0 devs/90d
Tarsier Crawl & Search 3.4 Vision utilities for web interaction agents 👀 1.8k★0/90d0 devs/90d