Crawl4AI
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
Crawl4AI scores 6.9/10 (Solid) in Crawl & Search.
Language Python License Apache-2.0 Stars 70.8k★ Type library Self-host Yes Repo unclecode/crawl4ai Homepage https://crawl4ai.com
Quality
Solid 6.9
Adoption 71
Activity 64
Maturity 92
Community 72
Capability 12
Show the math
Overall 6.9/10
value = 6.9/10
score = value
Adoption 7.4/10
value = 71.3/100
score = 1 + 9 * value / 100
Activity 6.8/10
value = 63.9/100
score = 1 + 9 * value / 100
Maturity 9.2/10
value = 91.5/100
score = 1 + 9 * value / 100
Community 7.5/10
value = 72.1/100
score = 1 + 9 * value / 100
Capability 2.1/10
value = 12/100
score = 1 + 9 * value / 100
Key metrics
70.8k★ GitHub stars
65/90d Recent commits
9 devs/90d Recent contributors
12 pages/s Crawl throughput (corpus avg, competitor benchmark) spider.cloud benchmark (run by competitor Spider) over 1,000 URLs: Crawl4AI 12 pages/s corpus average (out-of-the-box, no residential proxies), 89.7% success rate, recall@5 84.5%. Competitor-run benchmark - flagged. Source: https://spider.cloud/blog/firecrawl-vs-crawl4ai-vs-spider-honest-benchmark
2.2y Project age
active Last commit
Gotchas
No gotchas documented yet. Contribute one if you know a constraint we missed.
Labels
- Self-hosted
Self-hosted
You deploy and operate it yourself; there is no hosted option here.
crawler markdown extraction python rag