Agent Runtime

Multi-Agent Lead Scraper that Outputs a Structured Leads Table

A CrewAI crew where a researcher agent searches and scrapes the web and an analyst agent normalizes the findings into a structured, deduplicated leads table written to CSV or a database.

What This Builds

This recipe builds an agentic extraction pipeline using a small crew of role-playing agents instead of a single hardcoded scraper. A researcher agent is given a search tool and a scrape tool; it finds candidate company pages, scrapes them, and pulls out raw signals (name, website, contact, description). An analyst agent then cleans, deduplicates, and shapes those signals into a consistent leads schema, and the final task writes the result to CSV or a database table.

The agentic framing matters when sources are heterogeneous: the researcher can decide which links are worth scraping and retry with different queries, rather than failing on a rigid selector path.

The Stack

  • CrewAI orchestrates the agents, tasks, and tools. You define agents with a role/goal/backstory, give them tools, and chain tasks (research → analyze → write) into a Crew.
  • Serper (Google Search API) backs the search tool — CrewAI ships a SerperDevTool. A web-search MCP server is an equally valid drop-in.
  • Firecrawl backs the scrape tool, returning clean markdown/structured JSON for each candidate page so the agent reasons over readable text.
  • OpenAI (or any chat model CrewAI supports) is the reasoning engine for both agents.

Step-by-Step Outline

  1. pip install crewai crewai-tools and set API keys for the model, Serper, and Firecrawl.
  2. Define a researcher agent with SerperDevTool and a Firecrawl scrape tool; its goal is to find and extract raw company data for a target query (industry + region).
  3. Define an analyst agent (no tools) whose goal is to normalize the researcher’s output into a fixed schema and drop duplicates.
  4. Write a research_task (search the web, scrape the most promising results, return raw extracted fields) assigned to the researcher.
  5. Write an analysis_task whose context is the research task, instructing the analyst to emit a clean list of leads matching the schema; use output_pydantic/output_json to force structured output.
  6. Assemble Crew(agents=[...], tasks=[...], process="sequential"), run crew.kickoff(inputs=...), and write the validated structured result to CSV or insert it into a database table.

Why This Shape Works

Splitting roles keeps each agent’s prompt narrow and its tool surface small, which is the practical lesson from CrewAI’s own examples. The researcher tolerates messy, varied sources because it can search and re-scrape; the analyst guarantees the output conforms to a schema. Forcing structured output (Pydantic/JSON) on the final task is what turns a chatty agent run into a clean, storable ETL artifact.

Source

Based on CrewAI’s official multi-agent examples and the Firecrawl + CrewAI tutorial “Building Multi-Agent Systems With CrewAI”. CrewAI examples: https://github.com/crewAIInc/crewAI-examples — Tutorial: https://www.firecrawl.dev/blog/crewai-multi-agent-systems-tutorial