Agent Runtime

Deep Research Agent with Firecrawl and an Agent SDK

An agent that takes a research question, fans out web searches, scrapes the most promising sources into clean markdown with Firecrawl, and synthesizes a cited report — an open-source alternative to OpenAI Deep Research.

What This Builds

This recipe builds a deep research agent: you give it a question, and it autonomously decides what to search for, reads dozens of web pages, and returns a synthesized answer with citations. It is the open-source shape of products like OpenAI’s Deep Research — but you control which sources are trusted, keep the evidence trail, and can trigger it from your own workflow via an API call.

The agent loops: plan sub-questions, search the web, scrape the best results into clean markdown, decide whether it has enough, and either search more or write the final report. Firecrawl does the heavy lifting on the web side — its /search endpoint returns ranked results and its scrape returns LLM-ready markdown instead of raw HTML, so the model reasons over readable text.

The Stack

  • Firecrawl — search and scrape in one API. /search finds sources, scrape converts JavaScript-heavy pages into clean markdown, and /extract pulls structured fields when you need them. This is the agent’s eyes on the web.
  • OpenAI Agents SDK — the agent loop and tool-calling runtime; you register Firecrawl search and scrape as tools and let the model orchestrate them.
  • OpenAI (or Claude via the Anthropic Startup Program) — the reasoning model that plans queries, judges sources, and writes the report.

Step-by-Step Outline

  1. Get a Firecrawl API key and pip install firecrawl-py openai-agents.
  2. Wrap Firecrawl /search and scrape as agent tools (input: query or URL; output: results list or markdown).
  3. Write the agent’s instructions: break the question into sub-questions, search each, scrape the strongest hits, and stop when coverage is sufficient.
  4. Add a synthesis step that takes the gathered markdown and writes a structured report with inline [Source: URL] citations.
  5. Run the agent on a question and stream its tool calls so you can watch the search → scrape → reason loop.
  6. Optionally expose it behind an API endpoint or CLI so other systems can trigger research runs programmatically.

Why This Shape Works

The expensive, error-prone part of research agents is turning messy web pages into clean text the model can use — Firecrawl removes that entirely, returning markdown the LLM ingests directly. Letting the agent decide what to search next (rather than a fixed pipeline) is what makes it handle vague or evolving questions, and forcing citations on the final report keeps the output auditable.

Source