How to crawl websites for LLMs - using Firecrawl

On this page
Firecrawl is a crawler that turns websites into clean markdown, HTML or structured JSON that you can hand straight to an LLM. This post shows how to scrape a single page and crawl a whole site with the Firecrawl Python SDK, how to extract structured data with a Pydantic schema, how to load crawled pages into LangChain, and how to self-host Firecrawl with Docker.
Why bother? Many LLM applications, like customer service bots, research tools or general-purpose chatbots, need data from the public web. But websites are HTML pages full of navigation, scripts and layout markup. An LLM works much better with the plain content, and that's the part Firecrawl extracts. If you want to feed that content into a RAG pipeline, this is the first step.
Update (November 2026): Firecrawl's Python SDK has a new interface since this post. The client is
now Firecrawl, and scrape_url, crawl_url, async_crawl_url and check_crawl_status became scrape,
crawl, start_crawl and get_crawl_status, with options passed as keyword arguments instead of
a params dict (see the Python SDK docs). For LangChain, Firecrawl
now recommends its own langchain-firecrawl package,
and the repository has moved to firecrawl/firecrawl.
What is Firecrawl?
Firecrawl is a web crawling and scraping tool that converts entire websites into clean, structured data in formats ready for large language models. Its main features:
- It handles websites that render content with JavaScript, so modern, dynamic sites work.
- It converts scraped pages into markdown or other structured formats that you can feed into LLMs or content pipelines.
- It caches and rate-limits requests, so you don't overload the websites you scrape.
- It scales from a single website to thousands.
- It is open source, so you can self-host and customize it.
- The documentation is clear and the API is simple, even if you are new to web scraping.
- It can extract structured data, defined as pydantic models, using an LLM.
If you prefer a fully open-source alternative that you run in Docker from the start, see our Crawl4AI tutorial.
Crawling websites with Firecrawl
Let's see Firecrawl in action. We start with the managed cloud version of Firecrawl, as it makes getting started easy. If you want to self-host, jump straight to the last section of this article.
-
Navigate to Firecrawl and sign up for an account.
-
Head over to your Firecrawl keys section.
-
Create a new API key, or use the default one. Copy the key.
-
Install the Firecrawl Python client:
-
Use the following code snippet to scrape a single URL without any sub-pages:
This outputs the pondhouse-data website in markdown and HTML format.
-
To crawl a website with multiple sub-pages, use the following code snippet:
This crawls all pages of the site (up to the limit of 300), polls for the status every 10 seconds and creates a markdown representation of each page. The output looks similar to this:
Asynchronous crawling with Firecrawl
The scrape_url and crawl_url examples above run synchronously. When you
crawl a large site with many sub-pages, a synchronous call can block your
application for many minutes. Firecrawl can also run the crawl job
asynchronously:
-
Create a crawl job:
This creates a crawl job and returns its job id, which you use to check the status of the job.
-
Check the status of the job and retrieve the data:
Once the job is done, this prints the same result as the synchronous crawl.
Integrating Firecrawl with LangChain
You can also use Firecrawl through LangChain, a popular framework for building LLM applications. LangChain has a large collection of data loaders: Python classes that load data from many different sources into a standardized document format. One of them is the Firecrawl data loader.
-
In addition to
firecrawl-py, install the LangChain community package: -
Then use the following code snippet to load the data from the pondhouse-data website:
For more details, refer to Firecrawl's LangChain documentation.
Parsing specific content from websites using schema-based parsing
Besides returning formatted versions of the full page content, Firecrawl can use an LLM to extract specific information from a website. You tell it what to extract by providing a Pydantic schema.
-
Define the schema:
Give the fields descriptive names, because the LLM uses them to decide what to extract.
-
Next, pass the schema to the Firecrawl call:
The output is a JSON object with the extracted data:
This is one of Firecrawl's most useful features: you get structured information out of a messy website with about five lines of code. For the same idea applied to PDFs and other files, see our post on LLM document extraction.
Running your self-hosted version of Firecrawl
Firecrawl is open source, so you can run it on your own infrastructure. That's useful if you have specific data privacy or data security requirements.
-
Install Docker, as outlined in the Docker documentation.
-
Clone the Firecrawl GitHub repository:
Navigate into the repository:
-
Create a
.envfile with the following content:Advanced features like authentication, AI features, JS block support or PDF parsing need additional environment variables. Read more in the Firecrawl self-hosting guide.
-
Build and run the Docker containers:
Your Firecrawl instance is now available at
http://localhost:3002. To use your self-hosted Firecrawl API, instantiate yourFirecrawlAppas follows:Note: you need to set the
api_keyparameter, even though we haven't enabled Firecrawl authentication in this example. The Python SDK raises an error ifapi_keyis not set, so set it to any string.
Conclusion
Firecrawl gets web content into a shape LLMs can use with very little code.
scrape_url handles single pages, crawl_url and async_crawl_url handle
whole sites, and a Pydantic schema turns a page into structured JSON. If
you already use LangChain, FireCrawlLoader gives you the same data as
LangChain documents.
If you can't send data to a managed service, the Docker Compose setup above gives you a self-hosted Firecrawl instance that the same Python SDK can talk to.
Crawled markdown is usually the input for a retrieval system. Our introduction to Retrieval Augmented Generation explains how such a system chunks, embeds and retrieves that content.
Further reading
- Using AI directly from your PostgreSQL database
- Use Vector Search in BigQuery
- How to host your own LLM - including HTTPS?
- RAGFlow: an open-source RAG engine with deep document understanding
- How to evaluate the performance of your RAG pipeline with Ragas
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



