How to crawl websites for LLMs - using Firecrawl

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 5 min read
How to crawl websites for LLMs - using Firecrawl
On this page

Firecrawl is a crawler that turns websites into clean markdown, HTML or structured JSON that you can hand straight to an LLM. This post shows how to scrape a single page and crawl a whole site with the Firecrawl Python SDK, how to extract structured data with a Pydantic schema, how to load crawled pages into LangChain, and how to self-host Firecrawl with Docker.

Why bother? Many LLM applications, like customer service bots, research tools or general-purpose chatbots, need data from the public web. But websites are HTML pages full of navigation, scripts and layout markup. An LLM works much better with the plain content, and that's the part Firecrawl extracts. If you want to feed that content into a RAG pipeline, this is the first step.

Update (November 2026): Firecrawl's Python SDK has a new interface since this post. The client is now Firecrawl, and scrape_url, crawl_url, async_crawl_url and check_crawl_status became scrape, crawl, start_crawl and get_crawl_status, with options passed as keyword arguments instead of a params dict (see the Python SDK docs). For LangChain, Firecrawl now recommends its own langchain-firecrawl package, and the repository has moved to firecrawl/firecrawl.

What is Firecrawl?

Firecrawl is a web crawling and scraping tool that converts entire websites into clean, structured data in formats ready for large language models. Its main features:

  • It handles websites that render content with JavaScript, so modern, dynamic sites work.
  • It converts scraped pages into markdown or other structured formats that you can feed into LLMs or content pipelines.
  • It caches and rate-limits requests, so you don't overload the websites you scrape.
  • It scales from a single website to thousands.
  • It is open source, so you can self-host and customize it.
  • The documentation is clear and the API is simple, even if you are new to web scraping.
  • It can extract structured data, defined as pydantic models, using an LLM.

If you prefer a fully open-source alternative that you run in Docker from the start, see our Crawl4AI tutorial.

Crawling websites with Firecrawl

Let's see Firecrawl in action. We start with the managed cloud version of Firecrawl, as it makes getting started easy. If you want to self-host, jump straight to the last section of this article.

  1. Navigate to Firecrawl and sign up for an account.

  2. Head over to your Firecrawl keys section.

  3. Create a new API key, or use the default one. Copy the key.

  4. Install the Firecrawl Python client:

    1pip install firecrawl-py
  5. Use the following code snippet to scrape a single URL without any sub-pages:

    1 from firecrawl import FirecrawlApp
    2 app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
    3
    4 scrape_result = app.scrape_url('pondhouse-data.com', params={'formats': ['markdown', 'html']})
    5 print(scrape_result)

    This outputs the pondhouse-data website in markdown and HTML format.

    1{
    2 "markdown": "# Custom AI and Data engineering for your business\n\nWe\\'re a boutique agency specializing in cutting-edge AI and data solutions. From self-h....",
    3 "html": "<!DOCTYPE html><html lang=\"en\" class=\"text-base antialiased\"><!--$!--><!--/$--><body><div class=\"relative flex min-h-screen flex-col bg-background...."
    4}
  6. To crawl a website with multiple sub-pages, use the following code snippet:

    1from firecrawl import FirecrawlApp
    2
    3app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
    4
    5crawl_status = app.crawl_url(
    6 'https://pondhouse-data.com',
    7 params={
    8 'limit': 300,
    9 'scrapeOptions': {'formats': ['markdown']}
    10 },
    11 poll_interval=10
    12)
    13print(crawl_status)

    This crawls all pages of the site (up to the limit of 300), polls for the status every 10 seconds and creates a markdown representation of each page. The output looks similar to this:

    1{
    2 "status":"completed",
    3 "completed":36,
    4 "total":36,
    5 "creditsUsed":36,
    6 "expiresAt":"2024-09-07T16:34:46.000Z",
    7 "data":[
    8 {
    9 "markdown":"##### Pondhouse Data Blog\n\nPreviousNext\n\nShowing1to10of29results",
    10 "metadata":{
    11 "title":"Pondhouse AI Blog",
    12 "ogImage":"https://www.pondhouse-data.com/pondhouse-data-header.png",
    13 "ogTitle":"Pondhouse AI Blog",
    14 "language":"en",
    15 "sourceURL":"https://www.pondhouse-data.com/blog",
    16 "description":"Blogs and articles about AI applications and technology.",
    17 "ogDescription":"Blogs and articles about AI applications and technology.",
    18 "ogLocaleAlternate":[
    19
    20 ],
    21 "statusCode":200
    22 }
    23 },
    24 {
    25 "markdown":"# Advanced RAG: Recursive Retrieval with llamaindex\n\n![blog preview](https://www.pondhouse-data.com/_next/image?url=%2Fblog%2FBlogHeader_RecursiveRetrieval.jpg&w=3840&q=75)\n\nWhen it comes to [Retrieval Augmented Generation\\\\\n(RAG)](/blog/integrating-knowledge-and-llms), the quality of both the\ncreated document index and the retrieval process is crucial for getting\ngood and consistent answers based on your documentation. One especially\nchallenging aspect is how to model relationships between text chunks of\nyour documents.\n\nAs a quick reminder, in text-based RAGstatus":"completed",
    26 "scrapedUrls":300,
    27 "totalUrls":300
    28 }
    29 ...

Asynchronous crawling with Firecrawl

The scrape_url and crawl_url examples above run synchronously. When you crawl a large site with many sub-pages, a synchronous call can block your application for many minutes. Firecrawl can also run the crawl job asynchronously:

  1. Create a crawl job:

    1crawl_status = app.async_crawl_url(
    2 'https://pondhouse-data.com',
    3 params={
    4 'limit': 300,
    5 'scrapeOptions': {'formats': ['markdown']}
    6 }
    7)
    8print(crawl_status)

    This creates a crawl job and returns its job id, which you use to check the status of the job.

  2. Check the status of the job and retrieve the data:

    1import asyncio
    2
    3async def check_status():
    4 while True:
    5 crawl_data = app.check_crawl_status(crawl_status['id'])
    6 if crawl_data['status'] == 'completed':
    7 print(crawl_data)
    8 break
    9 await asyncio.sleep(10) # Wait for 10 seconds before checking again
    10
    11asyncio.run(check_status())

    Once the job is done, this prints the same result as the synchronous crawl.

Integrating Firecrawl with LangChain

You can also use Firecrawl through LangChain, a popular framework for building LLM applications. LangChain has a large collection of data loaders: Python classes that load data from many different sources into a standardized document format. One of them is the Firecrawl data loader.

  1. In addition to firecrawl-py, install the LangChain community package:

    1pip install langchain_community
  2. Then use the following code snippet to load the data from the pondhouse-data website:

    1from langchain_community.document_loaders import FireCrawlLoader
    2import os
    3
    4os.environ["FIRECRAWL_API_KEY"] = "fc-YOUR_API_KEY"
    5
    6loader = FireCrawlLoader(
    7 url = "https://pondhouse-data.com",
    8 mode = "crawl"
    9)
    10
    11data = loader.load()
    12# or use data = await loader.aload()

For more details, refer to Firecrawl's LangChain documentation.

Parsing specific content from websites using schema-based parsing

Besides returning formatted versions of the full page content, Firecrawl can use an LLM to extract specific information from a website. You tell it what to extract by providing a Pydantic schema.

  1. Define the schema:

    1from pydantic import BaseModel, Field
    2
    3class ExtractSchema(BaseModel):
    4 company_name: str
    5 company_sector: str
    6 is_ai_company: bool

    Give the fields descriptive names, because the LLM uses them to decide what to extract.

  2. Next, pass the schema to the Firecrawl call:

    1from firecrawl import FirecrawlApp
    2import json
    3app = FirecrawlApp(api_key='your_api_key')
    4data = app.scrape_url('https://www.pondhouse-data.com', {
    5 'formats': ['extract'],
    6 'extract': {
    7 'schema': ExtractSchema.model_json_schema(),
    8 }
    9})
    10print(json.dumps(data["extract"], indent=2))

    The output is a JSON object with the extracted data:

    1{
    2 "company_name": "Pondhouse Data OG",
    3 "company_sector": "AI and Data Solutions",
    4 "is_ai_company": true
    5}

This is one of Firecrawl's most useful features: you get structured information out of a messy website with about five lines of code. For the same idea applied to PDFs and other files, see our post on LLM document extraction.

Running your self-hosted version of Firecrawl

Firecrawl is open source, so you can run it on your own infrastructure. That's useful if you have specific data privacy or data security requirements.

  1. Install Docker, as outlined in the Docker documentation.

  2. Clone the Firecrawl GitHub repository:

    1git clone https://github.com/firecrawl/firecrawl.git

    Navigate into the repository:

    1cd firecrawl
  3. Create a .env file with the following content:

    1NUM_WORKERS_PER_QUEUE=8
    2PORT=3002
    3HOST=0.0.0.0
    4REDIS_URL=redis://redis:6379
    5REDIS_RATE_LIMIT_URL=redis://redis:6379

    Advanced features like authentication, AI features, JS block support or PDF parsing need additional environment variables. Read more in the Firecrawl self-hosting guide.

  4. Build and run the Docker containers:

    1docker compose up --build

    Your Firecrawl instance is now available at http://localhost:3002. To use your self-hosted Firecrawl API, instantiate your FirecrawlApp as follows:

    1app = FirecrawlApp(api_key="something", api_url='http://localhost:3002')

    Note: you need to set the api_key parameter, even though we haven't enabled Firecrawl authentication in this example. The Python SDK raises an error if api_key is not set, so set it to any string.

Conclusion

Firecrawl gets web content into a shape LLMs can use with very little code. scrape_url handles single pages, crawl_url and async_crawl_url handle whole sites, and a Pydantic schema turns a page into structured JSON. If you already use LangChain, FireCrawlLoader gives you the same data as LangChain documents.

If you can't send data to a managed service, the Docker Compose setup above gives you a self-hosted Firecrawl instance that the same Python SDK can talk to.

Crawled markdown is usually the input for a retrieval system. Our introduction to Retrieval Augmented Generation explains how such a system chunks, embeds and retrieves that content.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.