Crawl4AI Docker Tutorial: Crawl Websites for LLMs via REST API

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 7 min read
Crawl4AI Docker Tutorial: Crawl Websites for LLMs via REST API
On this page

Crawl4AI is an open-source web crawler built for AI applications. It renders JavaScript-heavy pages and returns clean markdown, cleaned HTML, page metadata and links that an LLM can process directly. It can also call an LLM to extract specific information. This tutorial builds the Crawl4AI Docker image, runs it as a REST API server, and walks through simple crawls, Python usage, advanced parameters and LLM-based extraction.

Building modern LLM applications often takes more than connecting to OpenAI's API. Real-world use cases need current, specific information beyond the model's training data, which is the idea behind Retrieval Augmented Generation (RAG).

Think of a competitive analysis bot that tracks competitor websites, a research assistant that needs access to academic papers, or an e-commerce AI that monitors prices across multiple stores. Accessing the data is only half the problem. You also need it in a format that LLMs can process well.

Modern websites rarely serve their content as simple HTML anymore. They use JavaScript to load content dynamically after the initial page load, so basic HTTP requests aren't enough. You need a crawler that executes JavaScript and waits for content to load, the way a real browser does. On an e-commerce site, for example, product prices might only appear after JavaScript runs; on a news site, articles might load as you scroll down the page.

Crawl4AI has two advantages that matter for AI applications. First, it handles many requests at once through asynchronous processing. When your application needs to analyze hundreds of pages quickly, this concurrency matters a lot for performance.

Second, Crawl4AI is designed to work with LLMs, both in how it formats its output and in how it can use AI to improve extraction. Instead of wrestling with complex HTML cleaning or writing brittle parsing rules, you get clean, structured data that's ready for an LLM.

Because it's open source, you can inspect the code, modify it for your needs and scale your infrastructure as required, without the limits of proprietary tools. (For another option, see our guide on crawling websites for LLMs using Firecrawl.)

In this guide, we set up Crawl4AI using Docker. The dockerized version of Crawl4AI runs a web server that handles multiple crawl requests at the same time. It also provides a REST API for crawling, so it's easy to integrate into virtually any application.

Update (November 2026): Crawl4AI's Docker server has changed a lot since this post. Since v0.5.0 it needs a .llm.env file for API keys, and /crawl now takes browser_config and crawler_config objects and returns results directly, with /crawl/stream for streaming and /crawl/job for async jobs. Official images now support both amd64 and arm64, so you don't need to build your own; the task_id flow below only works with the 0.3.x server. See the self-hosting guide for the current API.

Building the Crawl4AI Docker image

At the time of writing, Crawl4AI is at version 0.3.73. It recently introduced the full API and Docker support, including official docker containers, but only for arm64-based CPUs.

Since you most likely want to run Crawl4AI on an x86-based server, we have to build the image ourselves.

Note: It's worth checking their official docker hub for new images. I assume they will provide x86 images soon; then you can skip this whole preparation step.

  1. Clone the repository to your machine:

    1 git clone https://github.com/unclecode/crawl4ai.git
  2. Next, navigate into the cloned repository:

    1 cd crawl4ai
  3. Open the file main.py and remove the following line from the file.

    1 app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")

    This line mounts the Crawl4AI documentation and serves it as part of the web server. We don't need it, and it would require a second build step, which we want to avoid for now.

    Note: This might change in the future. If you can't find this line in the file, don't worry, just continue with the next steps.

  4. Now run the following command to build the docker image:

    1docker build -f Dockerfile -t pondhouse/crawl4ai:beta-1 --build-arg INSTALL_TYPE=all .

    Note: Instead of setting INSTALL_TYPE to all, you can also set it to basic. The full version is required for AI-powered extraction; if you only need web crawling without AI extraction, the basic version is enough.

How to use the Crawl4AI API

With the docker image built, let's start it and see how to use it.

1docker run -p 11235:11235 \
2-e OPENAI_API_KEY=<your-openai-api-key> \
3-e MAX_CONCURRENT_TASKS=5 \
4pondhouse/crawl4ai:beta-1
  • OPENAI_API_KEY: Set this to your OpenAI API key. We need it for the AI extraction tasks later on.

    Note: Crawl4AI uses litellm to connect to LLMs, so you can use most of the providers offered by liteLLM. Just change the provider parameter in the requests described below.

  • MAX_CONCURRENT_TASKS: The maximum number of concurrent tasks the crawler handles. Adjust it to your server's capabilities.

That's all we need to run Crawl4AI. The examples below use simple curl commands, but you can use any REST client.

A simple web crawl

First, a simple web crawl:

1curl -X POST 'http://127.0.0.1:11235/crawl' \
2-H 'Content-Type: application/json' \
3-d '{
4 "urls": "https://pondhouse-data.com"
5}'

The result looks like this:

1{ "task_id": "377fd003-2cd2-49bf-9f18-ee843ca21179" }

We don't get the crawl result directly, because Crawl4AI crawls asynchronously. (Asynchronous processing of multiple requests is one of its best features.) What we get back is the ID of the crawling task.

Now we have to wait for the crawl to finish. We do this by repeatedly calling the following endpoint:

1curl 'http://127.0.0.1:11235/task/<task_id_from_above>'

The response is a huge JSON object, but what we care about is its status field.

To stick with curl and bash, we can use jq to extract the status:

1curl 'http://127.0.0.1:11235/task/377fd003-2cd2-49bf-9f18-ee843ca21179' | jq .status

Once the status is completed, the task is done, and the task response also contains the crawl results. There is no need to call another API endpoint.

The results are in the result field, which has 4 interesting properties:

  1. result.cleaned_html contains a cleaned HTML version of the page:

    1 curl 'http://127.0.0.1:11235/task/<task-id>' | jq .result.cleaned_html
  2. result.markdown contains a markdown version of the page. For most AI use cases, this is the better format, as you usually don't need the HTML tags from the option above.

    1 curl 'http://127.0.0.1:11235/task/<task-id>' | jq .result.markdown
  3. result.metadata contains the header metadata of the page, like title, page description or og:image.

    1curl 'http://127.0.0.1:11235/task/<task-id>' | jq .result.metadata

    Results look like this:

    1{
    2 "title": "Pondhouse Data - Customized AI Solutions for your business",
    3 "description": "Find relevant information faster than ever!",
    4 "keywords": null,
    5 "author": null,
    6 "og:title": "Pondhouse AI - Customized AI Solutions for your business",
    7 "og:description": "Find relevant information faster than ever!",
    8 "og:image": "https://www.pondhouse-data.com/pondhouse-data-header.png",
    9 "twitter:card": "summary_large_image",
    10 "twitter:site": "@techscienceandy",
    11 "twitter:title": "Pondhouse AI - Customized AI Solutions for your business",
    12 "twitter:description": "Find relevant information faster than ever!",
    13 "twitter:image": "https://www.pondhouse-data.com/pondhouse-data-header.png"
    14}
  4. result.links is quite interesting: Crawl4AI has significant features for links. First and foremost, it extracts all internal and external links from the page and classifies them as such. Link analysis is beyond this post, but Crawl4AI has a good tutorial on it in their docs.

    1curl 'http://127.0.0.1:11235/task/<task-id>' | jq .result.links

Using the Crawl4AI API from Python

For completeness, here is how you'd make the requests above in Python, using the requests library:

1import time
2
3import requests
4
5timeout = 120 # seconds
6
7request_data = {
8 "urls": "https://pondhouse-data.com"
9}
10response = requests.post("http://127.0.0.1:11235/crawl", json=request_data)
11task_id = response.json()["task_id"]
12
13start_time = time.time()
14while True:
15 # Check for timeout
16 if time.time() - start_time > timeout:
17 raise TimeoutError(f"Task {task_id} timeout")
18
19 result = requests.get(f"http://127.0.0.1:11235/task/{task_id}")
20 status = result.json()
21
22 if status["status"] == "completed":
23 break
24
25 time.sleep(2)

Advanced parameters for the Crawl4AI API

Starting a simple crawling job is easy, but there are a number of parameters you might want to set:

1 "urls": "<your-url>",
2 "crawler_params": {
3 # Browser Configuration
4 "headless": True, # Run in headless mode
5 "browser_type": "chromium", # chromium/firefox/webkit
6 "user_agent": "custom-agent", # Custom user agent
7 "proxy": "http://proxy:8080", # Proxy configuration
8
9 # Performance & Behavior
10 "page_timeout": 30000, # Page load timeout (ms)
11 "verbose": True, # Enable detailed logging
12
13 # Anti-Detection Features
14 "simulate_user": True, # Simulate human behavior
15 "magic": True, # Advanced anti-detection
16 "override_navigator": True, # Override navigator properties
17
18 # Session Management
19 "user_data_dir": "./browser-data", # Browser profile location
20 "use_managed_browser": True, # Use persistent browser
21 }

Most of these parameters are self-explanatory, but the last 2 need some explanation. Session management in Crawl4AI lets you keep state across multiple requests and handle complex multi-page crawling tasks, which is particularly useful for dynamic websites.

See these two documentation pages for more:

  • Session management: here
  • Dynamic content handling: here

Extracting data with the Crawl4AI LLM extraction

So far we've run simple crawling jobs. Now let's step it up and use the AI extraction capabilities of Crawl4AI.

Say we want to crawl websites and find very specific information, like who owns a company.


Before we continue...

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

The following request makes Crawl4AI invoke an LLM to extract that information automatically:

1curl -X POST 'http://127.0.0.1:11235/crawl' \
2-H 'Content-Type: application/json' \
3-d '{
4 "urls": "https://pondhouse-data.com",
5 "extraction_config": {
6 "type": "llm",
7 "params": {
8 "provider": "openai/gpt-4o",
9 "api_token": "<your-openai-api-token>",
10 "instruction": "From the provided homepage data, extract the owners of the company."
11 }
12 }
13}'

The extracted results are in the result.extracted_content field. It is JSON-encoded, so we use jq's fromjson filter to decode it:

1curl 'http://127.0.0.1:11235/task/<task-id>' | jq '.result.extracted_content | fromjson'
1[
2 {
3 "index": 0,
4 "tags": ["owners"],
5 "content": ["Sascha Gstir", "Andreas Nigg"],
6 "error": false
7 }
8]

The LLM correctly returned both company owners, and getting this whole pipeline working took just two API calls.

Note: At the time of writing, if you create a crawling job for the same URL multiple times, the parameters seem to be cached. If you call the same URL twice with different extraction instructions, you get the cached extraction. Keep this in mind.

Using different LLM providers with Crawl4AI

To use a different LLM provider, like Azure OpenAI, refer to the liteLLM providers manual, as Crawl4AI uses liteLLM under the hood. Crawl4AI passes api_token to liteLLM as the API key, so set it to your Azure OpenAI API key. liteLLM also needs your endpoint and API version, which you can set with the AZURE_API_BASE and AZURE_API_VERSION environment variables when you start the container. The part after azure/ is your deployment name.

An example request using Azure OpenAI:

1curl -X POST 'http://127.0.0.1:11235/crawl' \
2-H 'Content-Type: application/json' \
3-d '{
4 "urls": "https://pondhouse-data.com",
5 "extraction_config": {
6 "type": "llm",
7 "params": {
8 "provider": "azure/gpt-4o",
9 "api_token": "<your-azure-openai-api-key>",
10 "instruction": "From the provided homepage data, extract the owners of the company."
11 }
12 }
13}'

The same kind of LLM extraction works on PDFs and scanned documents. We show it with an open-source vision model in extracting text and tables from documents with LLMs.

Further reading

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.