LLM Document Extraction: Pull Structured Data from Complex PDFs

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 8 min read
LLM Document Extraction: Pull Structured Data from Complex PDFs
On this page

Large Language Models with vision capabilities can extract structured data from documents that defeat conventional tools, such as multi-column PDFs and complex tables. In this post we convert each PDF page to an image, send it to the open-source MiniCPM-Llama3-V2.5 model, and prompt it for the output format we want. For a model with only 8B parameters, the results were excellent: it extracted both text and tables in a very usable format.

We at Pondhouse AI rely heavily on extracting data from documents: Word, PDF, Excel, you name it. All of these formats give the creator almost unlimited freedom in how to lay out the content. That's great for writing documents and a nightmare for extracting data from them, especially when the content is technical and uses complicated layouts like multi-column PDFs or tables.

Traditional extraction methods, such as rule-based systems or conventional machine learning, often struggle with how complex and varied document layouts are. LLMs can understand context, interpret natural language and adapt to different formats, which makes them a promising alternative.

This post looks at why document extraction is hard, where traditional methods fall short, and how to use an open-source LLM for the job, with working Python code.

Update (November 2026): OpenBMB has since released newer MiniCPM-V models, most recently the MiniCPM-V 4.7 family in October 2026 (see the OpenBMB models on Hugging Face). The code below uses MiniCPM-Llama3-V 2.5, which is still available.

TL;DR If you are just interested in the code for how to use an Open Source LLM for retrieving data from documents, simply jump to the Conclusion.

The challenge of document data extraction

The main challenge in document data extraction is the sheer diversity of formats and layouts. Documents come in many file types, including:

  • PDFs (Portable Document Format)
  • Microsoft Word documents (.doc, .docx)
  • Excel spreadsheets (.xls, .xlsx)
  • Scanned images (.jpg, .png, .tiff)
  • HTML files
  • Plain text files (.txt)

Each format has its own structure and quirks. And within each format, there's an almost infinite variety of layouts. Documents can include:

  • Single or multiple columns
  • Tables and charts
  • Headers and footers
  • Sidebars and text boxes
  • Images and diagrams
  • Varying fonts, sizes, and styles

This variability makes a one-size-fits-all solution extremely hard to build. A method that works perfectly for one document may fail entirely on another, even if both contain similar information.

Why technical documents and multi-column PDFs are hard

Technical documents and multi-column PDFs bring their own problems:

  • Technical jargon: these documents often contain specialized terminology and acronyms that need domain knowledge to interpret correctly.

  • Complex tables: technical documents frequently include tables with merged cells, nested headers, or footnotes.

  • Equations and formulas: mathematical or chemical equations are hard to parse and interpret programmatically.

  • Multi-column layouts: PDFs with multiple columns confuse text extraction algorithms, which have to work out the correct reading order across columns.

  • Footnotes and references: these interrupt the main text flow and complicate extraction.

  • Diagrams and flowcharts: technical diagrams often contain important information but are hard to interpret automatically.

Limitations of conventional extraction methods

Traditional extraction methods have several limitations with complex documents:

  • Rule-based systems rely on predefined rules. They work for standardized documents, but struggle with variability and need constant updates as document formats change.

  • Regular expressions are powerful for pattern matching, but they don't understand context and break easily when layouts change.

  • Optical Character Recognition (OCR) has improved a lot, but it still produces errors, especially with poor-quality scans or complex layouts.

  • Template matching works well for documents with a fixed structure but fails when a layout deviates from the expected template.

  • Conventional methods often extract data based on position or pattern, without understanding the meaning of the information.

  • Many traditional methods struggle with free-form text or documents without a clear structure.

  • Conventional systems often need extensive retraining or reprogramming for new document types or formats.

Document extraction needs approaches that are more flexible and actually understand the content. LLMs address many of these problems.

Using LLMs for structured data extraction from documents

What would the ideal document extraction system look like? It would:

  • Understand the context of the document

  • Interpret natural language

  • Adapt to various formats and layouts

  • Handle technical jargon and complex tables

  • Handle various languages and writing styles

  • Preserve the structure of the document (e.g., tables, columns)

  • Be accurate and reliable

Looking at these requirements, Large Language Models are worth a try. LLMs understand language well, they are not bound to formats, you can tell them to format the output exactly as you need it, and they can be trained on technical jargon.

That leaves two questions: How do we even get our documents into the LLM? And does it perform well enough for this job? Let's find out.

Disclaimer: This post shows a hands-on example of how to use LLMs for document extraction. We comment on how well the test went, but this is not a full benchmark. You will, however, get what you need to build your own.

Hands-on: MiniCPM-Llama3-V2.5, an open-source LLM for structured data extraction

For this test we use the MiniCPM-Llama3-V2.5 model, an open-source Large Language Model based on Llama3 with strong vision capabilities. According to the benchmarks its creators published at the release in May 2024, this 8B-parameter model beats widely used proprietary models like GPT-4V-1106, Gemini Pro, Claude 3 and Qwen-VL-Max on both vision and language tasks.

The model also looks very good at OCR. According to the same May 2024 announcement, MiniCPM-Llama3-V 2.5 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), and scores 700+ on OCRBench, ahead of proprietary models such as GPT-4o, GPT-4V-0409, Qwen-VL-Max and Gemini Pro.

How to provide documents to the Large Language Model

How do you "send" a document to an LLM? LLMs can't open and read document files. But modern multimodal LLMs can "read" and "understand" images. You see where this is going: we convert the documents to images and give these images to the model. The model then uses its vision capabilities to understand the document and extract the text.

At first we were a little hesitant to extract data this way, because it seems quite brute-force. But consider how humans read documents. Our eyes create images of the page and send them to the brain, where the understanding (and extraction, if you will) happens. That's not so different from what we're doing here.

The system we are about to build looks like this:

Each document page is screenshotted and sent to the LLM as an image

The code to extract structured data from documents

Enough theory, let's get our hands dirty.

Note: This tutorial assumes you have a CUDA-enabled GPU with at least 16 GB of VRAM.

First, install the required libraries:

1pip install Pillow torch torchvision transformers sentencepiece pymupdf

Note: We use PyMuPDF to turn the PDF pages into images. It has a copyleft license, so check whether it fits your use case. Any other library that can render PDF pages to PNGs works too.

Next, import the required libraries and load the model and tokenizer:

1import torch
2from PIL import Image
3from transformers import AutoModel, AutoTokenizer
4import fitz # PyMuPDF
5
6model = AutoModel.from_pretrained(
7 "openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True, torch_dtype=torch.float16
8)
9model = model.to(device="cuda")
10
11tokenizer = AutoTokenizer.from_pretrained(
12 "openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True
13)
14model.eval()

This might take a while, so grab a coffee.

Now we create images from our document. For this tutorial we use a PDF file and create one image per page:

1pdf_path = "mypdf.pdf"
2pdf_document = fitz.open(pdf_path)
3
4images = []
5
6# Loop through each page in the PDF
7for page_number in range(len(pdf_document)):
8 page = pdf_document.load_page(page_number)
9 pix = page.get_pixmap()
10 img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
11
12 images.append(img)

The code loops through all pages of the document and creates an image from each page. The images are collected in the images list.

Finally, we send the images to the model and extract the text. (The code shows a single image, but you can easily loop over all images in the list.)

1question = """Extract all the text in this image.
2If there is a header or a footer, just ignore it.
3Extract tables as markdown tables.
4Don't use the subtitles for the list items, just return the list as text.
5"""
6msgs = [{"role": "user", "content": question}]
7
8res = model.chat(
9 image=images[0],
10 msgs=msgs,
11 tokenizer=tokenizer,
12 sampling=True,
13 temperature=0.7,
14 # system_prompt="" # pass system_prompt if needed
15)
16print(res)

We pass the image to the model along with a question, and the model returns the extracted text.

As the question variable shows, we can simply tell the model how we want the text extracted. In our experience the model's understanding of complex requests is a little limited, as it's based on the Llama3 8B model. Still, a surprising number of requests work very well, like removing headers and footers from the output.

Note: Make sure to use a concurrent and batched version of this sample when dealing with larger amounts of data.

For an off-the-shelf, rather small LLM, the results are phenomenal. The model extracted both the text and the tables in our examples in a very reasonable format.

One thing we noticed during testing: the model always outputs the text with prefixes, like:

1**title**: "The actual title"
2**text**: "The actual text"
3**subtitle**: "The subtitle"
4**text**: "The text of this paragraph"
5...

This has upsides and downsides. You get the semantics of the text right in the output, which is especially useful if you want to chunk the text later. But if you just want the text, the prefixes are a little annoying.

We tried to remove them with prompting, but the model is very stubborn here. This output format seems to be deeply ingrained by its training. The best strategy is to strip the prefixes after extraction. In our tests they were very consistent, so a simple regex might do the trick.

Conclusion

This post covered why extracting data from complex documents is hard, where traditional methods fall short, and how to use an open-source LLM for document data extraction instead.

Our hands-on example showed that LLMs can be highly effective at extracting text from documents, even with complex layouts and technical content. The model we used is relatively small, yet it performed well and gave accurate results.

We are very excited about these results. The extracted text was good, and the approach gives our data extraction pipelines a new level of flexibility, because we can instruct the LLM exactly how to extract the data. LLM-based extraction now runs in production on customer orders at Meusburger, where 96% of orders arriving by email, PDF or Excel go through without correction.

The model also handled tables impressively well (along with various other layouts we tested but didn't describe here). This is a big improvement over anything we had in the past.

As a next step, we plan to compare this small and cheap model with GPT-4o, in mid-2024 widely seen as the strongest vision model, to see whether MiniCPM-Llama3-V2.5 holds up or whether a much bigger model is necessary for document extraction. Our follow-up post shows how to parse PDF, Word and Excel documents with GPT-4o. For images that are not documents, such as charts and dashboards, see analyzing images with GPT-4o vision.

Summary: The code for the complete example

1import torch
2from PIL import Image
3from transformers import AutoModel, AutoTokenizer
4import fitz # PyMuPDF
5
6model = AutoModel.from_pretrained(
7 "openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True, torch_dtype=torch.float16
8)
9model = model.to(device="cuda")
10
11tokenizer = AutoTokenizer.from_pretrained(
12 "openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True
13)
14model.eval()
15
16pdf_path = "mypdf.pdf"
17pdf_document = fitz.open(pdf_path)
18
19images = []
20
21for page_number in range(len(pdf_document)):
22 page = pdf_document.load_page(page_number)
23 pix = page.get_pixmap()
24 img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
25
26 images.append(img)
27
28pdf_document.close()
29
30question = """Extract all the text in this image.
31If there is a header or a footer, just ignore it.
32Extract tables as markdown tables.
33Don't use the subtitles for the list items, just return the list as text.
34"""
35msgs = [{"role": "user", "content": question}]
36
37res = model.chat(
38 image=images[0], # loop through all images in images list here
39 msgs=msgs,
40 tokenizer=tokenizer,
41 sampling=True,
42 temperature=0.7,
43 # system_prompt='' # pass system_prompt if needed
44)
45print(res)

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.