Advanced RAG: Recursive Retrieval with llamaindex

On this page
Recursive retrieval is a RAG technique where the retriever does not stop at the most similar chunk. If a retrieved node points to another retriever or query engine, the retriever follows that link and pulls in the related data as well. This guide builds it with LlamaIndex: we index a PDF about the world's billionaires, extract its tables with camelot into pandas dataframes, and let LlamaIndex's RecursiveRetriever send table questions to a PandasQueryEngine.
The problem it solves shows up in any Retrieval Augmented Generation (RAG) system. The quality of the document index and of the retrieval step decides whether you get good, consistent answers from your documentation. One of the harder parts is modeling the relationships between the text chunks of your documents.
In text-based RAG, documents are first parsed into text, which is then split into smaller chunks. We split because LLMs have a maximum input length and are billed per token, so we want to keep the input short. The chunks are indexed and then retrieved during generation.
That creates the challenge: if we retrieve chunks, how do we make sure we get all the relevant information when it is scattered across several of them? Tables make this harder, because they usually need different handling than flowing text. How do you capture the relationship between a table and the text around it?
Note: This guide is heavily influenced by this very good tutorial from LlamaIndex. We add more context and explanation to make it easier to follow, but full credit goes to the LlamaIndex team.
What is recursive retrieval?
In normal retrieval, we use the user query to find the documents the LLM needs to answer that query. This is mostly done by comparing the semantic meaning of the query with the semantic meaning of the documents in our index. (Strictly speaking, we compare embeddings of the query and the documents, which is not exactly semantic meaning, but it's a good enough approximation for now.)
These documents are chunks. We split the full texts of our source documents into smaller pieces and index those, so we can retrieve relevant information even when a whole document is too long for the LLM.
Chunking has a downside. If the relevant information is spread across multiple chunks, a single retrieval might not find all of it. Tables are a good example: often the "semantic meaning" of a table is not captured by the table itself, but by the text surrounding it.
Recursive retrieval solves this by looking not only at the semantically most similar chunks, but also at the chunks and data sources linked to them. That way we capture the relevant information even when it spans several chunks.
So recursive retrieval needs two things:
- A way to define relationships between document chunks
- A way to recursively retrieve the related chunks
There are several ways to implement recursive retrieval. We use LlamaIndex, because it ships a proven implementation (and is a good RAG framework in general). For a different way of keeping context across chunks, see our post on late chunking.
What is LlamaIndex?
LlamaIndex is a Python and TypeScript library for building LLM applications for "context augmentation" (which basically means RAG). It has tools for indexing documents, retrieving relevant documents and chunks, and generating answers from the retrieved documents.
Its main components are:
- Data connectors ingest existing data from its native source and format. These could be APIs, PDFs, SQL, and (much) more.
- Data indexes structure your data in intermediate representations that are easy and fast for LLMs to consume.
- Engines provide natural language access to your data. For example:
- Query engines are retrieval interfaces for knowledge-augmented output.
- Chat engines are conversational interfaces for multi-message, "back and forth" interactions with your data.
- Data agents are LLM-powered knowledge workers augmented by tools, from simple helper functions to API integrations and more.
- Application integrations tie LlamaIndex back into the rest of your ecosystem. This could be LangChain, Flask, Docker, ChatGPT, or many others.
You'll find more in the LlamaIndex documentation, which is excellent.
How to implement recursive retrieval with LlamaIndex
Recursive retrieval in LlamaIndex is built from two parts: data indexes and query engines. Rather than more theory, let's go straight to a hands-on example.
First, download the example data from our website.
We use camelot to extract tables from the PDF. It needs these system dependencies:
Then install LlamaIndex and its dependencies:
Note: LlamaIndex changed a lot between versions 0.9 and 0.10. It's best to remove the old version first and then install the latest one.
Next, import the required libraries and define which OpenAI models to
use. Replace Your-api-key with your own OpenAI API key.
Optionally, set up debug logging to see exactly which prompts LlamaIndex sends to the LLM and which responses it gets back.
Now we can extract the text from the PDF. Note that this loads the tables as plain text, not as tabular data. We'll handle the tables properly in the next step.
The docs object now contains the text of the PDF plus metadata like
page numbers. LlamaIndex makes loading documents and extracting their
text very easy.
Extract PDF tables with camelot
We already have the text of the tables, but parsing tables as plain text is not ideal and often loses important information. The main reason is that the PDF standard does not define a table as an object. A table is just text and lines, and normal text parsers have a hard time extracting that information.
camelot is built specifically to recognize tables in PDFs and extract
them as tabular data, such as a pandas dataframe.
This snippet extracts the tables on pages 3 and 24 and stores them in a list of pandas dataframes. We'll use these dataframes to give the tables a more structured representation in our index.
Query the tables with PandasQueryEngine
So far we have extracted the information from the source document. How do we make it accessible to the LLM? In other words, how do we search for relevant information at query time?
This is where LlamaIndex's QueryEngine comes in. It abstracts the data
and provides an interface to "connect" that data to an LLM. Asking
questions against our parsed tables takes only a few lines of code:
LlamaIndex has query engines for various data sources, like SQL, CSV,
and more. The PandasQueryEngine is built for pandas dataframes. At
query time, it sends the user query along with df.head() to the LLM and
asks it to return Python code that answers the question.
So the LLM works with the data in the dataframe indirectly, without needing to see the whole dataframe.
Link tables and text in one VectorStoreIndex
Now that we can query the tables, we link them to the flowing text. For
that we build a VectorStoreIndex, an index that stores and retrieves
document chunks based on their semantic similarity. Before the code,
here is the strategy.
LlamaIndex represents the data in an index as "nodes", and nodes can have relationships to other nodes. For example, a node with the full text of a document can be related to nodes that represent the tables in that document. So we can do this:
- Create a node for each of the tables, with either a short description or (better) related text, so that we can retrieve them based on the user query.
- Create nodes from the text of the PDF.
- Combine the table nodes and the text nodes into one index.
Each table node is an IndexNode with an index_id (pandas0,
pandas1), and df_id_query_engine_mapping maps those IDs to the
matching pandas query engines. That mapping is the link the recursive
retriever follows.
Note: In the example above, we describe the table nodes by hand. In a real-world scenario, you would probably extract this information automatically by sending parts of the tables to an LLM and asking it to describe them, or use the text surrounding each table.
Build the RecursiveRetriever and query engine
The VectorStoreIndex now contains the table nodes and the text nodes.
We use it to create a RecursiveRetriever and a RetrieverQueryEngine.
The RetrieverQueryEngine again gives us a handy interface for asking
questions via the LLM.
Note: In the example above, we use a response_synthesizer to make
the LLM's response nicer. It is optional and can be omitted. More
information about the response_synthesizer is in the LlamaIndex
documentation.
To ask a question, call the query method.
And that's it: recursive retrieval with LlamaIndex is working.
Conclusion
Recursive retrieval helps when the answer to a question lives in
a structure that plain chunking handles badly, such as a table in a PDF.
In this example, camelot extracts the tables as pandas dataframes, each
table gets a PandasQueryEngine, and a short IndexNode summary
represents each table in the same VectorStoreIndex as the text chunks.
When vector search returns a table node, LlamaIndex's RecursiveRetriever
follows its index_id to the matching query engine, and the LLM answers
from the actual table data instead of from table text mangled by the PDF
parser.
To apply this to your own documents, start with the table descriptions. We wrote them by hand here. For more documents, generate them with an LLM or take them from the text around each table. Then try the code on your own PDFs and check whether your answers improve, for example with the RAGAS evaluation approach described in our other post.
If camelot misses tables in your PDFs, a vision model can read them from page images instead, as in parsing PDF, Word and Excel documents with GPT-4o.
Further reading
- Increase RAG performance using ColBERT reranker
- How to test your RAG pipeline?
- Semantic search for databases
- Improving retrieval with Hypothetical Document Embeddings (HyDE)
- RAGFlow: open-source RAG engine with deep document understanding
- Building a RAG system with Azure AI Search
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



