Advanced RAG: Improving Retrieval-Augmented Generation with Hypothetical Document Embeddings (HyDE)

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 6 min read
Advanced RAG: Improving Retrieval-Augmented Generation with Hypothetical Document Embeddings (HyDE)
On this page

Hypothetical Document Embeddings (HyDE) improve the retrieval step of Retrieval-Augmented Generation (RAG). Instead of searching your knowledge base with the embedding of the user's question, HyDE first asks a language model to write a fake document that answers the question. It then searches with the embedding of that fake document. Because the fake answer looks more like the real documents than a short question does, the vector search tends to find more relevant chunks.

This post explains why plain query-to-document similarity often falls short, how HyDE works, and how to use it in your own RAG system with LlamaIndex's HyDEQueryTransform, including how to customize the prompt that generates the fake documents. If you are new to RAG, start with our introduction to Retrieval-Augmented Generation.

What are Hypothetical Document Embeddings (HyDE)?

In a traditional RAG setup, retrieval is based on the similarity between the user's query and the source document chunks. That similarity is mostly calculated with embeddings, which are vector representations of text. To a good approximation, embeddings capture the semantic meaning of both the documents and the search query. Long story short, traditional RAG looks for document chunks whose meaning is similar to the meaning of the search query.

One problem with this approach is that the semantic meaning of the search query is not always well represented in the document embeddings. A simple example makes this obvious. If you search for "What is the capital of France?", you expect the retrieval system to return chunks that are semantically similar to "capital of France". But document chunks are usually longer than the query and contain much more information, e.g. about the country France, the city of Paris, its population and so on. So the document embeddings are not well aligned with the query embedding. Less relevant chunks might be returned instead, e.g. a chunk about the capitals of South America, simply because it mentions "capitals" a lot.

For domains the embedding model was trained on, this is not much of an issue. It becomes a real problem in out-of-domain scenarios, for example with highly technical topics, mainly because the embedding model can't encode the specifics of out-of-domain semantic meaning.

HyDE addresses this gap. It generates hypothetical documents from the search query, embeds them, and uses those embeddings to retrieve relevant documents from the knowledge base. The hypothetical embeddings steer retrieval towards documents that are more likely to contain the information you need, which improves the overall performance of the RAG system.

How does HyDE work?

The HyDE process is quite simple:

  1. Use a large language model like GPT-3.5 to generate fake documents based on the search query. GPT-3.5 is prompted to "write a passage containing information about the search query". In more technical domains, it could be prompted to "write a maintenance manual for the part mentioned in the search query".

  2. Use an embedding model to encode these fake documents into embeddings.

  3. Use vector similarity search to find the document chunks in your knowledge base that are most similar to the hypothetical document embeddings. We don't use the search query to find relevant documents, but the fake HyDE ones.

  4. Use the retrieved document chunks to generate the final response.

The technique comes from the paper Precise Zero-Shot Dense Retrieval without Relevance Labels by Luyu Gao, Xueguang Ma, Jimmy Lin and Jamie Callan (2022). The figure below is taken from it: an instruction plus the query goes to GPT, which writes a generated document, and the Contriever retriever uses that document to find real documents.

HyDE process from the paper 'Precise Zero-Shot Dense Retrieval without Relevance Labels'

NOTE: The fake documents are only used to find document chunks in your knowledge base. They are not sent to the answer LLM and not used to generate the final response.

Example: using HyDE with LlamaIndex

In this example, we use the LlamaIndex library to implement HyDE.

LlamaIndex is an open-source Python library for building RAG systems over unstructured or semi-structured data. You build an index over your data and then retrieve relevant information with natural language queries, using GPT-style models.

LlamaIndex comes with built-in data loaders for sources like text files, CSV files, JSON files and web pages. It can build different types of indexes, such as vector stores, keyword tables and knowledge graphs, and it offers query interfaces to retrieve information from them with natural language. Query transformations let you modify or preprocess a query before it is used to retrieve information from the index. LlamaIndex also integrates with GPT-style language models like OpenAI's GPT-3 to generate responses from the retrieved information, and most of these parts can be customized.

For HyDE, we mostly need two of these features: query transformations and the language model integration.

Before getting started, download the example document from our website.

First, install the LlamaIndex library:

1pip install llama-index

Next, import the required modules and set up logging.

1import logging
2import sys
3
4from llama_index.core import PromptTemplate, VectorStoreIndex, SimpleDirectoryReader
5from llama_index.core.indices.query.query_transform import HyDEQueryTransform
6from llama_index.core.query_engine import TransformQueryEngine
7
8logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
9logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

For this example, we use a simple vector store index to store the document embeddings.

1documents = SimpleDirectoryReader(
2 input_files=["./HypotheticalSoftwareSDK.pdf"]
3 ).load_data()
4index = VectorStoreIndex.from_documents(documents)

Now we set up the HyDE query transformation and wrap the index's regular query engine in a TransformQueryEngine.

1query_engine = index.as_query_engine()
2hyde = HyDEQueryTransform(include_original=True)
3hyde_query_engine = TransformQueryEngine(query_engine, hyde)

Note the include_original=True parameter in the HyDEQueryTransform. It specifies whether the original query is included in the embedding strings for similarity search. With False, only the hypothetical documents are used for retrieval. With True, the hypothetical documents are combined with the original query.

Now we can use the query engine to generate responses or to retrieve documents.

1query = "How to install the software development kit?"
2response = hyde_query_engine.query(query)
3print(response)

To retrieve the relevant documents for the query instead:

1query = "How to install the software development kit?"
2query_bundle = hyde(query)
3
4retrieved_documents = hyde_query_engine.retrieve(query_bundle)
5print(retrieved_documents)

Finally, you can print the generated fake documents to inspect what HyDE actually searched with.

1query_bundle = hyde(query)
2hyde_docs = query_bundle.embedding_strs
3print(hyde_docs)

Customizing the prompt to create the HyDE documents

The default prompt LlamaIndex uses to generate the fake documents is:

1Please write a passage to answer the question\nTry to include as many key details as possible.\n\n\n{context_str}\n\n\nPassage:"""\n

This is a good starting point, but you often want more control. If your domain is highly technical, you might want to give the language model more context, so it generates better and more relevant fakes.

You might also want to be more specific about the type of document to generate. "Passage" can be too generic; "manual page" or "installation guide" might fit your knowledge base better.

To use a custom prompt, wrap it in a PromptTemplate and pass it as the hyde_prompt parameter to the HyDEQueryTransform constructor. Keep the {context_str} placeholder: LlamaIndex fills it with the user's query.

1prompt = PromptTemplate(
2 "Please write a manual page for the software development kit\nTry to include as many key details as possible.\n\n\n{context_str}\n\n\nManual Page:\n"
3)
4hyde = HyDEQueryTransform(include_original=True, hyde_prompt=prompt)

Conclusion: when to use HyDE

Hypothetical Document Embeddings (HyDE) improve RAG retrieval by searching with the embedding of an LLM-generated answer instead of the raw query. The fake answer is closer to what real documents look like, so it can guide retrieval towards more relevant chunks and, in turn, better answers.

In LlamaIndex, HyDE takes only a few lines: build a vector store index, wrap the query engine in a TransformQueryEngine with a HyDEQueryTransform, and query as usual.

Spend some time on the prompt that generates the hypothetical documents. Tailoring it to your domain and document type (a manual page rather than a generic passage) tends to produce more relevant fakes and better retrieval results. HyDE also combines well with a second retrieval step: retrieve more chunks than you need with the fake document, then let a ColBERT reranker pick the best ones.

HyDE is most interesting for out-of-domain scenarios, such as highly technical documentation, where embedding models struggle to capture the meaning of a short query. It is not a guaranteed win, though. Before you put it in production, evaluate it on your own data and compare retrieval quality with and without HyDE. Our guide on evaluating RAG pipelines with Ragas shows one way to do that.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.