RAG Evaluation with Ragas: How to Measure RAG Pipeline Quality

On this page
TLDR: If you are just looking for the code example without any explanations, see the full script at the end of this blog post.
Ragas is a Python framework for evaluating Retrieval Augmented Generation (RAG) pipelines. It works in two steps. First, it uses an LLM to generate a test set of questions and ground-truth answers from your own documents. Then it runs your pipeline's answers through an LLM-based evaluation and scores them on four metrics: answer relevancy, faithfulness, context recall and context precision.
This guide sets up Ragas 0.1 with LangChain and OpenAI, walks through test set generation and evaluation, and ends with best practices for interpreting the results, such as averaging the worst three of several evaluation runs.
Why evaluate at all? A RAG pipeline adds external knowledge to an LLM by retrieving relevant documents and passing them to the model as context. That makes answers more accurate and specific, but it also adds failure points: the retriever can return irrelevant chunks, and the LLM can ignore or misread the context. Without measurements, you can't tell whether a change to chunking, embeddings or prompts made things better or worse. For a broader introduction to RAG, see our previous blog post.
Update (November 2026): This guide uses Ragas 0.1. Ragas 0.2 rebuilt
test set generation around a knowledge graph, and current releases no longer
have the ragas.testset.generator and ragas.testset.evolutions modules or
the ground_truths column used below; see the v0.1 to v0.2 migration
guide.
OpenAI has also shut down gpt-4-turbo-preview, gpt-4 and gpt-3.5-turbo
(deprecations page), which
the code and the TestsetGenerator.with_openai() defaults use, so pass current
model names to ChatOpenAI and with_openai(generator_llm=..., critic_llm=...).
The langchain_community import of ChatOpenAI is deprecated in favour of
from langchain_openai import ChatOpenAI.
High-level overview of Ragas
Ragas has several modules for evaluating RAG. The two main ones, and the ones we use in this guide, are:
-
TestsetGenerator: generates test sets for evaluating RAG pipelines. It supports several question types, including simple, reasoning and multi-context questions. -
evaluate: evaluates RAG pipelines using the generated test sets. It provides several metrics, including answer relevancy, faithfulness, context recall and context precision.
The TestsetGenerator loads a set of documents or text chunks and uses an
LLM to generate questions about them, together with answers based on the
same documents. These generated answers serve as "ground truth" when you
evaluate the RAG pipeline later.
The evaluate module then scores the answers your RAG pipeline gives to
the test set questions. It uses an LLM to judge those answers and how well
the retrieved contexts fit the questions. Finally, it
compares your pipeline's answer with the ground truth answer from the
test set.
Step-by-step guide: setting up Ragas to evaluate RAG pipelines
-
Install the required dependencies
Note: The code below needs Ragas 0.1. Version 0.1 was a large refactoring, so the code won't work with older versions, and Ragas 0.2 changed the API again (see the update note at the top).
-
Import the necessary modules
-
Set up the LLM: add your OpenAI key and define which model to use. We strongly recommend a strong model (GPT-4 Turbo when we wrote this; today one of OpenAI's current flagship models, see the update note above), as its more consistent output gives you better, more comparable results.
Generating test sets with the Ragas TestsetGenerator
Writing a test set for RAG evaluation by hand is tedious, and it's the step that stops most teams from evaluating at all. The Ragas test set generator removes most of that work.
The test set generator does the following:
- Loads a set of documents and parses them into plain text
- Uses an LLM to create questions based on these documents
- Creates a result set with the columns
question,answerandcontext(contextbeing the text chunk that was used to generate the question and answer)
We use the question and answer columns as validation questions and
answers in the evaluation step, with answer being the ground truth.
Note: Instead of using Ragas test set generation, you can also
create a test set manually. Just make sure you have a dataframe with
the columns question and answer.
-
Load the documents. Ragas needs a set of either LangChain or LlamaIndex documents, so you can use any LangChain or LlamaIndex document loader. Make sure you load the same documents (or a subset of them) that your RAG pipeline under test has access to.
Note: The
DirectoryLoaderis just one of many document loaders. -
Generate the test set with the
TestsetGeneratormodule. Thedistributionsparameter defines what share of the questions each question type gets:reasoning: Questions that require reasoning based on the documents.simple: Simple questions based on the documents.multi_context: Questions generated from multiple contexts in the documents.
Note: Save the test set so you can reuse it for later evaluations. Don't use different test questions for different RAG evaluation runs; always use the same ones.
-
Now feed the generated questions to the RAG pipeline you want to test. Take the
questioncolumn of thetest_dfwe just created and let your RAG pipeline answer each question. For each question, store the question, the answer and the context, ideally in a dataframe. In the end, you should have a dataframe with the following columns:question: string; the question that was asked to your RAG pipelineanswer: string; the answer your RAG pipeline providedcontexts: list; the contexts that were used to find the answerground_truths: list; the ground truths for the question. Ground truths are the known correct answers for the question. You can use theanswergenerated by theTestsetGeneratoras ground truth, because theTestsetGeneratorcreated each question from a context that contained the answer. This needs to be a list, as theevaluatemodule expects a list of ground truths for each question.
For this guide, we cheat a little and reuse our test set as our RAG output, because setting up a RAG pipeline is not part of this guide. In your own code, replace the code below with calls to your RAG pipeline and build the dataframe as described above.
Evaluating the RAG pipeline with Ragas evaluate
With the test set and the RAG pipeline's output in place, the Ragas
evaluate module scores the pipeline. It provides several metrics,
including answer relevancy, faithfulness, context recall and context
precision:
answer_relevancy: measures how relevant the answer is to the question.faithfulness: measures how faithful the answer is to the provided context.context_recall: measures whether the retriever retrieves all the information needed to answer the question.context_precision: measures how relevant the retrieved contexts are, that is, whether the retriever returns the necessary information rather than noise.
Running the evaluation is simple now: call the evaluate method and pass
the output of your RAG pipeline, including the ground truths.
The result is a dictionary with the four evaluation metrics and their values. Use these values to compare different RAG pipelines, for example the same pipeline with and without a ColBERT reranker.
Bringing your own prompts to Ragas
Ragas comes with a good default set of LLM prompts for both test set generation and evaluation. As you get more experienced with your RAG pipeline and how to evaluate it, you might want to use your own prompts. Ragas provides an interface for that. As this interface has changed several times since I wrote this, here is the relevant page in the current Ragas documentation: Modify prompts
Best practices for RAG evaluation with Ragas
-
Use a good LLM. The quality of the LLM used for test set generation and evaluation directly determines the quality of the evaluation. Use a strong model, such as one of the current flagship models from OpenAI or Anthropic. It costs more, but it's worth it even in the short run.
-
Use a standardized question set. Evaluate with a consistent set of questions and ground truths, so you can measure improvements or regressions in performance over time.
-
Make the questions realistic. Curate the questions and ground truths so they represent the kind of queries your RAG pipeline will get in production. Otherwise the evaluation tells you little about real-world quality.
-
Run the evaluation several times and average the worst results. LLMs are probabilistic, so each run gives somewhat different results, even if the RAG pipeline stays the same. Run the evaluation multiple times and take the average of the worst three results. This conservative number tells you how the pipeline performs under less-than-ideal conditions.
-
Re-run the evaluation after every change. After each iteration on your RAG pipeline, re-run the evaluation to see the impact of the change. Automate this by adding Ragas to your Continuous Integration (CI) pipeline, so every change to the RAG pipeline is evaluated automatically. For tracing individual requests alongside these scores, see our post on LLM tracing with Opik or the open source alternative Langfuse, which lets you attach evaluation scores to individual traces.
-
Develop custom evaluation prompts. Over time, build your own set of evaluation prompts for Ragas that reflect the specific challenges of your domain. Update them regularly to cover new topics, formats and query types as your data and application change.
Conclusion
Ragas turns RAG evaluation into a repeatable process. The
TestsetGenerator creates questions and ground-truth answers from your
own documents, and evaluate scores your pipeline on answer relevancy,
faithfulness, context recall and context precision. With those numbers,
you can see whether a change to retrieval, chunking or prompts actually
helped, and catch performance regressions before they reach users.
Retrieval changes worth testing this way include late
chunking and hypothetical document
embeddings (HyDE).
To get reliable results, use a strong evaluation LLM, keep the test set fixed, run the evaluation several times and look at the worst runs, and re-run it on every change to your pipeline. The full script below is a good starting point: replace the demo "RAG pipeline" with calls to your own.
TLDR: code to validate RAG pipelines using Ragas
Further reading
- Advanced RAG: Recursive Retrieval using llamaindex
- How to securely self host your own LLM with vLLM and Caddy
- How to integrate knowledge with LLMs
- Automated prompt optimization against a metric with DSPy
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



