RAG Evaluation with Ragas: How to Measure RAG Pipeline Quality

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 8 min read
RAG Evaluation with Ragas: How to Measure RAG Pipeline Quality
On this page

TLDR: If you are just looking for the code example without any explanations, see the full script at the end of this blog post.

Ragas is a Python framework for evaluating Retrieval Augmented Generation (RAG) pipelines. It works in two steps. First, it uses an LLM to generate a test set of questions and ground-truth answers from your own documents. Then it runs your pipeline's answers through an LLM-based evaluation and scores them on four metrics: answer relevancy, faithfulness, context recall and context precision.

This guide sets up Ragas 0.1 with LangChain and OpenAI, walks through test set generation and evaluation, and ends with best practices for interpreting the results, such as averaging the worst three of several evaluation runs.

Why evaluate at all? A RAG pipeline adds external knowledge to an LLM by retrieving relevant documents and passing them to the model as context. That makes answers more accurate and specific, but it also adds failure points: the retriever can return irrelevant chunks, and the LLM can ignore or misread the context. Without measurements, you can't tell whether a change to chunking, embeddings or prompts made things better or worse. For a broader introduction to RAG, see our previous blog post.

Update (November 2026): This guide uses Ragas 0.1. Ragas 0.2 rebuilt test set generation around a knowledge graph, and current releases no longer have the ragas.testset.generator and ragas.testset.evolutions modules or the ground_truths column used below; see the v0.1 to v0.2 migration guide. OpenAI has also shut down gpt-4-turbo-preview, gpt-4 and gpt-3.5-turbo (deprecations page), which the code and the TestsetGenerator.with_openai() defaults use, so pass current model names to ChatOpenAI and with_openai(generator_llm=..., critic_llm=...). The langchain_community import of ChatOpenAI is deprecated in favour of from langchain_openai import ChatOpenAI.

High-level overview of Ragas

Ragas has several modules for evaluating RAG. The two main ones, and the ones we use in this guide, are:

  • TestsetGenerator: generates test sets for evaluating RAG pipelines. It supports several question types, including simple, reasoning and multi-context questions.

  • evaluate: evaluates RAG pipelines using the generated test sets. It provides several metrics, including answer relevancy, faithfulness, context recall and context precision.

The TestsetGenerator loads a set of documents or text chunks and uses an LLM to generate questions about them, together with answers based on the same documents. These generated answers serve as "ground truth" when you evaluate the RAG pipeline later.

The evaluate module then scores the answers your RAG pipeline gives to the test set questions. It uses an LLM to judge those answers and how well the retrieved contexts fit the questions. Finally, it compares your pipeline's answer with the ground truth answer from the test set.

Step-by-step guide: setting up Ragas to evaluate RAG pipelines

  1. Install the required dependencies

    1pip install openai langchain ragas==0.1.0rc1

    Note: The code below needs Ragas 0.1. Version 0.1 was a large refactoring, so the code won't work with older versions, and Ragas 0.2 changed the API again (see the update note at the top).

  2. Import the necessary modules

    1import os
    2from langchain_community.document_loaders import DirectoryLoader
    3from langchain_community.chat_models.openai import ChatOpenAI
    4
    5from ragas.testset.generator import TestsetGenerator
    6from ragas.testset.evolutions import simple, reasoning, multi_context
    7from ragas.metrics import (
    8 answer_relevancy,
    9 faithfulness,
    10 context_recall,
    11 context_precision,
    12)
    13import pandas as pd
    14from ragas import evaluate
    15from datasets import Dataset
  3. Set up the LLM: add your OpenAI key and define which model to use. We strongly recommend a strong model (GPT-4 Turbo when we wrote this; today one of OpenAI's current flagship models, see the update note above), as its more consistent output gives you better, more comparable results.

    1os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
    2gpt4 = ChatOpenAI(model_name="gpt-4-turbo-preview")

Generating test sets with the Ragas TestsetGenerator

Writing a test set for RAG evaluation by hand is tedious, and it's the step that stops most teams from evaluating at all. The Ragas test set generator removes most of that work.

The test set generator does the following:

  • Loads a set of documents and parses them into plain text
  • Uses an LLM to create questions based on these documents
  • Creates a result set with the columns question, answer and context (context being the text chunk that was used to generate the question and answer)

We use the question and answer columns as validation questions and answers in the evaluation step, with answer being the ground truth.

Note: Instead of using Ragas test set generation, you can also create a test set manually. Just make sure you have a dataframe with the columns question and answer.

  1. Load the documents. Ragas needs a set of either LangChain or LlamaIndex documents, so you can use any LangChain or LlamaIndex document loader. Make sure you load the same documents (or a subset of them) that your RAG pipeline under test has access to.

    1loader = DirectoryLoader("public/blog/validate-rag/demo-docs")
    2documents = loader.load()

    Note: The DirectoryLoader is just one of many document loaders.

  2. Generate the test set with the TestsetGenerator module. The distributions parameter defines what share of the questions each question type gets:

    • reasoning: Questions that require reasoning based on the documents.
    • simple: Simple questions based on the documents.
    • multi_context: Questions generated from multiple contexts in the documents.
    1generator = TestsetGenerator.with_openai()
    2testset = generator.generate_with_langchain_docs(
    3 documents,
    4 test_size=100,
    5 distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
    6)
    7test_df = testset.to_pandas()

Note: Save the test set so you can reuse it for later evaluations. Don't use different test questions for different RAG evaluation runs; always use the same ones.

  1. Now feed the generated questions to the RAG pipeline you want to test. Take the question column of the test_df we just created and let your RAG pipeline answer each question. For each question, store the question, the answer and the context, ideally in a dataframe. In the end, you should have a dataframe with the following columns:

    • question: string; the question that was asked to your RAG pipeline
    • answer: string; the answer your RAG pipeline provided
    • contexts: list; the contexts that were used to find the answer
    • ground_truths: list; the ground truths for the question. Ground truths are the known correct answers for the question. You can use the answer generated by the TestsetGenerator as ground truth, because the TestsetGenerator created each question from a context that contained the answer. This needs to be a list, as the evaluate module expects a list of ground truths for each question.

    For this guide, we cheat a little and reuse our test set as our RAG output, because setting up a RAG pipeline is not part of this guide. In your own code, replace the code below with calls to your RAG pipeline and build the dataframe as described above.

    1test_questions = test_df["question"].values.tolist()
    2test_answers = test_df["answer"].values.tolist()
    3
    4df = pd.DataFrame(
    5 {
    6 "question": test_questions,
    7 "answer": test_answers,
    8 "contexts": test_df["context"].values.tolist(),
    9 "ground_truths": test_df["answer"].values.tolist(),
    10 }
    11)
    12
    13df["contexts"] = df["contexts"].apply(lambda x: [x] if isinstance(x, str)
    14 else [])
    15df["ground_truths"] = df["ground_truths"].apply(
    16 lambda x: [x] if isinstance(x, str) else []
    17)

Evaluating the RAG pipeline with Ragas evaluate

With the test set and the RAG pipeline's output in place, the Ragas evaluate module scores the pipeline. It provides several metrics, including answer relevancy, faithfulness, context recall and context precision:

  • answer_relevancy: measures how relevant the answer is to the question.
  • faithfulness: measures how faithful the answer is to the provided context.
  • context_recall: measures whether the retriever retrieves all the information needed to answer the question.
  • context_precision: measures how relevant the retrieved contexts are, that is, whether the retriever returns the necessary information rather than noise.

Running the evaluation is simple now: call the evaluate method and pass the output of your RAG pipeline, including the ground truths.

1result = evaluate(
2 Dataset.from_pandas(df),
3 metrics=[
4 context_precision,
5 faithfulness,
6 answer_relevancy,
7 context_recall,
8 ],
9 llm=gpt4,
10)

The result is a dictionary with the four evaluation metrics and their values. Use these values to compare different RAG pipelines, for example the same pipeline with and without a ColBERT reranker.

Bringing your own prompts to Ragas

Ragas comes with a good default set of LLM prompts for both test set generation and evaluation. As you get more experienced with your RAG pipeline and how to evaluate it, you might want to use your own prompts. Ragas provides an interface for that. As this interface has changed several times since I wrote this, here is the relevant page in the current Ragas documentation: Modify prompts

Best practices for RAG evaluation with Ragas

  • Use a good LLM. The quality of the LLM used for test set generation and evaluation directly determines the quality of the evaluation. Use a strong model, such as one of the current flagship models from OpenAI or Anthropic. It costs more, but it's worth it even in the short run.

  • Use a standardized question set. Evaluate with a consistent set of questions and ground truths, so you can measure improvements or regressions in performance over time.

  • Make the questions realistic. Curate the questions and ground truths so they represent the kind of queries your RAG pipeline will get in production. Otherwise the evaluation tells you little about real-world quality.

  • Run the evaluation several times and average the worst results. LLMs are probabilistic, so each run gives somewhat different results, even if the RAG pipeline stays the same. Run the evaluation multiple times and take the average of the worst three results. This conservative number tells you how the pipeline performs under less-than-ideal conditions.

  • Re-run the evaluation after every change. After each iteration on your RAG pipeline, re-run the evaluation to see the impact of the change. Automate this by adding Ragas to your Continuous Integration (CI) pipeline, so every change to the RAG pipeline is evaluated automatically. For tracing individual requests alongside these scores, see our post on LLM tracing with Opik or the open source alternative Langfuse, which lets you attach evaluation scores to individual traces.

  • Develop custom evaluation prompts. Over time, build your own set of evaluation prompts for Ragas that reflect the specific challenges of your domain. Update them regularly to cover new topics, formats and query types as your data and application change.

Conclusion

Ragas turns RAG evaluation into a repeatable process. The TestsetGenerator creates questions and ground-truth answers from your own documents, and evaluate scores your pipeline on answer relevancy, faithfulness, context recall and context precision. With those numbers, you can see whether a change to retrieval, chunking or prompts actually helped, and catch performance regressions before they reach users. Retrieval changes worth testing this way include late chunking and hypothetical document embeddings (HyDE).

To get reliable results, use a strong evaluation LLM, keep the test set fixed, run the evaluation several times and look at the worst runs, and re-run it on every change to your pipeline. The full script below is a good starting point: replace the demo "RAG pipeline" with calls to your own.

TLDR: code to validate RAG pipelines using Ragas

1# ! pip install openai langchain ragas==0.1.0rc1
2import os
3from langchain_community.document_loaders import DirectoryLoader
4from langchain_community.chat_models.openai import ChatOpenAI
5
6from ragas.testset.generator import TestsetGenerator
7from ragas.testset.evolutions import simple, reasoning, multi_context
8from ragas.metrics import (
9 answer_relevancy,
10 faithfulness,
11 context_recall,
12 context_precision,
13)
14import pandas as pd
15from ragas import evaluate
16from datasets import Dataset
17
18os.environ["OPENAI_API_KEY"] = "sk-xxxxxxxxxxxxxxxxxxxxx"
19
20gpt4 = ChatOpenAI(model_name="gpt-4-turbo-preview")
21
22loader = DirectoryLoader("public/blog/validate-rag/demo-docs")
23documents = loader.load()
24
25for document in documents:
26 document.metadata["file_name"] = document.metadata["source"]
27
28generator = TestsetGenerator.with_openai()
29testset = generator.generate_with_langchain_docs(
30 documents,
31 test_size=10,
32 distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
33)
34
35test_df = testset.to_pandas()
36
37# Add your actual RAG pipeline retrieval here.
38# We use the created test set as our 'RAG pipeline' - just for demo purposes.
39# In the dataframe below, change answer and contexts column to the output of your
40# RAG pipeline.
41test_questions = test_df["question"].values.tolist()
42test_answers = test_df["answer"].values.tolist()
43
44df = pd.DataFrame(
45 {
46 "question": test_questions,
47 "answer": test_answers,
48 "contexts": test_df["context"].values.tolist(),
49 "ground_truths": test_df["answer"].values.tolist(),
50 }
51)
52
53df["contexts"] = df["contexts"].apply(lambda x: [x] if isinstance(x, str) else [])
54df["ground_truths"] = df["ground_truths"].apply(
55 lambda x: [x] if isinstance(x, str) else []
56)
57
58result = evaluate(
59 Dataset.from_pandas(df),
60 metrics=[
61 context_precision,
62 faithfulness,
63 answer_relevancy,
64 context_recall,
65 ],
66 llm=gpt4,
67)
68result

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.