DSPy Tutorial: Build Better AI Systems with Automated Prompt Optimization

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 12 min read
DSPy Tutorial: Build Better AI Systems with Automated Prompt Optimization
On this page

DSPy is a framework that replaces hand-written prompts with code. You declare what each step of your LLM pipeline takes in and puts out, define a metric, and let a DSPy optimizer find the prompt that scores best on your data. This tutorial walks through DSPy's 8-step workflow and then builds a text summarizer: the unoptimized pipeline scored 42% on our metric, and after running the BootstrapFewShotWithRandomSearch optimizer it scored 68%.

Why bother? Because manual prompt engineering is broken. After hours of work you finally get your LLM to produce exactly what you want: clear, accurate, perfectly formatted. You lean back, feeling victorious, and the very next run produces something completely different.

This "works once, fails often" problem is the daily reality of prompt engineering. It's like trying to nail jelly to a wall. Because LLMs are random, a prompt that seems to have cracked the problem is often just a lucky hit, not a reproducible solution. The usual process is tweaking words and contexts by gut feeling and trial and error, only to find that the prompts need a complete rework when you switch models or change the use case slightly.

DSPy takes a systematic, programmatic approach instead. It uses machine learning techniques to optimize prompts automatically, which turns prompt engineering from guesswork into a reproducible process that carries over to different language models and use cases.

Update (November 2026): The code in this post was written for DSPy 2.5, and current dspy releases no longer offer the pgvector extra, so pin dspy[pgvector]==2.5.20 if you want the exact setup below. DSPy 2.6 removed the old dsp clients such as dspy.OpenAI used in the tweet metric, and OpenAI shut down gpt-4-1106-preview in October 2026; use dspy.LM with a current model, as in the summarizer code. DSPy Assertions are deprecated in favour of dspy.Refine and dspy.BestOfN. The current docs, including the installation guide, are at dspy.ai.

Installing DSPy

First, install the package. We install base DSPy along with the pgvector extras, which you need for optimizing prompts in combination with pgvector. We are big fans of PostgreSQL and pgvector.

1pip install dspy[pgvector]

There are many other extras. The official installation guide for DSPy 2.5 lists them.

The 8 steps of the DSPy workflow

The DSPy 2.5 documentation describes 8 steps for building a DSPy program. We go through them in more detail here, because the example later follows them one by one.

We'll throw a lot of new concepts at you, so bear with us. Many things will get clearer in the next chapter, when we run our first example.

1. Define your task

The first step is to define clearly the problem you're trying to solve. Yes, I know this sounds boring, but it matters, for DSPy and for any other LLM project.

Consider the following aspects:

  • The expected input and output behavior. What type of system are you building (a chatbot, a code assistant, an information extraction system)?
  • Your quality and cost requirements. What are your budget constraints and performance targets? Our list of ways to reduce LLM costs helps with the budget side.
  • Which language model suits your task (e.g., GPT-3.5, Mistral-7B, GPT-4-turbo)? If you don't know yet, start with a state-of-the-art model, at the time of writing (November 2024) GPT-4o or Llama 3.1 70B. Downsizing comes later, for example with LLM routing.

Pro tip: Write 3-4 example inputs and outputs to help you picture your task.

Note: Write these things down. We are not even kidding.

2. Design your DSPy pipeline

Next, outline the structure of your DSPy program:

  • Work out the steps needed to solve your problem, as a sequence of operations. What steps would you, as a human, need to take? An LLM has to take the same steps.
  • Decide whether you need additional tools (e.g., retrieval, a calculator, API integrations).
  • Start simple: begin with a single dspy.ChainOfThought module and add complexity step by step.

Again, write this down!

3. Explore examples

Run your initial examples through the pipeline:

  • Use a powerful LLM for exploration to see what's possible
  • Test with different LLMs to compare performance
  • Find where the simple version falls short. Where is a simple prompt not enough?
  • Keep track of interesting examples, both easy and challenging ones
  • Run each of your simple examples 5 times. Again, not kidding. Just because it works once doesn't mean it will always work. Note down the cases where it failed on repetition.

4. Prepare training data

Now we know what to build, what to expect and where simple prompts reach their limits. Next comes optimization.

Optimization needs a dataset, and this is where DSPy's main strength lies.

For the optimizers to work, we need examples. Good examples. By examples we mean questions and answers, or inputs and the outputs your pipeline should produce. Say you build a simple chatbot. Then you need a set of questions and correct answers. In production the LLM produces the answers, but for the optimization step we have to "show" it what a good answer looks like.

If you build a classification system, your test data needs sample texts and their correct classifications.

If you build a RAG system, you need a question, the relevant context and the correct answer for each user question.

You get the idea.

Prepare your training and validation data:

  • Aim for 50-100 examples, with 300-500 being ideal
  • Use existing datasets if available (e.g., from HuggingFace)
  • Consider creating a small set of hand-crafted examples for unique tasks
  • Collect data from an initial deployment if possible. Yes, deploy your system to production and let real users provide real-world use cases. You officially have permission to test in production!
  • Strong LLMs can also create synthetic data, but hand-crafted examples are preferable.

Note: When preparing the data, look ahead to the metric section, because your validation data has to match the format of your metric. For example, if your metric gives a score from 0 to 10, your validation data also needs a column with a score per row. Alternatively, you can use your metric program to calculate a score for your validation data. The example below does exactly that.

5. Define a DSPy metric

Before we can run any optimization, we need a way to validate the results: a metric. In DSPy, metrics guide both the evaluation and the optimization of your language model pipeline. A well-defined metric lets you track progress and gives DSPy something to optimize for.

In DSPy, a metric is a function that takes two inputs:

  • An example from your dataset
  • The output (prediction) from your DSPy program

The metric function returns a score for the quality of the output. The score can be a float, integer or boolean, depending on your task. (For evaluating RAG pipelines specifically, see our post on RAG evaluation with Ragas.)

There are three main types of metrics you can use in DSPy.

Simple metrics

For straightforward tasks, you can use basic metrics such as:

  • Accuracy
  • Exact match
  • F1 score

Here's a simple metric that compares the predicted answer to the correct answer:

1def validate_answer(example, pred):
2 return example.answer.lower() == pred.answer.lower()

DSPy also provides built-in utility metrics:

  • dspy.evaluate.metrics.answer_exact_match
  • dspy.evaluate.metrics.answer_passage_match

Metrics that check multiple properties

More sophisticated applications, especially those with long-form outputs, may need a metric that checks several properties. This example evaluates both answer accuracy and context relevance:

1def validate_context_and_answer(example, pred, trace=None):
2 # check the gold label and the predicted answer are the same
3 answer_match = example.answer.lower() == pred.answer.lower()
4 # check the predicted answer comes from one of the retrieved contexts
5 context_match = any((pred.answer.lower() in c) for c in pred.context)
6
7 if trace is None: # if we're doing evaluation or optimization
8 return (answer_match + context_match) / 2.0
9 else: # if we're doing bootstrapping, i.e. self-generating good demonstrations of each step
10 return answer_match and context_match

Using AI feedback as a metric

For complex, long-form outputs, you can (and most likely want to) use feedback from language models to assess several dimensions of quality. The reason: it's hard, often impossible, to judge whether a natural language answer is correct with code alone. You usually need an LLM to decide whether the answer is right.

Here's an example that uses GPT-4 to evaluate generated tweets (shamelessly taken from DSPy's documentation).

1class Assess(dspy.Signature):
2 """Assess the quality of a tweet along the specified dimension."""
3 assessed_text = dspy.InputField()
4 assessment_question = dspy.InputField()
5 assessment_answer = dspy.OutputField(desc="Yes or No")
6
7gpt4T = dspy.OpenAI(model='gpt-4-1106-preview', max_tokens=1000, model_type='chat')
8
9def metric(gold, pred):
10 question, answer, tweet = gold.question, gold.answer, pred.output
11
12 engaging = "Does the assessed text make for a self-contained, engaging tweet?"
13 correct = f"The text should answer `{question}` with `{answer}`. Does the assessed text contain this answer?"
14
15 with dspy.context(lm=gpt4T):
16 correct = dspy.Predict(Assess)(assessed_text=tweet, assessment_question=correct)
17 engaging = dspy.Predict(Assess)(assessed_text=tweet, assessment_question=engaging)
18
19 correct, engaging = [m.assessment_answer.lower() == 'yes' for m in [correct, engaging]]
20 score = (correct + engaging) if correct and (len(tweet) <= 280) else 0
21
22 return score / 2.0

This metric checks whether the tweet:

  1. Correctly answers the given question
  2. Is engaging
  3. Stays within Twitter's character limit

Note: The avid reader might object: if I need an LLM to judge my other LLM, who tells me that the second LLM is correct? Fair point. That's why DSPy lets you 'compile' the metric pipeline as well. Read more about it in the last section here.

Refining DSPy metrics iteratively

Defining a good metric is an iterative process. Start with a simple metric, run evaluations, analyze the results and refine the metric based on what you learn. Over a few rounds, you'll end up with a more complete and accurate way to assess your DSPy program's performance.

In short, develop a way to measure the quality of your system's outputs:

  • Start with simple metrics for basic tasks (e.g., accuracy, F1 score)
  • For complex tasks, create a DSPy program that evaluates multiple output properties
  • Iterate on your metric definition as you refine your system

6. Collect preliminary evaluations

Creating the data and the metric are the most important and the most time-consuming steps. From here on you can relax a bit.

Get a sense of your pipeline's performance and note down the metric results before optimization. This is your baseline.

  • Run evaluations on your pipeline using your data and metric
  • Analyze outputs and scores to find major issues
  • Establish a baseline for measuring improvements

An observability tool such as Langfuse keeps these baseline runs as datasets and traces, so you can compare them with the runs after optimization.

7. Optimize with a DSPy optimizer

Finally, optimization. DSPy has quite a few optimizers. Choose one based on how much data you have:

  • For very limited data (≈10 examples): Use BootstrapFewShot
  • For moderate data (≈50 examples): Try BootstrapFewShotWithRandomSearch
  • For larger datasets (300+ examples): Implement MIPROv2
  • To optimize for efficiency with a smaller LLM: Apply BootstrapFinetune

8. Iterate and refine

After the first optimization:

  • Revisit previous steps to find areas for improvement
  • Consider updating your task definition, data collection, metric or program structure
  • Explore advanced features like DSPy Assertions
  • Experiment with multiple optimizers in sequence

Working with DSPy is an iterative process. Keep refining your approach based on the results to get the most effective LM pipeline for your task.

Note: We could generalize: Working with LLMs is an iterative process. This is not unique to DSPy.


Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

DSPy example: optimizing a text summarizer

To get started, let's take a very simple pipeline: summarizing text. You might object that this is way too simple, but think about it more carefully. A good summary finds the most important information in a text without getting too long. It sticks to the truth and keeps the tone and target group of the original. And the summarizer has to deliver consistent quality across texts with different content, lengths and complexity.

Text summarization is one of those cases where LLM projects are regularly classified as 'failures'. Early summarization attempts and prototypes produce great results. But once these programs run in production, the many flaws of simple summarization systems show up. And all too often the AI is blamed for being 'dumb' when really the prompts are bad.

Let's go through the 8 steps from above. First the plan, then the complete code.

  1. Define your task: Summarize text, keep the same tone and style as the original, maximum length of 300 characters.

  2. Define your pipeline: Use a simple dspy.ChainOfThought module with an LLM. No tools, retrievers or APIs required.

  3. Explore examples: Homework: Run a few examples using plain openai model calls. Take a single long-form text and run a summarization prompt 5 times. See how much the results vary.

  4. Define the data: Following our task definition, we need a set of long-form texts and their summaries. We could use existing datasets like CNN/DailyMail or create our own. Here we use the billsum dataset from Hugging Face, which has the columns text, summary and title.

    We treat the summary in this dataset as the 'gold' summary. Our own summarizer should produce a summary as close to it as possible and is judged by that.

  5. Define your metric: It's practically impossible to write a purely programmatic metric that judges whether our summary is close to the gold summary, so we create an AI judge that scores the likeness.

    How do you build such a metric? Scoring summaries is surprisingly complex. The most common strategy is:

    • break down the text into its main ideas
    • score the importance of each idea
    • compare a generated summary to the key ideas. Are they represented?
  6. Collect preliminary evaluations: Run the pipeline with the initial setup and note down the results.

  7. Compile with a DSPy optimizer: Use the BootstrapFewShotWithRandomSearch optimizer to optimize the summarization prompt.

  8. Iterate and refine: After optimization, revisit the previous steps to identify areas for improvement. Consider updating your task definition, data collection, metric, or program structure. Experiment with multiple optimizers in sequence.

Full code for the DSPy summarization pipeline

Now we're finally ready for some code. The inline comments explain each part.

1import pandas as pd
2import dspy
3import os
4
5os.environ["OPENAI_API_KEY"] = "sk-your-key"
6
7# 4. Defining the data
8
9
10splits = {
11 "train": "data/train-00000-of-00001.parquet",
12 "test": "data/test-00000-of-00001.parquet",
13}
14df = pd.read_parquet("hf://datasets/FiscalNote/billsum/" + splits["train"])
15# Let's divide the dataset into train, dev and validation sets.
16df_train = df[0:40]
17df_dev = df[40:80]
18df_val = df[80:120]
19# Note: This dataset only has columns 'text', 'summary' and 'title', but no 'score'
20# which is required for our validation. We'll deal with that later.
21
22dataset_train = df_train.apply(
23 lambda row: dspy.Example(text_section=row["text"], summary=row["summary"]), axis=1
24).tolist()
25
26dataset_dev = df_dev.apply(
27 lambda row: dspy.Example(text_section=row["text"], summary=row["summary"]), axis=1
28).tolist()
29
30dataset_val = df_val.apply(
31 lambda row: dspy.Example(text_section=row["text"], summary=row["summary"]), axis=1
32).tolist()
33
34trainset = [x.with_inputs("text_section") for x in dataset_train]
35valset = [x.with_inputs("text_section") for x in dataset_val]
36devset = [x.with_inputs("text_section") for x in dataset_dev]
37
38
39# 5. Define your metric
40class KeyIdeas(dspy.Signature):
41 """
42 You'll get a long-form text section. Break it down into key ideas.
43 Rate the importance of each key idea with High, Medium or Low.
44 """
45
46 text_section = dspy.InputField()
47 key_ideas: str = dspy.OutputField(
48 desc="list of key ideas, with one key idea per line."
49 + "Each key idea get's a number."
50 + "e.g. 1. <Idea here>: High"
51 )
52 importances: list[str] = dspy.OutputField(
53 desc="list of importance levels for each key idea."
54 + 'e.g. ["High", "Medium", "Low"].'
55 )
56
57
58class SummaryRating(dspy.Signature):
59 """
60 You get an auto-generated summary. Compare it to the key ides from
61 the text section it was generated from.
62 Create a binary score for each key idea. 1, if the key idea is present
63 in the summary, 0 if not.
64 Finally, create an overall score based on the binary scores.
65 """
66
67 key_ideas: str = dspy.InputField(
68 desc="key ideas present in the text section to summarize"
69 )
70 summary: str = dspy.InputField()
71 binary_scores: list[bool] = dspy.OutputField(
72 desc="list of binary scores for each key idea, " + "e.g. [1, 0, 1]"
73 )
74 overall_score: float = dspy.OutputField(
75 desc="overall score for the summary out of 1.0"
76 )
77
78
79## We are still in step 5, defining the metric. This below is now the
80# final DSPy program used to calculate the metric.
81class Metric(dspy.Module):
82 """
83 Compute a score for the correctness of a summary.
84 """
85
86 def __init__(self):
87 self.extracted_key_ideas = dspy.ChainOfThought(KeyIdeas)
88 self.rate = dspy.ChainOfThought(SummaryRating)
89
90 def forward(self, example, pred, trace=None):
91 extracted_key_ideas = self.extracted_key_ideas(
92 text_section=example.text_section
93 )
94 key_ideas = extracted_key_ideas.key_ideas
95 importances = extracted_key_ideas.importances
96
97 scores = self.rate(
98 key_ideas=key_ideas,
99 summary=pred.summary,
100 )
101
102 try:
103 weight_map = {"High": 1.0, "Medium": 0.7}
104 score = sum(
105 weight_map.get(g, 0.2) * int(b)
106 for g, b in zip(importances, scores.binary_scores)
107 )
108 score /= sum(weight_map.get(g, 0.2) for g in importances)
109 except Exception:
110 score = float(scores.overall_score)
111
112 return score if trace is None else score >= 0.75
113
114
115def metric(gold, pred, trace=None):
116 metric_program = Metric()
117 example = dspy.Example(text_section=gold.text_section)
118 predicted = dspy.Example(summary=pred)
119 pred_score = metric_program(example=example, pred=predicted)
120 # In the next line we use our own metric program to generate a score for our
121 # gold data. This ensures that both, our gold data as well as our
122 # predicted data are scored by the same program. However, requires
123 # our metric program to be quite good. Keep that in mind.
124 gold_score = metric_program(example=example, pred=gold, trace=None)
125 # check if they are almost equal
126 return abs(float(gold_score) - float(pred_score)) < 0.2
127
128
129# 6. Collect preliminary evaluations
130## Prerequisite: create our pipeline program
131## Note: DSPy programs are always classes with an __init__ and a forward method
132
133
134class SummarizeSignature(dspy.Signature):
135 """
136 Given a text section, generate a summary.
137 """
138
139 text_section = dspy.InputField(desc="a text to summarize")
140 summary: str = dspy.OutputField(desc="a concise summary of the text section")
141
142
143class Summarize(dspy.Module):
144 def __init__(self):
145 self.summarize = dspy.ChainOfThought(SummarizeSignature)
146
147 def forward(self, text_section: str):
148 summary = self.summarize(text_section=text_section)
149 # Note: You can add multiple dspy modules here, for multi-step pipelines
150 # If you add multiple modules, DSPy will optimize all modules in one go
151 return summary
152
153
154## Next, we can define the LLM we want to use. We want to use gpt-4o-mini
155# for summarizing and gpt-4o as teacher model - for finetuning our summary prompt.
156lm = dspy.LM("openai/gpt-4o-mini", max_tokens=1000, cache=False)
157gpt4T = dspy.LM("openai/gpt-4o", max_tokens=1000, cache=False)
158dspy.settings.configure(lm=lm)
159
160# And finally, let's create the program (the actual pipeline) we want to
161# validate and optimize.
162program = Summarize()
163
164evaluate = dspy.Evaluate(
165 devset=devset,
166 metric=metric,
167 display_progress=True,
168 display_table=True,
169 provide_traceback=True,
170)
171
172res = evaluate(program, devset=devset)
173print(res)

This code prints a percentage of how many summaries were good. In my case I got 42, meaning 42% of the summaries in the dev set passed our key-idea metric.

Which prompt did DSPy generate for the summaries?

You might wonder which prompt was actually used to generate these summaries. The good news: we don't need to think in prompts anymore, only in Signatures. A Signature defines what your program should do by declaring its inputs and outputs. DSPy then creates the prompt for us.

If you want to see the actual prompt, run:

1dspy.inspect_history(n=1)

This will output something like:

1[2024-10-30T19:48:46.135956]
2
3System message:
4
5Your input fields are:
61. `key_ideas` (str): key ideas present in the text section to summarize
72. `summary` (str)
8
9Your output fields are:
101. `reasoning` (str)
112. `binary_scores` (list[bool]): list of binary scores for each key idea, e.g. [1, 0, 1]
123. `overall_score` (float): overall score for the summary out of 1.0
13
14All interactions will be structured in the following way, with the appropriate values filled in.
15
16[[ ## key_ideas ## ]]
17{key_ideas}
18
19[[ ## summary ## ]]
20{summary}
21
22[[ ## reasoning ## ]]
23{reasoning}
24
25[[ ## binary_scores ## ]]
26{binary_scores} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "boolean"}}
27
28[[ ## overall_score ## ]]
29{overall_score} # note: the value you produce must be a single float value
30
31[[ ## completed ## ]]
32
33In adhering to this structure, your objective is:
34 You get an auto-generated summary. Compare it to the key ides from
35 the text section it was generated from.
36 Create a binary score for each key idea. 1, if the key idea is present
37 in the summary, 0 if not.
38 Finally, create an overall score based on the binary scores.
39
40
41User message:
42
43....

This output shows nicely what DSPy does under the hood.

But 42% is not good enough. To improve it, we use the BootstrapFewShotWithRandomSearch optimizer, which improves the prompt automatically.

1tp = dspy.BootstrapFewShotWithRandomSearch(
2 metric=metric,
3 num_threads=24,
4 max_bootstrapped_demos=8,
5 max_labeled_demos=8,
6 num_candidate_programs=10,
7 teacher_settings=dict(lm=gpt4T),
8)
9
10optimized_program = tp.compile(
11 Summarize(),
12 trainset=trainset,
13 valset=valset,
14)
15
16result = evaluate(optimized_program)

In my example, the result was 68, a big improvement over the baseline of 42.

To compare the prompt from before the optimization with the one after, use:

1dspy.inspect_history(n=2)

What DSPy changes about prompt engineering

DSPy automates prompt and LLM parameter optimization and removes manual prompt engineering as the bottleneck. In our summarization example, the optimizer took the score from 42% to 68% without us writing a single prompt. The main shift is from manual prompts to programmatic interaction with language models: DSPy Signatures define program behavior declaratively, and optimizers find good prompts from training data.

For companies building production AI systems, this automates the most time-consuming part of LLM development. It also makes switching models much easier. Moving from GPT-4o to a self-hosted model like Llama only requires changing the DSPy configuration and re-running the optimization, instead of re-engineering every prompt by hand. That makes DSPy pipelines more reliable and easier to maintain than hand-tuned prompts.

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.