DSPy Tutorial: Build Better AI Systems with Automated Prompt Optimization

On this page
DSPy is a framework that replaces hand-written prompts with code. You declare what each step of your LLM pipeline takes in and puts out, define a metric, and let a DSPy optimizer find the prompt that scores best on your data. This tutorial walks through DSPy's 8-step workflow and then builds a text summarizer: the unoptimized pipeline scored 42% on our metric, and after running the BootstrapFewShotWithRandomSearch optimizer it scored 68%.
Why bother? Because manual prompt engineering is broken. After hours of work you finally get your LLM to produce exactly what you want: clear, accurate, perfectly formatted. You lean back, feeling victorious, and the very next run produces something completely different.
This "works once, fails often" problem is the daily reality of prompt engineering. It's like trying to nail jelly to a wall. Because LLMs are random, a prompt that seems to have cracked the problem is often just a lucky hit, not a reproducible solution. The usual process is tweaking words and contexts by gut feeling and trial and error, only to find that the prompts need a complete rework when you switch models or change the use case slightly.
DSPy takes a systematic, programmatic approach instead. It uses machine learning techniques to optimize prompts automatically, which turns prompt engineering from guesswork into a reproducible process that carries over to different language models and use cases.
Update (November 2026): The code in this post was written for DSPy 2.5, and current dspy releases no longer offer the pgvector extra, so pin dspy[pgvector]==2.5.20 if you want the exact setup below. DSPy 2.6 removed the old dsp clients such as dspy.OpenAI used in the tweet metric, and OpenAI shut down gpt-4-1106-preview in October 2026; use dspy.LM with a current model, as in the summarizer code. DSPy Assertions are deprecated in favour of dspy.Refine and dspy.BestOfN. The current docs, including the installation guide, are at dspy.ai.
Installing DSPy
First, install the package. We install base DSPy along with the pgvector extras, which you need for optimizing prompts in combination with pgvector. We are big fans of PostgreSQL and pgvector.
There are many other extras. The official installation guide for DSPy 2.5 lists them.
The 8 steps of the DSPy workflow
The DSPy 2.5 documentation describes 8 steps for building a DSPy program. We go through them in more detail here, because the example later follows them one by one.
We'll throw a lot of new concepts at you, so bear with us. Many things will get clearer in the next chapter, when we run our first example.
1. Define your task
The first step is to define clearly the problem you're trying to solve. Yes, I know this sounds boring, but it matters, for DSPy and for any other LLM project.
Consider the following aspects:
- The expected input and output behavior. What type of system are you building (a chatbot, a code assistant, an information extraction system)?
- Your quality and cost requirements. What are your budget constraints and performance targets? Our list of ways to reduce LLM costs helps with the budget side.
- Which language model suits your task (e.g., GPT-3.5, Mistral-7B, GPT-4-turbo)? If you don't know yet, start with a state-of-the-art model, at the time of writing (November 2024) GPT-4o or Llama 3.1 70B. Downsizing comes later, for example with LLM routing.
Pro tip: Write 3-4 example inputs and outputs to help you picture your task.
Note: Write these things down. We are not even kidding.
2. Design your DSPy pipeline
Next, outline the structure of your DSPy program:
- Work out the steps needed to solve your problem, as a sequence of operations. What steps would you, as a human, need to take? An LLM has to take the same steps.
- Decide whether you need additional tools (e.g., retrieval, a calculator, API integrations).
- Start simple: begin with a single
dspy.ChainOfThoughtmodule and add complexity step by step.
Again, write this down!
3. Explore examples
Run your initial examples through the pipeline:
- Use a powerful LLM for exploration to see what's possible
- Test with different LLMs to compare performance
- Find where the simple version falls short. Where is a simple prompt not enough?
- Keep track of interesting examples, both easy and challenging ones
- Run each of your simple examples 5 times. Again, not kidding. Just because it works once doesn't mean it will always work. Note down the cases where it failed on repetition.
4. Prepare training data
Now we know what to build, what to expect and where simple prompts reach their limits. Next comes optimization.
Optimization needs a dataset, and this is where DSPy's main strength lies.
For the optimizers to work, we need examples. Good examples. By examples we mean questions and answers, or inputs and the outputs your pipeline should produce. Say you build a simple chatbot. Then you need a set of questions and correct answers. In production the LLM produces the answers, but for the optimization step we have to "show" it what a good answer looks like.
If you build a classification system, your test data needs sample texts and their correct classifications.
If you build a RAG system, you need a question, the relevant context and the correct answer for each user question.
You get the idea.
Prepare your training and validation data:
- Aim for 50-100 examples, with 300-500 being ideal
- Use existing datasets if available (e.g., from HuggingFace)
- Consider creating a small set of hand-crafted examples for unique tasks
- Collect data from an initial deployment if possible. Yes, deploy your system to production and let real users provide real-world use cases. You officially have permission to test in production!
- Strong LLMs can also create synthetic data, but hand-crafted examples are preferable.
Note: When preparing the data, look ahead to the metric section, because your validation data has to match the format of your metric. For example, if your metric gives a score from 0 to 10, your validation data also needs a column with a score per row. Alternatively, you can use your metric program to calculate a score for your validation data. The example below does exactly that.
5. Define a DSPy metric
Before we can run any optimization, we need a way to validate the results: a metric. In DSPy, metrics guide both the evaluation and the optimization of your language model pipeline. A well-defined metric lets you track progress and gives DSPy something to optimize for.
In DSPy, a metric is a function that takes two inputs:
- An example from your dataset
- The output (prediction) from your DSPy program
The metric function returns a score for the quality of the output. The score can be a float, integer or boolean, depending on your task. (For evaluating RAG pipelines specifically, see our post on RAG evaluation with Ragas.)
There are three main types of metrics you can use in DSPy.
Simple metrics
For straightforward tasks, you can use basic metrics such as:
- Accuracy
- Exact match
- F1 score
Here's a simple metric that compares the predicted answer to the correct answer:
DSPy also provides built-in utility metrics:
dspy.evaluate.metrics.answer_exact_matchdspy.evaluate.metrics.answer_passage_match
Metrics that check multiple properties
More sophisticated applications, especially those with long-form outputs, may need a metric that checks several properties. This example evaluates both answer accuracy and context relevance:
Using AI feedback as a metric
For complex, long-form outputs, you can (and most likely want to) use feedback from language models to assess several dimensions of quality. The reason: it's hard, often impossible, to judge whether a natural language answer is correct with code alone. You usually need an LLM to decide whether the answer is right.
Here's an example that uses GPT-4 to evaluate generated tweets (shamelessly taken from DSPy's documentation).
This metric checks whether the tweet:
- Correctly answers the given question
- Is engaging
- Stays within Twitter's character limit
Note: The avid reader might object: if I need an LLM to judge my other LLM, who tells me that the second LLM is correct? Fair point. That's why DSPy lets you 'compile' the metric pipeline as well. Read more about it in the last section here.
Refining DSPy metrics iteratively
Defining a good metric is an iterative process. Start with a simple metric, run evaluations, analyze the results and refine the metric based on what you learn. Over a few rounds, you'll end up with a more complete and accurate way to assess your DSPy program's performance.
In short, develop a way to measure the quality of your system's outputs:
- Start with simple metrics for basic tasks (e.g., accuracy, F1 score)
- For complex tasks, create a DSPy program that evaluates multiple output properties
- Iterate on your metric definition as you refine your system
6. Collect preliminary evaluations
Creating the data and the metric are the most important and the most time-consuming steps. From here on you can relax a bit.
Get a sense of your pipeline's performance and note down the metric results before optimization. This is your baseline.
- Run evaluations on your pipeline using your data and metric
- Analyze outputs and scores to find major issues
- Establish a baseline for measuring improvements
An observability tool such as Langfuse keeps these baseline runs as datasets and traces, so you can compare them with the runs after optimization.
7. Optimize with a DSPy optimizer
Finally, optimization. DSPy has quite a few optimizers. Choose one based on how much data you have:
- For very limited data (≈10 examples): Use
BootstrapFewShot - For moderate data (≈50 examples): Try
BootstrapFewShotWithRandomSearch - For larger datasets (300+ examples): Implement
MIPROv2 - To optimize for efficiency with a smaller LLM: Apply
BootstrapFinetune
8. Iterate and refine
After the first optimization:
- Revisit previous steps to find areas for improvement
- Consider updating your task definition, data collection, metric or program structure
- Explore advanced features like DSPy Assertions
- Experiment with multiple optimizers in sequence
Working with DSPy is an iterative process. Keep refining your approach based on the results to get the most effective LM pipeline for your task.
Note: We could generalize: Working with LLMs is an iterative process. This is not unique to DSPy.
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms
DSPy example: optimizing a text summarizer
To get started, let's take a very simple pipeline: summarizing text. You might object that this is way too simple, but think about it more carefully. A good summary finds the most important information in a text without getting too long. It sticks to the truth and keeps the tone and target group of the original. And the summarizer has to deliver consistent quality across texts with different content, lengths and complexity.
Text summarization is one of those cases where LLM projects are regularly classified as 'failures'. Early summarization attempts and prototypes produce great results. But once these programs run in production, the many flaws of simple summarization systems show up. And all too often the AI is blamed for being 'dumb' when really the prompts are bad.
Let's go through the 8 steps from above. First the plan, then the complete code.
-
Define your task: Summarize text, keep the same tone and style as the original, maximum length of 300 characters.
-
Define your pipeline: Use a simple
dspy.ChainOfThoughtmodule with an LLM. No tools, retrievers or APIs required. -
Explore examples: Homework: Run a few examples using plain openai model calls. Take a single long-form text and run a summarization prompt 5 times. See how much the results vary.
-
Define the data: Following our task definition, we need a set of long-form texts and their summaries. We could use existing datasets like CNN/DailyMail or create our own. Here we use the billsum dataset from Hugging Face, which has the columns
text,summaryandtitle.We treat the summary in this dataset as the 'gold' summary. Our own summarizer should produce a summary as close to it as possible and is judged by that.
-
Define your metric: It's practically impossible to write a purely programmatic metric that judges whether our summary is close to the gold summary, so we create an AI judge that scores the likeness.
How do you build such a metric? Scoring summaries is surprisingly complex. The most common strategy is:
- break down the text into its main ideas
- score the importance of each idea
- compare a generated summary to the key ideas. Are they represented?
-
Collect preliminary evaluations: Run the pipeline with the initial setup and note down the results.
-
Compile with a DSPy optimizer: Use the
BootstrapFewShotWithRandomSearchoptimizer to optimize the summarization prompt. -
Iterate and refine: After optimization, revisit the previous steps to identify areas for improvement. Consider updating your task definition, data collection, metric, or program structure. Experiment with multiple optimizers in sequence.
Full code for the DSPy summarization pipeline
Now we're finally ready for some code. The inline comments explain each part.
This code prints a percentage of how many summaries were good. In my case I got 42, meaning 42% of the summaries in the dev set passed our key-idea metric.
Which prompt did DSPy generate for the summaries?
You might wonder which prompt was actually used to generate these summaries. The good news: we don't need to think in prompts anymore, only in Signatures. A Signature defines what your program should do by declaring its inputs and outputs. DSPy then creates the prompt for us.
If you want to see the actual prompt, run:
This will output something like:
This output shows nicely what DSPy does under the hood.
But 42% is not good enough. To improve it, we use the BootstrapFewShotWithRandomSearch optimizer, which improves the prompt automatically.
In my example, the result was 68, a big improvement over the baseline of 42.
To compare the prompt from before the optimization with the one after, use:
What DSPy changes about prompt engineering
DSPy automates prompt and LLM parameter optimization and removes manual prompt engineering as the bottleneck. In our summarization example, the optimizer took the score from 42% to 68% without us writing a single prompt. The main shift is from manual prompts to programmatic interaction with language models: DSPy Signatures define program behavior declaratively, and optimizers find good prompts from training data.
For companies building production AI systems, this automates the most time-consuming part of LLM development. It also makes switching models much easier. Moving from GPT-4o to a self-hosted model like Llama only requires changing the DSPy configuration and re-running the optimization, instead of re-engineering every prompt by hand. That makes DSPy pipelines more reliable and easier to maintain than hand-tuned prompts.



