RouteLLM: Cut LLM Costs by Routing Prompts to the Right Model

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 9 min read
RouteLLM: Cut LLM Costs by Routing Prompts to the Right Model
On this page

LLM routing sends each prompt to the model that fits it: simple prompts go to a small, cheap model, complex ones to a strong, expensive model. You don't have to build the router yourself. RouteLLM from LMSYS is an open-source framework with 4 pre-trained routers. In LMSYS's benchmarks it cut costs by over 85% on MT Bench while keeping 95% of GPT-4's performance. This post explains how RouteLLM works, how to set it up and calibrate it, how to route between providers like OpenAI and Google Gemini, and how to run it as a drop-in replacement for the OpenAI API.

Update (November 2026): The model landscape has changed a lot since August 2024, so read the model recommendations below as 2024 judgments. Google shut down gemini-1.5-flash on September 29, 2025, and OpenAI will remove the gpt-4o and gpt-4o-mini snapshots from its API on January 14, 2027. The routing approach still works; pass current model names as strong_model and weak_model, for example Claude Sonnet 5.5 as the strong model and Claude Haiku 5.5 or Gemini 3.5 Flash-Lite as the weak one. The Gemini and Ollama examples below use current model names.

Why use more than one LLM?

With Large Language Models (LLMs) we mostly have two choices:

  1. A powerful model like GPT-4o or Claude 3.5 Sonnet, which produces all the good stuff: high-quality, highly intelligent (well, relatively speaking) outputs. But these models cost quite a lot. They are also slower and produce fewer tokens per second, which can limit the scalability and usefulness of your application.

  2. A smaller model, like GPT-4o mini or the remarkable Google Gemini 1.5 Flash. These models are enormously efficient and cheap, but let's face it, their outputs are often less impressive on more complex tasks.

Since most of us need good output quality, we go for the more powerful models. That often eats into our margins, or the project never starts because the calculated costs are just too high.

There is a better way. No rule says you need to use one model for all your tasks. We can integrate multiple models into our processes and applications and pick the right one for the task at hand: the cheap and fast one for simple, preparatory tasks, the powerful one for the tasks that produce the most value or involve the most complexity. (Routing is one of several strategies to reduce LLM costs.)

There are multiple ways to make this work. A rather brute-force one is to ask a fairly strong LLM whether a task is complex and then choose the model based on its answer. (I've seen this implemented quite a few times.) The problem is that this is inherently slow, and it can get quite costly over time, or at least it adds costs that aren't necessary at all. Another approach is to train a simple, very efficient, locally runnable model that routes tasks to the right model.

This second approach has very low latency and low inference costs, because these specialized "routing models" can be very small. Creating them, however, takes quite extensive effort.

Thanks to the fine people at LMSYS, that work is already done. They created 4 generic routers that implement LLM routing based on the task at hand. The package is called RouteLLM.

What is RouteLLM and how does it work?

RouteLLM is a Python package and framework to serve and evaluate LLM routers. It includes 4 pre-trained, well-performing routers.

These routers decide, based on the complexity of the task, whether a prompt goes to a "strong" model (like GPT-4o) or a "weak" model (like GPT-4o mini).

The RouteLLM announcement also claims that the default routers have learned some characteristics of which models are strong as of 2024 and which are weak. That makes sense from a theoretical perspective too: routing can't rely on task complexity alone, it also has to account for the particularities of each LLM.

So RouteLLM optimizes cost against output quality. The chart below plots model performance against cost (note the logarithmic scale of the x-axis). Always routing to the strongest model, GPT-4o, gives the best possible output, but also the highest costs. A good router, however, gets us 90% of the output quality at 10% of the costs. (The image is from the RouteLLM announcement blog.)

Cost vs. quality of output when using RouteLLM

The principle of using RouteLLM is simple. The framework provides an OpenAI API compatible interface and sits in front of the models it routes to. In a simple setup, your application, which already integrates with OpenAI, connects to RouteLLM. RouteLLM in turn connects to the OpenAI API and uses GPT-4o or GPT-4o mini, depending on the prompt.

RouteLLM concept as part of an application

Potential cost savings when using RouteLLM

According to the benchmarks provided by LMSYS, RouteLLM reduced costs by over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K, compared with using only GPT-4, "while still achieving 95% of GPT-4's performance". The main evaluations routed between GPT-4 Turbo as the strong model and Mixtral 8x7B as the weak one.

That's quite remarkable. The large differences in cost savings come from the different nature of the benchmarks. MT Bench is more general and less complex than GSM8K, so you'd expect much higher cost savings on the less complex benchmark.

  • MT Bench is a general purpose benchmark made of high-quality real-world question/answer pairs.
  • MMLU (Massive Multitask Language Understanding) is a benchmark designed to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. It is a balanced benchmark, with "easier" general knowledge questions and more complex ones that involve few-shot techniques.
  • GSM8K is a dataset of 8.5K high-quality, linguistically diverse grade school math word problems. Math is quite hard for LLMs, so you'd expect the router to pick the strong model more often than not.

All in all, the cost savings potential seems to be between 30 and 80%, depending on your use case!

Fewer calls to the large model also mean less energy per request. Our post on the energy footprint of generative AI puts numbers on that side.

How to set up RouteLLM to reduce LLM costs

Enough talk, let's put this into practice.

First, install RouteLLM. We also install the serve dependencies, more on that later.

1pip install "routellm[serve] pandarallel"

Then import the required modules and set the API key. This first example only uses OpenAI models, so we only need an OpenAI API key.

1import os
2from routellm.controller import Controller
3
4os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"

Finally, create a client instance to use RouteLLM:

1client = Controller(
2 routers=["mf"], # What router type to use
3 strong_model="gpt-4o",
4 weak_model="gpt-4o-mini",
5)
6response = client.chat.completions.create(
7 # the 0.116 is the cost/quality threshold for the router. more on that
8 # in the section below.
9 model="router-mf-0.116",
10 messages=[
11 {"role": "user", "content": "How to maintain my machine?"}
12 ]
13)

That's all you need to use RouteLLM in your application. RouteLLM takes the user prompt, uses the mf router to determine the complexity of the task, and routes it to the right model.

The routers parameter ("What router type to use") accepts 4 router types:

  • mf uses a matrix factorization model trained on the preference data. This router is the best choice for most use cases.
  • sw_ranking uses a weighted Elo calculation for routing, where each vote is weighted according to how similar it is to the user's prompt.
  • bert uses a BERT classifier trained on the preference data.
  • causal_llm uses an LLM-based classifier tuned on the preference data.

Long story short: from our experience (and what the authors of RouteLLM suggest), just use the mf router.

Where the number 0.116 comes from is explained in the next section.

How to calibrate the RouteLLM threshold

Each router can be tuned for how many prompts from a given dataset it sends to the strong model and how many to the weak one.

The authors suggest calibrating RouteLLM with the freely available Chatbot Arena dataset. It contains over 55,000 real-world conversations between users and LLMs, with user preferences, across over 70 state-of-the-art LLMs such as GPT-4, Claude 2, Llama 2, Gemini, and Mistral models. Each sample is a battle in which 2 LLMs answer the same question, and the user labels the result as prefer model A, prefer model B, tie, or tie (both bad).

To run the calibration, execute:

1OPENAI_API_KEY=sk-XXXXX python -m routellm.calibrate_threshold --task calibrate --routers mf --strong-model-pct 0.5

The output looks something like this:

1For 50.0% strong model calls for mf, threshold = 0.11593

This value (0.11593) is the router threshold. Set it in the client chat completion request as shown in the example above (like model="router-mf-0.11593").

Note: Setting the percentage to 50% during calibration does not mean that 50% of your actual queries are routed to the strong model. It means that 50% of the prompts in the calibration dataset would be sent to the strong one. If your average prompts are more or less complex than the calibration prompts, your actual percentage will differ.

The rule of thumb: start by calibrating for 50%. If you want to favour the strong model, set it higher; if you want to favour the weak model, set it lower. And test your results!

Routing between providers like OpenAI and Google Gemini

Let's say we want to route between GPT-4o and a Gemini Flash model. When we first wrote this in August 2024, that pairing made a lot of sense: GPT-4o was the stronger model, and Gemini 1.5 Flash was one of the best "weak" models, with a long context window, high speed and low costs. Gemini 1.5 Flash has been shut down since, so the example uses a current small model, Gemini 3.5 Flash-Lite. Compare current models on your own prompts before you pick a pair.

All we need to do is set the models in the client instantiation and provide API keys for both providers:

1os.environ["GEMINI_API_KEY"] = ""
2os.environ["OPENAI_API_KEY"] = ""
3
4client = Controller(
5 routers=["mf"],
6 strong_model="gpt-4o",
7 weak_model="gemini/gemini-3.5-flash-lite",
8)

The rest stays the same.

1response = client.chat.completions.create(
2 # the 0.116 is the cost/quality threshold for the router. more on that
3 # in the section below.
4 model="router-mf-0.116",
5 messages=[
6 {"role": "user", "content": "How to maintain my machine?"}
7 ]
8)

RouteLLM uses LiteLLM under the hood. LiteLLM is a package that provides OpenAI compatible SDK/API endpoints for almost any modern LLM. To use other models with RouteLLM, look up the API key name and model name in the supported models in LiteLLM section of their docs.

You can even use local models with Ollama. Run Ollama, then set up the Controller as follows:

1client = Controller(
2 routers=["mf"],
3 strong_model="gpt-4o",
4 weak_model="ollama/llama3.2",
5 api_base="http://localhost:11434" # set to your ollama server instance
6)

If you'd rather serve the weak model on your own server, our guide to self-hosting an LLM with vLLM and Caddy shows how. SGLang and llama.cpp also serve OpenAI-compatible APIs; see our guides to running DeepSeek R1 with SGLang and running GLM-4.7-Flash locally with llama.cpp.

Using the RouteLLM server as a drop-in replacement for the OpenAI API

In the examples above, we used the RouteLLM Python SDK inside a Python application. RouteLLM can also run as a standalone OpenAI API compatible server, which is probably the way to go for production applications.

Run the following command to start the server with the mf router:

1python -m routellm.openai_server --routers mf --strong-model gpt-4o --weak-model gpt-4o-mini

To use the server, replace the model name and API URL in your default OpenAI SDK client:

1from openai import OpenAI # Use the standard OpenAI SDK
2
3client = OpenAI(
4 base_url="http://localhost:6060/v1", # set to your routellm server instance
5 api_key="no_api_key"
6)
7
8response = client.chat.completions.create(
9 model="router-mf-0.11593", # Set this to the threshold you calibrated
10 messages=[
11 {"role": "user", "content": "How to maintain my machine?"}
12 ]
13)

Changing two lines of code in your existing application is a convenient way to potentially save hundreds of dollars a month.

Conclusion

RouteLLM routes each prompt to a strong or a weak model, which can save up to 80% of LLM costs. To use it, install the package, pick the mf router, calibrate the threshold on the Chatbot Arena dataset (start with 50% strong model calls), and set the calibrated threshold in the model name. Through LiteLLM, the strong and weak models can come from different providers like OpenAI and Google Gemini, or run locally with Ollama. For production, run the RouteLLM server and point your existing OpenAI SDK client at it.

We think RouteLLM is a great addition to any AI toolkit: with very little effort, it delivers almost free cost savings.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.