LLM Sampling with FastMCP: Using Client LLMs for Scalable AI Workflows in your MCP

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 4 min read
LLM Sampling with FastMCP: Using Client LLMs for Scalable AI Workflows in your MCP
On this page

LLM sampling lets an MCP server delegate its language model calls to the client. A server-side tool sends a prompt to the client, the client runs it through its own LLM, and the result comes back to the tool. With FastMCP, the server side is a single ctx.sample call and the client side is a sampling handler function. This post shows both, building on our earlier posts about creating your first MCP server and automating MCP server creation.

Sampling addresses one of the biggest challenges in MCP server architecture: scaling language model inference. AI-powered MCP servers often need to call an LLM inside their server-side functions. If the client handles those requests instead, the server stays lightweight and scalable, even as the number of concurrent users grows. There is no server-side bottleneck for LLM inference, and your server functions still get access to the model.

Sampling also changes the economics of your MCP server. You implement advanced AI workflows on the server, but the client is responsible for the actual LLM calls and therefore for their costs. (Another way to cut LLM spend is LLM routing.)

Update (November 2026): The MCP specification of 2026-07-28 deprecated sampling and suggests calling LLM provider APIs directly instead. It also reclassified the HTTP+SSE transport used below as deprecated, in favour of Streamable HTTP. FastMCP 4 removed ctx.sample(), so pin fastmcp<3 to run this code as written. FastMCP 3.x also still supports ctx.sample().

What is LLM sampling in MCP?

LLM sampling in MCP servers is the ability of server-side functions to request language model completions from the client's language model instead of a server-side LLM. It inverts the usual architecture: the server sends prompt instructions to the client, which performs the LLM inference and returns the result to the server. The MCP specification defines this as a standard way for servers to request completions via clients.

In traditional AI architectures, the server hosts both the application logic and the language model integration. With MCP sampling, your server-side code can make requests like "generate a summary of this text" or "analyze the sentiment of this comment", but the language model processing happens on the client side.

The flow works as follows:

  1. Your MCP server function needs LLM capabilities (e.g., to classify text or generate content)
  2. The function sends a prompt to the client
  3. The client receives this request and processes it using its own LLM (which could be a local model or another API)
  4. The client returns the generated text to your server function
  5. Your server function continues processing with the LLM-generated response, almost as if the server had performed the LLM inference itself

This architecture has several advantages for MCP applications:

  • Your server avoids the compute-intensive LLM inference, so it can handle more concurrent users.
  • Processing costs are spread across clients instead of being centralized on your server.
  • Clients can use different models based on their capabilities or preferences.
  • For some applications, response times improve because there is no server-side queue for LLM inference.

In short, sampling connects your server's processing logic with the language model the client has access to, and spreads the AI workload across the system.

How to use LLM sampling in a FastMCP server

To implement LLM sampling in your FastMCP server, use FastMCP's Context object. Here is a simple example.

First, create the FastMCP server object.

1from fastmcp import FastMCP, Context
2
3mcp = FastMCP(
4 name="Document Assistant",
5 instructions="This server allows to fetch, summarize and update documents related to PONDHOUSE DATA DOCS."
6)

Next, define a tool that summarizes a document. This is a simplified example; document summarization normally involves multiple steps.

Note: LLM sampling is available in the Context object. Add a parameter with the type hint Context to your tool function, and FastMCP injects the context object for you.

To use LLM sampling instead of a server-side LLM, call ctx.sample. Pass a message (a string or a list of strings), a system prompt and the model parameters, like temperature and max tokens.

1@mcp.tool()
2async def summarize_document(document_text: str, ctx: Context) -> str:
3 """Generate a summary of a document"""
4 response = await ctx.sample(
5 messages=f"Summarize the following document:\n{document_text}",
6 system_prompt="You are a document assistant. First extract the key ideas of the document. Then summarize it in a few sentences.",
7 temperature=0.7,
8 max_tokens=300,
9 )
10
11 summary = response.text
12 return f"Summary:\n{summary}"

Note: You can also pass a list of SamplingMessage objects to the messages parameter. This is helpful if you want to give the client's LLM the context of a previous conversation between a user and an assistant.

1# Use this if you want to provide the client LLM with context of a previous
2# conversation between a user and an assistant. Stick to the easier string
3# version if you don't need this.
4from mcp.types import SamplingMessage, TextContent
5sampling_messages = [
6 SamplingMessage(
7 role="user", content=TextContent(type="text", text="How are you?")
8 ),
9 SamplingMessage(
10 role="assistant", content=TextContent(type="text", text="I am fine.")
11 ),
12]

That's all you need on the server. Run your FastMCP server as described in our previous blog post and you're done. Keep in mind that the client also needs to support LLM sampling.

Implement LLM sampling on the client with the FastMCP client

To show how LLM sampling works on the client side, we use the FastMCP MCP client library. To support LLM sampling, you implement a sampling handler: a function that takes the incoming sampling request from the server and returns the generated text.

1from fastmcp.client import Client
2from fastmcp.client.sampling import (
3 SamplingMessage,
4 SamplingParams,
5 RequestContext,
6)
7# Note the `sse` path post-fix for sse servers
8# For all transports, see https://gofastmcp.com/clients/client#transports
9sse_url = "http://localhost:8000/sse"
10
11async def sampling_handler(
12 messages: list[SamplingMessage],
13 params: SamplingParams,
14 context: RequestContext
15) -> str:
16 messages = [{"role": m.role, "content": m.content.text} for m in messages]
17 system_prompt = params.systemPrompt
18 temperature = params.temperature
19 max_tokens = params.maxTokens
20
21 # Here you can implement your own LLM calling logic
22 # For example, using OpenAI's API or any other LLM provider
23 # For this example, we will just return a dummy response
24 return f"Generated text for prompt: {messages[-1]['content']}"
25
26client = Client(sse_url, sampling_handler=sampling_handler)

Summary

LLM sampling is a simple concept and easy to implement, because the low-level MCP protocol handles the communication between server and client. You need two pieces:

  • On the server side, a tool that calls ctx.sample to request LLM inference from the client
  • On the client side, a sampling handler that processes the incoming sampling request and returns the generated text

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

Further reading

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.