DeepSeek R1 On-Prem Setup: Run Advanced AI Models on Your Hardware with SGLang

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 5 min read
DeepSeek R1 On-Prem Setup: Run Advanced AI Models on Your Hardware with SGLang
On this page

You can run DeepSeek R1 on your own hardware by using its distilled version, DeepSeek-R1-Distill-Qwen-32B, and serving it with the SGLang framework. Two 24GB GPUs such as the NVIDIA RTX 3090 are enough, and SGLang gives you an OpenAI-compatible API, so existing client code keeps working. This guide covers the installation, the model download, a Docker setup and the first API calls.

Running a model this capable locally used to need enterprise data center hardware. Two things changed that: high-quality distilled models (and good smaller LLMs in general), and better model serving software like SGLang.

DeepSeek R1 is one of the most capable language models currently available. We use DeepSeek R1 Distill, a compressed version that keeps strong performance with much lower hardware requirements, together with SGLang, in our view the most advanced model serving framework available.

The result competes with GPT-4o on mathematical reasoning, coding and general knowledge tasks, on your own machine. For context, DeepSeek-R1-Distill-Qwen-32B outperforms GPT-4o on several benchmarks, including a Codeforces rating of 1691 (GPT-4o: 759), and the 14B version also performs very well for its size.

In this guide, we'll walk through:

  • What distilled models are and why we like them so much
  • Installing SGLang and setting it up for model serving
  • Running DeepSeek R1 Distill on your hardware

The guide focuses on DeepSeek R1, but the same steps work for other models too.

Understanding distilled models

Model distillation is what makes this deployment possible, so it's worth understanding before the technical setup.

What are distilled models?

Model distillation is a technique where a larger, more complex model (the "teacher") transfers its knowledge to a smaller, more efficient model (the "student"). Think of it as a concentrated version of the original model's capabilities. The DeepSeek team used this process to create several smaller versions of R1, ranging from 1.5B to 70B parameters.

To see how distilled models (or any LLM) fit into a wider AI system, read our post on integrating knowledge and LLMs.

The DeepSeek R1 Distill family

DeepSeek offers several distilled versions of R1:

  • DeepSeek-R1-Distill-Qwen-1.5B
  • DeepSeek-R1-Distill-Qwen-7B
  • DeepSeek-R1-Distill-Qwen-14B
  • DeepSeek-R1-Distill-Qwen-32B
  • DeepSeek-R1-Distill-Llama-70B

This guide uses the Qwen-32B model. It is the sweet spot between the original R1's capabilities and practical hardware requirements. According to DeepSeek's model card, DeepSeek-R1-Distill-Qwen-32B:

  • Achieves 72.6% accuracy on AIME 2024 (GPT-4o: 9.3%)
  • Scores a Codeforces rating of 1691 (GPT-4o: 759)
  • Reaches 94.3% accuracy on the MATH-500 benchmark
  • Outperforms OpenAI's o1-mini across various benchmarks

Why distilled models matter

Before distilled models like these, running state-of-the-art LLMs meant one of these:

  1. Enterprise-grade hardware with massive VRAM (often 80GB+ per GPU)
  2. Complex setups with model parallelism across multiple GPUs
  3. Significant compromises in model quality

With the 32B distilled model, you get close to original R1 performance on consumer-grade hardware (or at least hardware with consumer-grade pricing). Developers, researchers and hobbyists can now run an advanced reasoning model on standard hardware.

The next section covers SGLang, the serving framework that gets the most out of these models and makes the 32B version run efficiently on consumer hardware.

SGLang: model serving for DeepSeek R1

SGLang is one of the most efficient serving frameworks for large language models, which is why we chose it for running DeepSeek R1 Distill locally. Alternatives like vLLM exist (we use vLLM in our guide to self-hosting an LLM with HTTPS), but SGLang's optimizations make it a good fit for running the 32B R1 Distill on consumer hardware.

SGLang features for local deployment

These SGLang features matter for our setup:

  • Tensor parallelism support for efficient multi-GPU utilization
  • RadixAttention for optimized prefix caching
  • Continuous batching for improved throughput
  • FlashInfer kernels for faster inference
  • BF16 and FP8 support for a smaller memory footprint
  • Torch compile integration

Hardware requirements

For running the 32B model, you'll need:

  • 2x NVIDIA GPUs with 24GB VRAM each (NVIDIA RTX 3090 or better)
  • 32GB system RAM
  • NVMe SSD with at least 100GB free space

(For more tips on keeping your setup budget-friendly, see How to Save on LLM Costs.)

Basic installation

The install command below uses the FlashInfer wheels built for CUDA 12.4 and torch 2.5, so you need a CUDA 12.4 (or newer) toolkit and driver. If you haven't installed it, follow NVIDIA's instructions: NVIDIA CUDA Installation

1pip install sgl-kernel --force-reinstall --no-deps
2pip install "sglang[all]>=0.4.3.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

Model setup

First, download the model:

1from huggingface_hub import snapshot_download
2
3snapshot_download(
4 repo_id="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
5 local_dir="./models/deepseek-r1-32b"
6)

Then launch the server:

1python3 -m sglang.launch_server \
2 --model-path ./models/deepseek-r1-32b \
3 --trust-remote-code

Running SGLang with Docker

If you prefer Docker, use the following command:

1docker run --gpus all \
2 --shm-size 32g \
3 -p 30000:30000 \
4 -v ~/.cache/huggingface:/root/.cache/huggingface \
5 --env "HF_TOKEN=<secret>" \
6 --ipc=host \
7 lmsysorg/sglang:latest \
8 python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --host 0.0.0.0 --port 30000

The next section covers the client setup and basic inference patterns.

Running your first inferences with DeepSeek R1 and SGLang

Basic inference using the OpenAI client

SGLang is compatible with the OpenAI API, so you can use the OpenAI Python client to talk to your local server. That makes moving from a cloud API to local deployment very easy.

1import openai
2
3client = openai.Client(
4 base_url="http://localhost:30000/v1", # Default SGLang port
5 api_key="none"
6)
7
8response = client.chat.completions.create(
9 model="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
10 messages=[
11 {"role": "user", "content": "Explain how neural networks work"}
12 ],
13 temperature=0.7,
14 max_tokens=512
15)
16
17print(response.choices[0].message.content)

Structured output

For software-related problems or any task that needs structured output, use SGLang's JSON schema support:

1response = client.chat.completions.create(
2 model="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
3 messages=[
4 {"role": "user", "content": "Solve this calculus problem: Find the derivative of x³ + 2x² - 5x + 3"}
5 ],
6 temperature=0.1,
7 max_tokens=512,
8 response_format={
9 "type": "json_schema",
10 "json_schema": {
11 "name": "calculus_solution",
12 "schema": {
13 "type": "object",
14 "properties": {
15 "steps": {"type": "array", "items": {"type": "string"}},
16 "final_answer": {"type": "string"}
17 },
18 "required": ["steps", "final_answer"]
19 }
20 }
21 }
22)

Direct API requests

You can also use Python's requests library for more direct control:

1import requests
2
3response = requests.post(
4 "http://localhost:30000/v1/chat/completions",
5 json={
6 "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
7 "messages": [
8 {"role": "user", "content": "Explain quantum computing"}
9 ],
10 "temperature": 0.7,
11 "max_tokens": 512
12 }
13)
14
15print(response.json()['choices'][0]['message']['content'])

Streaming responses

Streaming is often useful for GUI applications: users see the first tokens earlier, so the LLM answers feel snappier. Use stream=True:

1import openai
2
3client = openai.Client(base_url=f"http://127.0.0.1:30000/v1", api_key="None")
4
5# Use stream=True for streaming responses
6response = client.chat.completions.create(
7 model="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
8 messages=[
9 {"role": "user", "content": "List 3 countries and their capitals."},
10 ],
11 temperature=0,
12 max_tokens=640,
13 stream=True,
14)
15
16# Handle the streaming output
17for chunk in response:
18 if chunk.choices[0].delta.content:
19 print(chunk.choices[0].delta.content, end="", flush=True)

Summary

DeepSeek-R1-Distill-Qwen-32B brings most of R1's reasoning performance to hardware you can buy: two 24GB GPUs, 32GB of RAM and 100GB of disk. SGLang serves it with one command (or one Docker container) and exposes an OpenAI-compatible API on port 30000. From there, the OpenAI Python client, JSON schema structured output and streaming all work the same way they do against a cloud API. If 32B is too much for your hardware, the 14B distill is the next step down. For coding work on a single 16GB GPU, see our guide to running GLM-4.7-Flash locally.

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

Further reading

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.