DeepSeek R1 On-Prem Setup: Run Advanced AI Models on Your Hardware with SGLang

On this page
You can run DeepSeek R1 on your own hardware by using its distilled version, DeepSeek-R1-Distill-Qwen-32B, and serving it with the SGLang framework. Two 24GB GPUs such as the NVIDIA RTX 3090 are enough, and SGLang gives you an OpenAI-compatible API, so existing client code keeps working. This guide covers the installation, the model download, a Docker setup and the first API calls.
Running a model this capable locally used to need enterprise data center hardware. Two things changed that: high-quality distilled models (and good smaller LLMs in general), and better model serving software like SGLang.
DeepSeek R1 is one of the most capable language models currently available. We use DeepSeek R1 Distill, a compressed version that keeps strong performance with much lower hardware requirements, together with SGLang, in our view the most advanced model serving framework available.
The result competes with GPT-4o on mathematical reasoning, coding and general knowledge tasks, on your own machine. For context, DeepSeek-R1-Distill-Qwen-32B outperforms GPT-4o on several benchmarks, including a Codeforces rating of 1691 (GPT-4o: 759), and the 14B version also performs very well for its size.
In this guide, we'll walk through:
- What distilled models are and why we like them so much
- Installing SGLang and setting it up for model serving
- Running DeepSeek R1 Distill on your hardware
The guide focuses on DeepSeek R1, but the same steps work for other models too.
Understanding distilled models
Model distillation is what makes this deployment possible, so it's worth understanding before the technical setup.
What are distilled models?
Model distillation is a technique where a larger, more complex model (the "teacher") transfers its knowledge to a smaller, more efficient model (the "student"). Think of it as a concentrated version of the original model's capabilities. The DeepSeek team used this process to create several smaller versions of R1, ranging from 1.5B to 70B parameters.
To see how distilled models (or any LLM) fit into a wider AI system, read our post on integrating knowledge and LLMs.
The DeepSeek R1 Distill family
DeepSeek offers several distilled versions of R1:
- DeepSeek-R1-Distill-Qwen-1.5B
- DeepSeek-R1-Distill-Qwen-7B
- DeepSeek-R1-Distill-Qwen-14B
- DeepSeek-R1-Distill-Qwen-32B
- DeepSeek-R1-Distill-Llama-70B
This guide uses the Qwen-32B model. It is the sweet spot between the original R1's capabilities and practical hardware requirements. According to DeepSeek's model card, DeepSeek-R1-Distill-Qwen-32B:
- Achieves 72.6% accuracy on AIME 2024 (GPT-4o: 9.3%)
- Scores a Codeforces rating of 1691 (GPT-4o: 759)
- Reaches 94.3% accuracy on the MATH-500 benchmark
- Outperforms OpenAI's o1-mini across various benchmarks
Why distilled models matter
Before distilled models like these, running state-of-the-art LLMs meant one of these:
- Enterprise-grade hardware with massive VRAM (often 80GB+ per GPU)
- Complex setups with model parallelism across multiple GPUs
- Significant compromises in model quality
With the 32B distilled model, you get close to original R1 performance on consumer-grade hardware (or at least hardware with consumer-grade pricing). Developers, researchers and hobbyists can now run an advanced reasoning model on standard hardware.
The next section covers SGLang, the serving framework that gets the most out of these models and makes the 32B version run efficiently on consumer hardware.
SGLang: model serving for DeepSeek R1
SGLang is one of the most efficient serving frameworks for large language models, which is why we chose it for running DeepSeek R1 Distill locally. Alternatives like vLLM exist (we use vLLM in our guide to self-hosting an LLM with HTTPS), but SGLang's optimizations make it a good fit for running the 32B R1 Distill on consumer hardware.
SGLang features for local deployment
These SGLang features matter for our setup:
- Tensor parallelism support for efficient multi-GPU utilization
- RadixAttention for optimized prefix caching
- Continuous batching for improved throughput
- FlashInfer kernels for faster inference
- BF16 and FP8 support for a smaller memory footprint
- Torch compile integration
Hardware requirements
For running the 32B model, you'll need:
- 2x NVIDIA GPUs with 24GB VRAM each (NVIDIA RTX 3090 or better)
- 32GB system RAM
- NVMe SSD with at least 100GB free space
(For more tips on keeping your setup budget-friendly, see How to Save on LLM Costs.)
Basic installation
The install command below uses the FlashInfer wheels built for CUDA 12.4 and torch 2.5, so you need a CUDA 12.4 (or newer) toolkit and driver. If you haven't installed it, follow NVIDIA's instructions: NVIDIA CUDA Installation
Model setup
First, download the model:
Then launch the server:
Running SGLang with Docker
If you prefer Docker, use the following command:
The next section covers the client setup and basic inference patterns.
Running your first inferences with DeepSeek R1 and SGLang
Basic inference using the OpenAI client
SGLang is compatible with the OpenAI API, so you can use the OpenAI Python client to talk to your local server. That makes moving from a cloud API to local deployment very easy.
Structured output
For software-related problems or any task that needs structured output, use SGLang's JSON schema support:
Direct API requests
You can also use Python's requests library for more direct control:
Streaming responses
Streaming is often useful for GUI applications: users see the first
tokens earlier, so the LLM answers feel snappier. Use stream=True:
Summary
DeepSeek-R1-Distill-Qwen-32B brings most of R1's reasoning performance to hardware you can buy: two 24GB GPUs, 32GB of RAM and 100GB of disk. SGLang serves it with one command (or one Docker container) and exposes an OpenAI-compatible API on port 30000. From there, the OpenAI Python client, JSON schema structured output and streaming all work the same way they do against a cloud API. If 32B is too much for your hardware, the 14B distill is the next step down. For coding work on a single 16GB GPU, see our guide to running GLM-4.7-Flash locally.
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



