On-prem models / DeepSeek
Run DeepSeek-V3.2 on-prem
December 2025 DeepSeek model introducing DeepSeek Sparse Attention and 'thinking with tools'. Still heavily downloaded and the best-supported large DeepSeek checkpoint across engines.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 1342 GB | 0.58 GB | 2.30 GB | 9.21 GB |
| FP8 | 671 GB | 0.29 GB | 1.15 GB | 4.61 GB |
| INT4 (AWQ / GPTQ) | 376 GB | 0.29 GB | 1.15 GB | 4.61 GB |
| GGUF Q4_K_M | 403 GB | 0.29 GB | 1.15 GB | 4.61 GB |
KV cache from the published config: 61 layers, MLA latent 512 + 64 RoPE dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A dependable large model for SQL generation and agents if you already run it, with a clean licence. New on-prem deployments would usually pick a smaller Qwen/GLM model or V4.1-Flash.
Strengths
- Very mature support in vLLM, SGLang and TensorRT-LLM
- Compact MLA KV cache
- Strong reasoning and tool use
- MIT licence
Watch out for
- About 690 GB in FP8; needs 8x H200/B200-class GPUs
- No Jinja chat template; uses DeepSeek's Python encoder
- Text only
- Superseded by the V4 family on quality and efficiency
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 8 | 389 / 460 GB | 103 |
| AMD Instinct MI325X 256GB | 2 | 380 / 460 GB | 143 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 383 / 460 GB | 131 |
| Apple Mac Studio M3 Ultra 512GB | 1 | 378 / 460 GB | 17 |
| Apple Mac Studio M5 Ultra 512GB | 1 | 378 / 460 GB | 25 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 2 | 380 / 486 GB | 168 |
| NVIDIA H200 SXM 141GB | 4 | 383 / 507 GB | 147 |
| NVIDIA H200 NVL 141GB | 4 | 383 / 507 GB | 147 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI300X 192GB | 4 | 689 / 691 GB | 108 |
| AMD Instinct MI325X 256GB | 4 | 689 / 921 GB | 116 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 8 | 695 / 921 GB | 109 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 4 | 689 / 972 GB | 134 |
| NVIDIA H200 SXM 141GB | 8 | 695 / 1015 GB | 121 |
| NVIDIA H200 NVL 141GB | 8 | 695 / 1015 GB | 121 |
| AMD Instinct MI355X 288GB | 4 | 689 / 1036 GB | 134 |
| NVIDIA B200 180GB | 8 | 695 / 1296 GB | 142 |
Serving and licence
vLLM and SGLang support DeepSeek-V3.2 (DSA kernels needed; use recent versions); TensorRT-LLM supports the V3 family. KTransformers can offload experts to CPU. Recommended sampling temperature 1.0, top_p 0.95.
MIT. MIT licence for code and weights: commercial use allowed without revenue or user caps.
Design a DeepSeek-V3.2 deployment
Preloaded with DeepSeek-V3.2. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does DeepSeek-V3.2 need?
The weights alone take about 1342 GB in BF16, 671 GB in FP8 and 376 GB in 4-bit. On top of that each request holds a KV cache of about 0.29 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 5.76 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs DeepSeek-V3.2?
For a single user at 8K context in 4-bit, 8 x AMD Instinct MI210 64GB (64 GB each) fits, at an estimated 103 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is DeepSeek-V3.2 allowed for commercial use?
DeepSeek-V3.2 is released under MIT. MIT licence for code and weights: commercial use allowed without revenue or user caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve DeepSeek-V3.2 on-prem?
vLLM and SGLang support DeepSeek-V3.2 (DSA kernels needed; use recent versions); TensorRT-LLM supports the V3 family. KTransformers can offload experts to CPU. Recommended sampling temperature 1.0, top_p 0.95.