On-prem models / Alibaba Qwen
Run Qwen3-32B on-prem
Dense 32.8B text model from the original Qwen3 release with switchable thinking and non-thinking modes. Widely deployed and supported by every major engine.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 65.6 GB | 2.15 GB | 8.59 GB | 34.4 GB |
| FP8 | 32.8 GB | 1.07 GB | 4.29 GB | 17.2 GB |
| INT4 (AWQ / GPTQ) | 18.4 GB | 1.07 GB | 4.29 GB | 17.2 GB |
| GGUF Q4_K_M | 19.7 GB | 1.07 GB | 4.29 GB | 17.2 GB |
KV cache from the published config: 64 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Still a safe, well-understood option for text-to-SQL and tool-calling agents where a team already has it validated. New deployments should usually start from Qwen3.8-27B instead.
Strengths
- Very mature tooling and quantisations everywhere
- Hybrid thinking with /think and /no_think soft switches
- 100+ languages per the card
- Apache 2.0
Watch out for
- Full-attention GQA on 64 layers makes long-context KV expensive
- Only 32K native context; YaRN needed beyond that
- Superseded by Qwen3.6/3.8-27B for most tasks
- Text only
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 36 / 36 GB | 27 |
| NVIDIA L4 24GB | 2 | 37 / 43 GB | 9 |
| NVIDIA A10 24GB | 2 | 37 / 43 GB | 18 |
| NVIDIA L40S 48GB | 1 | 36 / 43 GB | 15 |
| NVIDIA RTX A6000 48GB | 1 | 36 / 43 GB | 14 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 36 / 43 GB | 17 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 37 / 43 GB | 30 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 37 / 57 GB | 50 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H100 NVL 94GB | 1 | 78 / 84 GB | 40 |
| NVIDIA L4 24GB | 4 | 82 / 86 GB | 11 |
| NVIDIA A10 24GB | 4 | 82 / 86 GB | 20 |
| NVIDIA L40S 48GB | 2 | 79 / 86 GB | 16 |
| NVIDIA RTX A6000 48GB | 2 | 79 / 86 GB | 14 |
| NVIDIA RTX 6000 Ada 48GB | 2 | 79 / 86 GB | 18 |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96GB | 1 | 78 / 86 GB | 19 |
| NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB | 1 | 78 / 86 GB | 19 |
Serving and licence
sglang>=0.4.6.post1 or vllm>=0.8.5 with the qwen3 reasoning parser; llama.cpp, Ollama, LM Studio, MLX and KTransformers all support it. Enable YaRN only when you need more than 32K.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3-32B deployment
Preloaded with Qwen3-32B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3-32B need?
The weights alone take about 65.6 GB in BF16, 32.8 GB in FP8 and 18.4 GB in 4-bit. On top of that each request holds a KV cache of about 1.07 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 21.5 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3-32B?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 27 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3-32B allowed for commercial use?
Qwen3-32B is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3-32B on-prem?
sglang>=0.4.6.post1 or vllm>=0.8.5 with the qwen3 reasoning parser; llama.cpp, Ollama, LM Studio, MLX and KTransformers all support it. Enable YaRN only when you need more than 32K.