On-prem models / Alibaba Qwen
Run Qwen3-30B-A3B-Instruct-2507 on-prem
July 2025 refresh of the Qwen3 30B MoE in non-thinking form, 3.3B active. A very common local model because it is fast on modest GPUs and CPUs.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 61.0 GB | 0.81 GB | 3.22 GB | 12.9 GB |
| FP8 | 30.5 GB | 0.40 GB | 1.61 GB | 6.44 GB |
| INT4 (AWQ / GPTQ) | 17.1 GB | 0.40 GB | 1.61 GB | 6.44 GB |
| GGUF Q4_K_M | 18.3 GB | 0.40 GB | 1.61 GB | 6.44 GB |
KV cache from the published config: 48 layers, 4 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Good for cheap, fast assistants over ERP data: summarising orders, drafting supplier emails, simple tool calls. For new builds, Qwen3.6-35B-A3B gives vision and better agents at similar cost.
Strengths
- Fast decode on consumer and workstation hardware
- Broad engine support including Ollama and llama.cpp
- 256K native context
- Apache 2.0
Watch out for
- Non-thinking only (a separate Thinking-2507 repo exists)
- Full attention on all 48 layers, so KV grows faster than in Qwen3.5+ hybrids
- Largely superseded by Qwen3.6-35B-A3B
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 33 / 36 GB | 153 |
| NVIDIA L4 24GB | 2 | 34 / 43 GB | 59 |
| NVIDIA A10 24GB | 2 | 34 / 43 GB | 103 |
| NVIDIA L40S 48GB | 1 | 33 / 43 GB | 95 |
| NVIDIA RTX A6000 48GB | 1 | 33 / 43 GB | 86 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 33 / 43 GB | 104 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 34 / 43 GB | 148 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 34 / 57 GB | 205 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 50 / 57 GB | 96 |
| AMD Instinct MI210 64GB | 1 | 49 / 57 GB | 57 |
| NVIDIA A100 40GB PCIe | 2 | 50 / 72 GB | 86 |
| NVIDIA A100 80GB PCIe | 1 | 49 / 72 GB | 67 |
| NVIDIA A100 80GB SXM | 1 | 49 / 72 GB | 71 |
| NVIDIA H100 PCIe 80GB | 1 | 49 / 72 GB | 70 |
| NVIDIA H100 SXM 80GB | 1 | 49 / 72 GB | 109 |
| NVIDIA H100 NVL 94GB | 1 | 49 / 84 GB | 123 |
Serving and licence
vllm>=0.8.5 or sglang>=0.4.6.post1; Ollama, LM Studio, MLX-LM, llama.cpp and KTransformers supported. 1M mode needs vLLM DUAL_CHUNK_FLASH_ATTN backend and about 240 GB of GPU memory per the card.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3-30B-A3B-Instruct-2507 deployment
Preloaded with Qwen3-30B-A3B-Instruct-2507. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3-30B-A3B-Instruct-2507 need?
The weights alone take about 61.0 GB in BF16, 30.5 GB in FP8 and 17.1 GB in 4-bit. On top of that each request holds a KV cache of about 0.40 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 8.05 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3-30B-A3B-Instruct-2507?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 153 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3-30B-A3B-Instruct-2507 allowed for commercial use?
Qwen3-30B-A3B-Instruct-2507 is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3-30B-A3B-Instruct-2507 on-prem?
vllm>=0.8.5 or sglang>=0.4.6.post1; Ollama, LM Studio, MLX-LM, llama.cpp and KTransformers supported. 1M mode needs vLLM DUAL_CHUNK_FLASH_ATTN backend and about 240 GB of GPU memory per the card.