On-prem models / Alibaba Qwen
Run Qwen3.6-35B-A3B on-prem
Small hybrid MoE (35B total, 3B active) vision-language model focused on agentic coding and stability. Successor in practice to the Qwen3-30B-A3B class.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 72.0 GB | 0.17 GB | 0.67 GB | 2.68 GB |
| FP8 | 36.0 GB | 0.08 GB | 0.34 GB | 1.34 GB |
| INT4 (AWQ / GPTQ) | 20.2 GB | 0.08 GB | 0.34 GB | 1.34 GB |
| GGUF Q4_K_M | 21.6 GB | 0.08 GB | 0.34 GB | 1.34 GB |
KV cache from the published config: 40 layers, 2 KV heads x 256 dims, 10 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Good choice for high-volume, latency-sensitive jobs: classifying and extracting fields from invoices and delivery notes, routing tickets, and simple ERP lookups via tool calls. For hard multi-join SQL a dense 27B or larger MoE is safer.
Strengths
- Very fast decode for its quality (3B active)
- Fits one 48 GB GPU at 4-bit or one 80 GB GPU in FP8
- Tiny KV cache: 10 attention layers with 2 KV heads
- Thinking preservation option for multi-turn agents
- Apache 2.0
Watch out for
- MoE still needs all 35B weights in memory
- Weaker knowledge depth than 27B dense or larger MoE models
- Thinking on by default; disable it for low-latency extraction
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 2 | 40 / 43 GB | 67 |
| NVIDIA A10 24GB | 2 | 40 / 43 GB | 117 |
| NVIDIA L40S 48GB | 1 | 38 / 43 GB | 107 |
| NVIDIA RTX A6000 48GB | 1 | 38 / 43 GB | 97 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 38 / 43 GB | 117 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 40 / 43 GB | 166 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 40 / 57 GB | 227 |
| AMD Instinct MI210 64GB | 1 | 38 / 57 GB | 175 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 2 | 43 / 43 GB | 46 |
| NVIDIA A10 24GB | 2 | 43 / 43 GB | 83 |
| NVIDIA L40S 48GB | 1 | 41 / 43 GB | 73 |
| NVIDIA RTX A6000 48GB | 1 | 41 / 43 GB | 66 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 41 / 43 GB | 80 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 43 / 43 GB | 124 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 43 / 57 GB | 180 |
| AMD Instinct MI210 64GB | 1 | 41 / 57 GB | 124 |
Serving and licence
vllm>=0.19.0 and sglang>=0.5.10 recommended by the card; official FP8 repo. KTransformers guide available. llama.cpp/Ollama run community GGUFs on builds with qwen3.5 support.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3.6-35B-A3B deployment
Preloaded with Qwen3.6-35B-A3B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.6-35B-A3B need?
The weights alone take about 72.0 GB in BF16, 36.0 GB in FP8 and 20.2 GB in 4-bit. On top of that each request holds a KV cache of about 0.08 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 1.68 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.6-35B-A3B?
For a single user at 8K context in FP8, 2 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 67 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.6-35B-A3B allowed for commercial use?
Qwen3.6-35B-A3B is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.6-35B-A3B on-prem?
vllm>=0.19.0 and sglang>=0.5.10 recommended by the card; official FP8 repo. KTransformers guide available. llama.cpp/Ollama run community GGUFs on builds with qwen3.5 support.