On-prem models / Alibaba Qwen
Run Qwen3.5-122B-A10B on-prem
Mid-size Qwen3.5 MoE with 10B active parameters and native vision. Popular as a balance between quality and a single-node footprint.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 250 GB | 0.20 GB | 0.81 GB | 3.22 GB |
| FP8 | 125 GB | 0.10 GB | 0.40 GB | 1.61 GB |
| INT4 (AWQ / GPTQ) | 70.0 GB | 0.10 GB | 0.40 GB | 1.61 GB |
| GGUF Q4_K_M | 75.0 GB | 0.10 GB | 0.40 GB | 1.61 GB |
KV cache from the published config: 48 layers, 2 KV heads x 256 dims, 12 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A sensible on-prem workhorse for a manufacturer's central IT: handles multi-table text-to-SQL, OCR-style extraction from scanned paperwork and multi-step tool use with headroom. Use non-thinking mode for bulk extraction to control latency.
Strengths
- Runs on 2x80 GB in FP8, or one 80 to 96 GB GPU at short context with the official GPTQ-Int4 repo
- 10B active keeps throughput high
- Vision plus 201-language coverage
- Apache 2.0
Watch out for
- About 250 GB in BF16
- Older than Qwen3.6/3.8; coding and agent behaviour improved in later releases
- Thinking default adds tokens on simple tasks
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 80GB PCIe | 1 | 72 / 72 GB | 125 |
| NVIDIA A100 80GB SXM | 1 | 72 / 72 GB | 131 |
| NVIDIA H100 PCIe 80GB | 1 | 72 / 72 GB | 129 |
| NVIDIA H100 SXM 80GB | 1 | 72 / 72 GB | 190 |
| NVIDIA H100 NVL 94GB | 1 | 72 / 84 GB | 212 |
| NVIDIA L4 24GB | 4 | 77 / 86 GB | 64 |
| NVIDIA A10 24GB | 4 | 77 / 86 GB | 104 |
| NVIDIA L40S 48GB | 2 | 74 / 86 GB | 95 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 4 | 136 / 144 GB | 118 |
| NVIDIA A100 80GB PCIe | 2 | 133 / 144 GB | 98 |
| NVIDIA A100 80GB SXM | 2 | 133 / 144 GB | 102 |
| NVIDIA H100 PCIe 80GB | 2 | 133 / 144 GB | 101 |
| NVIDIA H100 SXM 80GB | 2 | 133 / 144 GB | 145 |
| NVIDIA B200 180GB | 1 | 131 / 162 GB | 206 |
| NVIDIA H100 NVL 94GB | 2 | 133 / 169 GB | 159 |
| NVIDIA L4 24GB | 8 | 142 / 172 GB | 54 |
Serving and licence
vLLM and SGLang support the Qwen3.5 hybrid (Gated DeltaNet + gated attention) architecture; use a recent release (Qwen3.6 card recommends vllm>=0.19.0 / sglang>=0.5.10), --reasoning-parser qwen3 and --tool-call-parser qwen3_coder; vLLM --language-model-only skips the vision encoder; MTP speculative decoding available. KTransformers supported; llama.cpp/Ollama community GGUFs exist.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3.5-122B-A10B deployment
Preloaded with Qwen3.5-122B-A10B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.5-122B-A10B need?
The weights alone take about 250 GB in BF16, 125 GB in FP8 and 70.0 GB in 4-bit. On top of that each request holds a KV cache of about 0.10 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 2.01 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.5-122B-A10B?
For a single user at 8K context in 4-bit, 1 x NVIDIA A100 80GB PCIe (80 GB each) fits, at an estimated 125 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.5-122B-A10B allowed for commercial use?
Qwen3.5-122B-A10B is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.5-122B-A10B on-prem?
vLLM and SGLang support the Qwen3.5 hybrid (Gated DeltaNet + gated attention) architecture; use a recent release (Qwen3.6 card recommends vllm>=0.19.0 / sglang>=0.5.10), --reasoning-parser qwen3 and --tool-call-parser qwen3_coder; vLLM --language-model-only skips the vision encoder; MTP speculative decoding available. KTransformers supported; llama.cpp/Ollama community GGUFs exist.