On-prem models / Alibaba Qwen
Run Qwen3.5-397B-A17B on-prem
The Qwen3.5 flagship open model: a 397B MoE with 17B active, native vision, and the hybrid Gated DeltaNet layout. It was Qwen's top Apache-licensed model before Qwen3.8.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 806 GB | 0.25 GB | 1.01 GB | 4.03 GB |
| FP8 | 403 GB | 0.13 GB | 0.50 GB | 2.01 GB |
| INT4 (AWQ / GPTQ) | 226 GB | 0.13 GB | 0.50 GB | 2.01 GB |
| GGUF Q4_K_M | 242 GB | 0.13 GB | 0.50 GB | 2.01 GB |
KV cache from the published config: 60 layers, 2 KV heads x 256 dims, 15 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Strong for group-level deployments that want one model for SQL generation, document vision and agents under a clean licence. Too heavy for most single-site installs; the 122B-A10B or 27B models cover most ERP tasks at a fraction of the cost.
Strengths
- Apache 2.0 at near-frontier scale
- 201 languages and dialects per the card
- Small KV footprint for a model of this size
- Official FP8 and GPTQ-Int4 repos
Watch out for
- About 800 GB BF16, about 400 GB FP8; needs an 8-GPU H200/B200-class node
- Superseded on quality by Qwen3.8 models
- Card advises at least 128K context to preserve thinking quality
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI325X 256GB | 1 | 228 / 230 GB | 198 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 229 / 230 GB | 170 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 228 / 243 GB | 240 |
| NVIDIA H200 SXM 141GB | 2 | 229 / 253 GB | 194 |
| NVIDIA H200 NVL 141GB | 2 | 229 / 253 GB | 194 |
| AMD Instinct MI355X 288GB | 1 | 228 / 259 GB | 240 |
| NVIDIA A100 40GB PCIe | 8 | 238 / 288 GB | 138 |
| NVIDIA A100 80GB PCIe | 4 | 232 / 288 GB | 140 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 8 | 421 / 460 GB | 107 |
| AMD Instinct MI325X 256GB | 2 | 412 / 460 GB | 149 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 415 / 460 GB | 136 |
| Apple Mac Studio M3 Ultra 512GB | 1 | 410 / 460 GB | 18 |
| Apple Mac Studio M5 Ultra 512GB | 1 | 410 / 460 GB | 25 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 2 | 412 / 486 GB | 175 |
| NVIDIA H200 SXM 141GB | 4 | 415 / 507 GB | 152 |
| NVIDIA H200 NVL 141GB | 4 | 415 / 507 GB | 152 |
Serving and licence
vLLM and SGLang support the Qwen3.5 hybrid (Gated DeltaNet + gated attention) architecture; use a recent release (Qwen3.6 card recommends vllm>=0.19.0 / sglang>=0.5.10), --reasoning-parser qwen3 and --tool-call-parser qwen3_coder; vLLM --language-model-only skips the vision encoder; MTP speculative decoding available. KTransformers supported; llama.cpp/Ollama community GGUFs exist. Official 8-GPU TP launch lines in the card.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3.5-397B-A17B deployment
Preloaded with Qwen3.5-397B-A17B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.5-397B-A17B need?
The weights alone take about 806 GB in BF16, 403 GB in FP8 and 226 GB in 4-bit. On top of that each request holds a KV cache of about 0.13 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 2.52 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.5-397B-A17B?
For a single user at 8K context in 4-bit, 1 x AMD Instinct MI325X 256GB (256 GB each) fits, at an estimated 198 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.5-397B-A17B allowed for commercial use?
Qwen3.5-397B-A17B is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.5-397B-A17B on-prem?
vLLM and SGLang support the Qwen3.5 hybrid (Gated DeltaNet + gated attention) architecture; use a recent release (Qwen3.6 card recommends vllm>=0.19.0 / sglang>=0.5.10), --reasoning-parser qwen3 and --tool-call-parser qwen3_coder; vLLM --language-model-only skips the vision encoder; MTP speculative decoding available. KTransformers supported; llama.cpp/Ollama community GGUFs exist. Official 8-GPU TP launch lines in the card.