On-prem models / Alibaba Qwen
Run Qwen3.8-Flash-Next on-prem
Experimental preview of the architecture planned for Qwen4: Gated DeltaNet plus Qwen Sparse Attention, gated residual streams and a large n-gram embedding table. About 6B parameters are active per token out of roughly 180B stored.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 360 GB | 0.20 GB | 0.81 GB | 3.22 GB |
| FP8 | 180 GB | 0.10 GB | 0.40 GB | 1.61 GB |
| INT4 (AWQ / GPTQ) | 101 GB | 0.10 GB | 0.40 GB | 1.61 GB |
| GGUF Q4_K_M | 108 GB | 0.10 GB | 0.40 GB | 1.61 GB |
KV cache from the published config: 48 layers, 2 KV heads x 256 dims, 12 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Attractive for on-prem agents that read POs, invoices and certificates (vision) and then call ERP APIs, because decode cost is close to a small model. Memory, not compute, is the constraint: plan for a 4 to 8 GPU node. Watch the licence if you plan to expose it as a service to customers.
Strengths
- Very low active compute (about 6B) for its quality tier
- Native image and video input
- Only 12 layers keep a KV cache and those use block-sparse attention, so long context is cheap
- Thinking can be switched off per request; reasoning_effort supported
Watch out for
- New architecture (model_type qwen4_exp) needs the latest vLLM/SGLang/Transformers; little third-party tooling yet
- About 360 GB of BF16 weights; FP8 repo roughly halves it
- Labelled experimental preview
- Licence blocks hosted-API resale without a separate deal
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 4 | 107 / 115 GB | 214 |
| AMD Instinct MI210 64GB | 2 | 104 / 115 GB | 196 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 1 | 103 / 115 GB | 282 |
| NVIDIA DGX Spark (GB10) 128GB | 1 | 103 / 115 GB | 34 |
| AMD Ryzen AI Max+ 395 128GB system | 1 | 103 / 115 GB | 32 |
| NVIDIA H200 SXM 141GB | 1 | 103 / 126 GB | 325 |
| NVIDIA H200 NVL 141GB | 1 | 103 / 126 GB | 325 |
| NVIDIA A100 40GB PCIe | 4 | 107 / 144 GB | 206 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 4 | 191 / 230 GB | 149 |
| AMD Instinct MI325X 256GB | 1 | 186 / 230 GB | 223 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 188 / 230 GB | 194 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 186 / 243 GB | 267 |
| NVIDIA H200 SXM 141GB | 2 | 188 / 253 GB | 220 |
| NVIDIA H200 NVL 141GB | 2 | 188 / 253 GB | 220 |
| AMD Instinct MI355X 288GB | 1 | 186 / 259 GB | 267 |
| NVIDIA A100 40GB PCIe | 8 | 197 / 288 GB | 162 |
Serving and licence
Official SGLang cookbook and vLLM recipe (Qwen3.8-Flash-Next); Transformers and TokenSpeed also listed. Use the -FP8 repo on Hopper/Blackwell. No mainstream llama.cpp support confirmed.
Qwen Community License 1.0. Commercial use allowed with name display above 100M MAU or US$20M monthly revenue, but any Model-as-a-Service or AI coding/office assistant business needs a separate licence regardless of size (internal use exempt).
Design a Qwen3.8-Flash-Next deployment
Preloaded with Qwen3.8-Flash-Next. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.8-Flash-Next need?
The weights alone take about 360 GB in BF16, 180 GB in FP8 and 101 GB in 4-bit. On top of that each request holds a KV cache of about 0.10 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 2.01 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.8-Flash-Next?
For a single user at 8K context in 4-bit, 4 x NVIDIA GeForce RTX 5090 32GB (32 GB each) fits, at an estimated 214 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.8-Flash-Next allowed for commercial use?
Qwen3.8-Flash-Next is released under Qwen Community License 1.0. Commercial use allowed with name display above 100M MAU or US$20M monthly revenue, but any Model-as-a-Service or AI coding/office assistant business needs a separate licence regardless of size (internal use exempt). Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.8-Flash-Next on-prem?
Official SGLang cookbook and vLLM recipe (Qwen3.8-Flash-Next); Transformers and TokenSpeed also listed. Use the -FP8 repo on Hopper/Blackwell. No mainstream llama.cpp support confirmed.