On-prem models / Alibaba Qwen
Run Qwen3.8-27B on-prem
Dense 27B vision-language model in the Qwen3.8 generation, using the Qwen3.5 hybrid layout of 3 Gated DeltaNet layers per gated-attention layer. Thinking is on by default and can be disabled per request.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 55.6 GB | 0.54 GB | 2.15 GB | 8.59 GB |
| FP8 | 27.8 GB | 0.27 GB | 1.07 GB | 4.29 GB |
| INT4 (AWQ / GPTQ) | 15.6 GB | 0.27 GB | 1.07 GB | 4.29 GB |
| GGUF Q4_K_M | 16.7 GB | 0.27 GB | 1.07 GB | 4.29 GB |
KV cache from the published config: 64 layers, 4 KV heads x 256 dims, 16 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Probably the best single-GPU default for plant or HQ on-prem use: good at text-to-SQL over large ERP schemas, structured extraction from scanned POs, invoices and mill certificates, and tool-calling agents. Supports 201 languages per the Qwen3.5 family card, which helps multilingual plants.
Strengths
- Strong all-rounder that fits one 80 GB GPU in FP8 or one 48 GB card at 4-bit
- Native image, document and video understanding
- Small KV cache for its size because only 16 layers keep KV
- Apache 2.0
Watch out for
- About 56 GB in BF16; long context still needs extra memory for the 16 attention layers
- Dense model, so decode is slower than 3B-active MoE options
- Card advises keeping at least 128K context for best thinking behaviour
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 30 / 36 GB | 32 |
| NVIDIA L4 24GB | 2 | 32 / 43 GB | 11 |
| NVIDIA A10 24GB | 2 | 32 / 43 GB | 22 |
| NVIDIA L40S 48GB | 1 | 30 / 43 GB | 18 |
| NVIDIA RTX A6000 48GB | 1 | 30 / 43 GB | 16 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 30 / 43 GB | 20 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 32 / 43 GB | 35 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 32 / 57 GB | 58 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 2 | 42 / 43 GB | 9 |
| NVIDIA A10 24GB | 2 | 42 / 43 GB | 18 |
| NVIDIA L40S 48GB | 1 | 41 / 43 GB | 15 |
| NVIDIA RTX A6000 48GB | 1 | 41 / 43 GB | 14 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 41 / 43 GB | 17 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 42 / 43 GB | 30 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 42 / 57 GB | 50 |
| AMD Instinct MI210 64GB | 1 | 41 / 57 GB | 28 |
Serving and licence
SGLang cookbook and vLLM recipe for Qwen3.8-27B; Qwen/Qwen3.8-27B-FP8 is official. Use --reasoning-parser qwen3 and qwen3_coder tool parser. Community GGUFs run in llama.cpp/Ollama builds that support the qwen3.5 hybrid architecture.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3.8-27B deployment
Preloaded with Qwen3.8-27B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.8-27B need?
The weights alone take about 55.6 GB in BF16, 27.8 GB in FP8 and 15.6 GB in 4-bit. On top of that each request holds a KV cache of about 0.27 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 5.37 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.8-27B?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 32 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.8-27B allowed for commercial use?
Qwen3.8-27B is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.8-27B on-prem?
SGLang cookbook and vLLM recipe for Qwen3.8-27B; Qwen/Qwen3.8-27B-FP8 is official. Use --reasoning-parser qwen3 and qwen3_coder tool parser. Community GGUFs run in llama.cpp/Ollama builds that support the qwen3.5 hybrid architecture.