On-prem models / Alibaba Qwen
Run Qwen3.8-2.4T-A95B on-prem
Qwen's first open Max-class model: a 2.4T-parameter MoE (95B active) built on the Qwen3.5 hybrid Gated DeltaNet plus gated-attention layout. The open weights are text-only and always think; the hosted Qwen3.8-Max adds vision and non-thinking modes.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 4892 GB | 0.77 GB | 3.09 GB | 12.3 GB |
| FP8 | 2446 GB | 0.39 GB | 1.54 GB | 6.17 GB |
| INT4 (AWQ / GPTQ) | 1370 GB | 0.39 GB | 1.54 GB | 6.17 GB |
| GGUF Q4_K_M | 1468 GB | 0.39 GB | 1.54 GB | 6.17 GB |
KV cache from the published config: 92 layers, 4 KV heads x 256 dims, 23 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Frontier-class quality for complex text-to-SQL, multi-step agents and long document analysis, but the hardware bill is far beyond what a single plant or mid-size manufacturer will justify. Best treated as a central/group-IT model or skipped in favour of Qwen3.8-27B or the Flash-Next model for on-prem ERP use.
Strengths
- Largest open Qwen model, aimed at long-horizon coding and agent work
- Only 23 of 92 layers keep a growing KV cache, so long context is cheaper than in a pure-attention model
- reasoning_effort control and preserved thinking across turns
- Official SGLang and vLLM recipes, FP8 variant published
Watch out for
- About 4.9 TB of BF16 weights (about 2.4 TB in the FP8 repo); needs multi-node or very large-memory GPU clusters
- Thinking cannot be disabled, so every reply carries reasoning tokens and latency
- Text-only in the open release
- Licence is not plain Apache; check the MaaS clause if you resell inference
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI300X 192GB | 8 | 1383 / 1382 GB | 95 |
| AMD Instinct MI325X 256GB | 8 | 1383 / 1843 GB | 99 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 8 | 1383 / 1944 GB | 107 |
| AMD Instinct MI355X 288GB | 8 | 1383 / 2073 GB | 107 |
20 people at once, 16K context, FP8
Needs more than one server of any GPU we track.
Serving and licence
SGLang and vLLM have official Qwen3.8 recipes; TokenSpeed also listed. Expect multi-node tensor/expert parallelism; use the FP8 repo (Qwen/Qwen3.8-2.4T-A95B-FP8) to halve memory. No practical llama.cpp/Ollama path at this size.
Qwen3.8-Max License. Commercial use allowed; products over 100M MAU or US$20M monthly revenue must display the model name, and Model-as-a-Service or AI coding/office assistant businesses with over US$50M annual revenue need a separate licence (internal use exempt).
Design a Qwen3.8-2.4T-A95B deployment
Preloaded with Qwen3.8-2.4T-A95B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.8-2.4T-A95B need?
The weights alone take about 4892 GB in BF16, 2446 GB in FP8 and 1370 GB in 4-bit. On top of that each request holds a KV cache of about 0.39 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 7.72 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.8-2.4T-A95B?
For a single user at 8K context in 4-bit, 8 x AMD Instinct MI300X 192GB (192 GB each) fits, at an estimated 95 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.8-2.4T-A95B allowed for commercial use?
Qwen3.8-2.4T-A95B is released under Qwen3.8-Max License. Commercial use allowed; products over 100M MAU or US$20M monthly revenue must display the model name, and Model-as-a-Service or AI coding/office assistant businesses with over US$50M annual revenue need a separate licence (internal use exempt). Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.8-2.4T-A95B on-prem?
SGLang and vLLM have official Qwen3.8 recipes; TokenSpeed also listed. Expect multi-node tensor/expert parallelism; use the FP8 repo (Qwen/Qwen3.8-2.4T-A95B-FP8) to halve memory. No practical llama.cpp/Ollama path at this size.