On-prem models / Alibaba Qwen
Run Qwen3.5-9B on-prem
Small dense Qwen3.5 vision-language model, one of the most downloaded Qwen releases of 2026. Same hybrid attention layout as the larger Qwen3.5 models.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 19.4 GB | 0.27 GB | 1.07 GB | 4.29 GB |
| FP8 | 9.70 GB | 0.13 GB | 0.54 GB | 2.15 GB |
| INT4 (AWQ / GPTQ) | 5.43 GB | 0.13 GB | 0.54 GB | 2.15 GB |
| GGUF Q4_K_M | 5.82 GB | 0.13 GB | 0.54 GB | 2.15 GB |
KV cache from the published config: 32 layers, 4 KV heads x 256 dims, 8 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Good for edge and shop-floor use: label or document field extraction, quick lookups and simple tool calls on modest hardware. Pair it with schema retrieval and validation if used for text-to-SQL.
Strengths
- Runs on a 24 GB GPU in BF16 and on laptops or edge boxes at 4-bit
- Image and document understanding at small size
- Very small KV cache (8 attention layers)
- Apache 2.0
Watch out for
- Limited reasoning depth for complex SQL or long agent chains
- More prone to schema hallucination than 27B+ models
- Thinking default; turn off for fast extraction
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 12 / 21 GB | 18 |
| NVIDIA A10 24GB | 1 | 12 / 21 GB | 35 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 12 / 21 GB | 57 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 12 / 28 GB | 94 |
| NVIDIA A100 40GB PCIe | 1 | 12 / 36 GB | 84 |
| NVIDIA L40S 48GB | 1 | 12 / 43 GB | 49 |
| NVIDIA RTX A6000 48GB | 1 | 12 / 43 GB | 44 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 12 / 43 GB | 54 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 17 / 21 GB | 14 |
| NVIDIA A10 24GB | 1 | 17 / 21 GB | 28 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 17 / 21 GB | 45 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 17 / 28 GB | 76 |
| NVIDIA A100 40GB PCIe | 1 | 17 / 36 GB | 67 |
| NVIDIA L40S 48GB | 1 | 17 / 43 GB | 39 |
| NVIDIA RTX A6000 48GB | 1 | 17 / 43 GB | 35 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 17 / 43 GB | 43 |
Serving and licence
vLLM and SGLang support the Qwen3.5 hybrid (Gated DeltaNet + gated attention) architecture; use a recent release (Qwen3.6 card recommends vllm>=0.19.0 / sglang>=0.5.10), --reasoning-parser qwen3 and --tool-call-parser qwen3_coder; vLLM --language-model-only skips the vision encoder; MTP speculative decoding available. KTransformers supported; llama.cpp/Ollama community GGUFs exist. Single-GPU (TP 1) launch lines are in the card.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3.5-9B deployment
Preloaded with Qwen3.5-9B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3.5-9B need?
The weights alone take about 19.4 GB in BF16, 9.70 GB in FP8 and 5.43 GB in 4-bit. On top of that each request holds a KV cache of about 0.13 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 2.68 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3.5-9B?
For a single user at 8K context in FP8, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 18 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3.5-9B allowed for commercial use?
Qwen3.5-9B is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3.5-9B on-prem?
vLLM and SGLang support the Qwen3.5 hybrid (Gated DeltaNet + gated attention) architecture; use a recent release (Qwen3.6 card recommends vllm>=0.19.0 / sglang>=0.5.10), --reasoning-parser qwen3 and --tool-call-parser qwen3_coder; vLLM --language-model-only skips the vision encoder; MTP speculative decoding available. KTransformers supported; llama.cpp/Ollama community GGUFs exist. Single-GPU (TP 1) launch lines are in the card.