On-prem models / Alibaba Qwen
Run Qwen3-VL-8B-Instruct on-prem
8B dense vision-language model from the Qwen3-VL line, the Instruct (non-thinking) edition. Still one of the most downloaded small VLMs on Hugging Face.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 17.6 GB | 1.21 GB | 4.83 GB | 19.3 GB |
| FP8 | 8.80 GB | 0.60 GB | 2.42 GB | 9.66 GB |
| INT4 (AWQ / GPTQ) | 4.93 GB | 0.60 GB | 2.42 GB | 9.66 GB |
| GGUF Q4_K_M | 5.28 GB | 0.60 GB | 2.42 GB | 9.66 GB |
KV cache from the published config: 36 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A proven, cheap choice for document extraction: purchase orders, invoices, packing lists, material certificates and nameplate photos into JSON for ERP entry. Not the model to use for complex SQL generation.
Strengths
- OCR in 32 languages, robust to blur, tilt and low light per the card
- Improved long-document structure parsing
- Fits a 24 GB GPU in BF16
- Visual agent and GUI operation abilities
- Mature support across vLLM, SGLang, llama.cpp and Ollama
Watch out for
- Full-attention GQA, so KV grows on all 36 layers at long context
- Text reasoning weaker than newer Qwen3.5/3.8 models of similar size
- Separate Thinking edition needed for step-by-step reasoning
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 11 / 21 GB | 19 |
| NVIDIA A10 24GB | 1 | 11 / 21 GB | 37 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 11 / 21 GB | 60 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 11 / 28 GB | 100 |
| NVIDIA A100 40GB PCIe | 1 | 11 / 36 GB | 89 |
| NVIDIA L40S 48GB | 1 | 11 / 43 GB | 52 |
| NVIDIA RTX A6000 48GB | 1 | 11 / 43 GB | 47 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 11 / 43 GB | 58 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 35 / 36 GB | 41 |
| NVIDIA L4 24GB | 2 | 36 / 43 GB | 15 |
| NVIDIA A10 24GB | 2 | 36 / 43 GB | 29 |
| NVIDIA L40S 48GB | 1 | 35 / 43 GB | 24 |
| NVIDIA RTX A6000 48GB | 1 | 35 / 43 GB | 21 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 35 / 43 GB | 26 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 36 / 43 GB | 46 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 36 / 57 GB | 76 |
Serving and licence
vLLM and SGLang support Qwen3-VL natively; Transformers >= 4.57. llama.cpp/Ollama support Qwen3-VL GGUFs. Enable flash attention for multi-image work.
Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps.
Design a Qwen3-VL-8B-Instruct deployment
Preloaded with Qwen3-VL-8B-Instruct. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Qwen3-VL-8B-Instruct need?
The weights alone take about 17.6 GB in BF16, 8.80 GB in FP8 and 4.93 GB in 4-bit. On top of that each request holds a KV cache of about 0.60 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 12.1 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Qwen3-VL-8B-Instruct?
For a single user at 8K context in FP8, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 19 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Qwen3-VL-8B-Instruct allowed for commercial use?
Qwen3-VL-8B-Instruct is released under Apache-2.0. Apache 2.0: commercial use, modification and redistribution allowed with notice; no usage caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Qwen3-VL-8B-Instruct on-prem?
vLLM and SGLang support Qwen3-VL natively; Transformers >= 4.57. llama.cpp/Ollama support Qwen3-VL GGUFs. Enable flash attention for multi-image work.