On-prem models / Zhipu AI (Z.ai)
Run GLM-5.3-Flash on-prem
First natively multimodal GLM-5 model: 320B total, 18B active, with a hybrid sparse plus linear attention design and manifold-constrained hyper-connections. Z.ai positions it above GLM-5.2 at a tenth of the price.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 640 GB | 0.09 GB | 0.37 GB | 1.48 GB |
| FP8 | 320 GB | 0.05 GB | 0.18 GB | 0.74 GB |
| INT4 (AWQ / GPTQ) | 179 GB | 0.05 GB | 0.18 GB | 0.74 GB |
| GGUF Q4_K_M | 192 GB | 0.05 GB | 0.18 GB | 0.74 GB |
KV cache from the published config: 45 layers, 64 KV heads x 256 dims, 11 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
One of the best-balanced options for on-prem ERP agents in late 2026: reads scanned documents, writes SQL, calls tools, and the MIT licence avoids legal friction. Fits a single 8-GPU server with room for long contexts.
Strengths
- Very high download volume (most downloaded GLM model at time of writing)
- Only 11 layers keep KV, so long context is cheap
- Native image input
- MIT licence
- Official FP8 and BF16 repos
Watch out for
- About 320 GB in FP8; plan for 4x H200 or 8x H100-class GPUs
- New architecture (glm5_next) needs current vLLM/SGLang
- Defaults to max reasoning effort; tune for latency
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 4 | 186 / 230 GB | 135 |
| AMD Instinct MI325X 256GB | 1 | 181 / 230 GB | 191 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 183 / 230 GB | 173 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 181 / 243 GB | 232 |
| NVIDIA H200 SXM 141GB | 2 | 183 / 253 GB | 200 |
| NVIDIA H200 NVL 141GB | 2 | 183 / 253 GB | 200 |
| AMD Instinct MI355X 288GB | 1 | 181 / 259 GB | 232 |
| NVIDIA A100 40GB PCIe | 8 | 192 / 288 GB | 155 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H100 NVL 94GB | 4 | 328 / 338 GB | 154 |
| NVIDIA L40S 48GB | 8 | 334 / 345 GB | 85 |
| NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB | 4 | 328 / 345 GB | 98 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB | 4 | 328 / 345 GB | 90 |
| AMD Instinct MI300X 192GB | 2 | 325 / 345 GB | 147 |
| AMD Instinct MI210 64GB | 8 | 334 / 460 GB | 120 |
| AMD Instinct MI325X 256GB | 2 | 325 / 460 GB | 159 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 328 / 460 GB | 150 |
Serving and licence
SGLang cookbook, vLLM recipe and KTransformers tutorial for GLM-5.3-Flash; use reasoning_effort low/high/max and clear_thinking=true for chat.
MIT. MIT licence: commercial use allowed without revenue or user caps.
Design a GLM-5.3-Flash deployment
Preloaded with GLM-5.3-Flash. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does GLM-5.3-Flash need?
The weights alone take about 640 GB in BF16, 320 GB in FP8 and 179 GB in 4-bit. On top of that each request holds a KV cache of about 0.05 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 0.92 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs GLM-5.3-Flash?
For a single user at 8K context in 4-bit, 4 x AMD Instinct MI210 64GB (64 GB each) fits, at an estimated 135 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is GLM-5.3-Flash allowed for commercial use?
GLM-5.3-Flash is released under MIT. MIT licence: commercial use allowed without revenue or user caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve GLM-5.3-Flash on-prem?
SGLang cookbook, vLLM recipe and KTransformers tutorial for GLM-5.3-Flash; use reasoning_effort low/high/max and clear_thinking=true for chat.