On-prem models / Ant Group inclusionAI
Run Ling-3.0-flash on-prem
Ant Group's hybrid reasoning MoE: 124B total, 5.1B active, with a 5:1 stack of Kimi Delta Attention and MLA and 1/64 expert sparsity. Trained across 10,000+ interactive agent environments.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 248 GB | 0.07 GB | 0.26 GB | 1.06 GB |
| FP8 | 124 GB | 0.03 GB | 0.13 GB | 0.53 GB |
| INT4 (AWQ / GPTQ) | 69.4 GB | 0.03 GB | 0.13 GB | 0.53 GB |
| GGUF Q4_K_M | 74.4 GB | 0.03 GB | 0.13 GB | 0.53 GB |
KV cache from the published config: 42 layers, 32 KV heads x 192 dims, 7 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Efficient choice for long-running ERP agents and text-to-SQL where many concurrent sessions matter, under a clean licence. Pair with a VLM for scanned documents.
Strengths
- Very low active compute and tiny KV cache
- Thinking on by default, switchable per request
- MIT licence
- Integrates with SGLang HiCache + Mooncake for long-session caching
Watch out for
- Text only (a separate Ling-3.0-flash-VL exists)
- Needs SGLang dev image or vLLM built from source
- About 250 GB in BF16; card suggests 4x 141 GB or 8x 80 GB GPUs
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 80GB PCIe | 1 | 71 / 72 GB | 208 |
| NVIDIA A100 80GB SXM | 1 | 71 / 72 GB | 216 |
| NVIDIA H100 PCIe 80GB | 1 | 71 / 72 GB | 213 |
| NVIDIA H100 SXM 80GB | 1 | 71 / 72 GB | 294 |
| NVIDIA H100 NVL 94GB | 1 | 71 / 84 GB | 319 |
| NVIDIA L4 24GB | 4 | 76 / 86 GB | 106 |
| NVIDIA A10 24GB | 4 | 76 / 86 GB | 158 |
| NVIDIA L40S 48GB | 2 | 73 / 86 GB | 155 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H200 SXM 141GB | 1 | 127 / 126 GB | 239 |
| NVIDIA H200 NVL 141GB | 1 | 127 / 126 GB | 239 |
| NVIDIA A100 40GB PCIe | 4 | 132 / 144 GB | 178 |
| NVIDIA A100 80GB PCIe | 2 | 129 / 144 GB | 166 |
| NVIDIA A100 80GB SXM | 2 | 129 / 144 GB | 171 |
| NVIDIA H100 PCIe 80GB | 2 | 129 / 144 GB | 169 |
| NVIDIA H100 SXM 80GB | 2 | 129 / 144 GB | 224 |
| NVIDIA B200 180GB | 1 | 127 / 162 GB | 322 |
Serving and licence
SGLang (lmsysorg/sglang:dev-Ling-3.0-flash, --reasoning-parser ling3, --tool-call-parser ling3, built-in MTP) and vLLM from source with --tool-call-parser ling3; GGUF available for llama.cpp.
MIT. MIT licence: commercial use allowed without revenue or user caps.
Design a Ling-3.0-flash deployment
Preloaded with Ling-3.0-flash. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Ling-3.0-flash need?
The weights alone take about 248 GB in BF16, 124 GB in FP8 and 69.4 GB in 4-bit. On top of that each request holds a KV cache of about 0.03 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 0.66 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Ling-3.0-flash?
For a single user at 8K context in 4-bit, 1 x NVIDIA A100 80GB PCIe (80 GB each) fits, at an estimated 208 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Ling-3.0-flash allowed for commercial use?
Ling-3.0-flash is released under MIT. MIT licence: commercial use allowed without revenue or user caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Ling-3.0-flash on-prem?
SGLang (lmsysorg/sglang:dev-Ling-3.0-flash, --reasoning-parser ling3, --tool-call-parser ling3, built-in MTP) and vLLM from source with --tool-call-parser ling3; GGUF available for llama.cpp.