On-prem models / Xiaomi MiMo
Run MiMo-V2.6-Pro-RL on-prem
Xiaomi's 1.02T-parameter omnimodal MoE (42B active) covering text, image, video and audio input, trained with a single mixed RL run across coding, agents, vision and security. Weights store MoE experts in MXFP4 with FP8 elsewhere.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 2040 GB | 0.55 GB | 2.06 GB | 8.10 GB |
| FP8 | 1020 GB | 0.28 GB | 1.03 GB | 4.05 GB |
| INT4 (AWQ / GPTQ) | 571 GB | 0.28 GB | 1.03 GB | 4.05 GB |
| GGUF Q4_K_M | 612 GB | 0.28 GB | 1.03 GB | 4.05 GB |
KV cache from the published config: 70 layers, 8 KV heads x 192 dims, 60 sliding-window layers (128 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Interesting for plants that want one model for documents, photos and voice notes from the floor, with a clean licence. Sized for data-centre deployment; the Flash variant is the realistic on-prem choice.
Strengths
- Very small long-context KV thanks to 6:1 sliding-window to global layers
- Audio and video input in the same model
- MIT licence
- Strong agent and coding results per the card
Watch out for
- Needs a large multi-GPU deployment (card examples use TP 16 in SGLang, TP 8 in vLLM)
- Stable vLLM may lag; card points to a dedicated Docker image
- Young ecosystem outside SGLang/vLLM
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA B200 180GB | 4 | 578 / 648 GB | 160 |
| NVIDIA H100 NVL 94GB | 8 | 584 / 676 GB | 128 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB | 8 | 584 / 691 GB | 92 |
| AMD Instinct MI300X 192GB | 4 | 578 / 691 GB | 139 |
| AMD Instinct MI325X 256GB | 4 | 578 / 921 GB | 145 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 8 | 584 / 921 GB | 126 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 4 | 578 / 972 GB | 160 |
| NVIDIA H200 SXM 141GB | 8 | 584 / 1015 GB | 135 |
20 people at once, 16K context, MXFP4
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 80GB PCIe | 8 | 564 / 576 GB | 93 |
| NVIDIA A100 80GB SXM | 8 | 564 / 576 GB | 96 |
| NVIDIA H100 PCIe 80GB | 8 | 564 / 576 GB | 95 |
| NVIDIA H100 SXM 80GB | 8 | 564 / 576 GB | 116 |
| NVIDIA B200 180GB | 4 | 558 / 648 GB | 152 |
| NVIDIA H100 NVL 94GB | 8 | 564 / 676 GB | 122 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB | 8 | 564 / 691 GB | 85 |
| AMD Instinct MI300X 192GB | 4 | 558 / 691 GB | 129 |
Serving and licence
SGLang MiMo cookbook (lmsysorg/sglang:latest, --tool-call-parser mimo, DP attention); vLLM MiMo recipe with vllm/vllm-openai:mimov25-cu129 image.
MIT. MIT licence: commercial use allowed without revenue or user caps.
Design a MiMo-V2.6-Pro-RL deployment
Preloaded with MiMo-V2.6-Pro-RL. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does MiMo-V2.6-Pro-RL need?
The weights alone take about 2040 GB in BF16, 1020 GB in FP8 and 571 GB in 4-bit. On top of that each request holds a KV cache of about 0.28 GB at 8K tokens of context (MXFP4 cache), so 20 people generating at once need roughly 5.51 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs MiMo-V2.6-Pro-RL?
For a single user at 8K context in 4-bit, 4 x NVIDIA B200 180GB (180 GB each) fits, at an estimated 160 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is MiMo-V2.6-Pro-RL allowed for commercial use?
MiMo-V2.6-Pro-RL is released under MIT. MIT licence: commercial use allowed without revenue or user caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve MiMo-V2.6-Pro-RL on-prem?
SGLang MiMo cookbook (lmsysorg/sglang:latest, --tool-call-parser mimo, DP attention); vLLM MiMo recipe with vllm/vllm-openai:mimov25-cu129 image.