On-prem models / Mistral AI
Run Mistral Medium 3.5 128B on-prem
Mistral's first merged flagship: a dense 128B model covering instruct, reasoning and coding, replacing Medium 3.1, Magistral and Devstral 2 in Mistral's products. The card reports 77.6 percent on SWE-Bench Verified. Mistral also ships an EAGLE draft model.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 256 GB | 2.95 GB | 11.8 GB | 47.2 GB |
| FP8 | 128 GB | 1.48 GB | 5.91 GB | 23.6 GB |
| INT4 (AWQ / GPTQ) | 71.7 GB | 1.48 GB | 5.91 GB | 23.6 GB |
| GGUF Q4_K_M | 76.8 GB | 1.48 GB | 5.91 GB | 23.6 GB |
KV cache from the published config: 88 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Excellent quality for complex ERP agents, SQL and code customization, but most mid-size and large manufacturers exceed the 20 million USD monthly revenue cap and would need a commercial licence. Smaller firms can use it freely.
Strengths
- Highest-quality Mistral open-weight model for agentic coding per the card
- Configurable reasoning effort
- Vision encoder trained from scratch for variable image sizes
- Wide language coverage
Watch out for
- Revenue cap licence excludes larger enterprises without a paid licence
- Dense 128B is expensive to serve and has a large KV cache
- Needs a multi-GPU node at bf16 or FP8
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H100 NVL 94GB | 1 | 75 / 84 GB | 31 |
| NVIDIA L4 24GB | 4 | 80 / 86 GB | 8 |
| NVIDIA A10 24GB | 4 | 80 / 86 GB | 16 |
| NVIDIA L40S 48GB | 2 | 77 / 86 GB | 12 |
| NVIDIA RTX A6000 48GB | 2 | 77 / 86 GB | 11 |
| NVIDIA RTX 6000 Ada 48GB | 2 | 77 / 86 GB | 14 |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96GB | 1 | 75 / 86 GB | 15 |
| NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB | 1 | 75 / 86 GB | 15 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 4 | 194 / 230 GB | 19 |
| AMD Instinct MI325X 256GB | 1 | 189 / 230 GB | 22 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 191 / 230 GB | 23 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 189 / 243 GB | 29 |
| NVIDIA H200 SXM 141GB | 2 | 191 / 253 GB | 30 |
| NVIDIA H200 NVL 141GB | 2 | 191 / 253 GB | 30 |
| AMD Instinct MI355X 288GB | 1 | 189 / 259 GB | 29 |
| NVIDIA A100 40GB PCIe | 8 | 200 / 288 GB | 30 |
Serving and licence
vLLM and SGLang with --tool-call-parser mistral --reasoning-parser mistral --enable-auto-tool-choice; mistral_common 1.11.1 or newer and Transformers 5.4 or newer. Use the EAGLE checkpoint for speculative decoding.
Modified MIT License. Commercial use is free only if your company's global consolidated monthly revenue is 20 million USD or less; larger companies need a commercial licence from Mistral.
Design a Mistral Medium 3.5 128B deployment
Preloaded with Mistral Medium 3.5 128B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Mistral Medium 3.5 128B need?
The weights alone take about 256 GB in BF16, 128 GB in FP8 and 71.7 GB in 4-bit. On top of that each request holds a KV cache of about 1.48 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 29.5 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Mistral Medium 3.5 128B?
For a single user at 8K context in 4-bit, 1 x NVIDIA H100 NVL 94GB (94 GB each) fits, at an estimated 31 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Mistral Medium 3.5 128B allowed for commercial use?
Mistral Medium 3.5 128B is released under Modified MIT License. Commercial use is free only if your company's global consolidated monthly revenue is 20 million USD or less; larger companies need a commercial licence from Mistral. Check the licence text on the model card with your legal team before production use.
Which engine should serve Mistral Medium 3.5 128B on-prem?
vLLM and SGLang with --tool-call-parser mistral --reasoning-parser mistral --enable-auto-tool-choice; mistral_common 1.11.1 or newer and Transformers 5.4 or newer. Use the EAGLE checkpoint for speculative decoding.