On-prem models / Mistral AI
Run Mistral Large 3 675B Instruct 2512 on-prem
A granular MoE with 128 experts (4 active), 41B active and 675B total parameters including a 2.5B vision encoder, trained from scratch on 3,000 H200s. The instruct model ships natively in FP8, with NVFP4 and BF16 variants.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 1350 GB | 0.58 GB | 2.30 GB | 9.21 GB |
| FP8 | 675 GB | 0.29 GB | 1.15 GB | 4.61 GB |
| INT4 (AWQ / GPTQ) | 378 GB | 0.29 GB | 1.15 GB | 4.61 GB |
| GGUF Q4_K_M | 405 GB | 0.29 GB | 1.15 GB | 4.61 GB |
KV cache from the published config: 61 layers, MLA latent 512 + 64 RoPE dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Suitable for central enterprise deployments needing a licence-clean frontier-class assistant for long documents, multilingual sites and tool-calling agents. Overkill for single-plant installs.
Strengths
- Largest Apache 2.0 Western open model
- FP8 fits a single 8x B200 or H200 node; NVFP4 fits 8x H100 or A100
- Compact MLA KV cache
- Strong long-document and enterprise knowledge work
Watch out for
- Not a dedicated reasoning model per Mistral
- Behind vision-first models on multimodal tasks per Mistral
- Complex to deploy efficiently
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 8 | 391 / 460 GB | 98 |
| AMD Instinct MI325X 256GB | 2 | 382 / 460 GB | 134 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 385 / 460 GB | 125 |
| Apple Mac Studio M3 Ultra 512GB | 1 | 380 / 460 GB | 16 |
| Apple Mac Studio M5 Ultra 512GB | 1 | 380 / 460 GB | 23 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 2 | 382 / 486 GB | 159 |
| NVIDIA H200 SXM 141GB | 4 | 385 / 507 GB | 141 |
| NVIDIA H200 NVL 141GB | 4 | 385 / 507 GB | 141 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI325X 256GB | 4 | 693 / 921 GB | 110 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 8 | 699 / 921 GB | 105 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 4 | 693 / 972 GB | 129 |
| NVIDIA H200 SXM 141GB | 8 | 699 / 1015 GB | 117 |
| NVIDIA H200 NVL 141GB | 8 | 699 / 1015 GB | 117 |
| AMD Instinct MI355X 288GB | 4 | 693 / 1036 GB | 129 |
| NVIDIA B200 180GB | 8 | 699 / 1296 GB | 139 |
| AMD Instinct MI300X 192GB | 8 | 699 / 1382 GB | 121 |
Serving and licence
vLLM is the reference engine (--tool-call-parser mistral --enable-auto-tool-choice); an EAGLE checkpoint is available. SGLang and TensorRT-LLM support DeepSeek-style MLA MoE.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Mistral Large 3 675B Instruct 2512 deployment
Preloaded with Mistral Large 3 675B Instruct 2512. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Mistral Large 3 675B Instruct 2512 need?
The weights alone take about 1350 GB in BF16, 675 GB in FP8 and 378 GB in 4-bit. On top of that each request holds a KV cache of about 0.29 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 5.76 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Mistral Large 3 675B Instruct 2512?
For a single user at 8K context in 4-bit, 8 x AMD Instinct MI210 64GB (64 GB each) fits, at an estimated 98 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Mistral Large 3 675B Instruct 2512 allowed for commercial use?
Mistral Large 3 675B Instruct 2512 is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Mistral Large 3 675B Instruct 2512 on-prem?
vLLM is the reference engine (--tool-call-parser mistral --enable-auto-tool-choice); an EAGLE checkpoint is available. SGLang and TensorRT-LLM support DeepSeek-style MLA MoE.