On-prem models / Mistral AI
Run Mistral Small 4 119B A6B (2603) on-prem
A 128-expert MoE (4 active plus 1 shared) that merges Mistral's Instruct, Magistral reasoning and Devstral coding lines into one model. Reasoning effort can be set to none or high per request. Mistral also ships NVFP4 and EAGLE speculative-decoding checkpoints.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 238 GB | 0.19 GB | 0.75 GB | 3.02 GB |
| FP8 | 119 GB | 0.09 GB | 0.38 GB | 1.51 GB |
| INT4 (AWQ / GPTQ) | 66.6 GB | 0.09 GB | 0.38 GB | 1.51 GB |
| GGUF Q4_K_M | 71.4 GB | 0.09 GB | 0.38 GB | 1.51 GB |
KV cache from the published config: 36 layers, MLA latent 256 + 64 RoPE dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A strong European option for ERP agents: tool calling, document extraction from images, text-to-SQL with reasoning when needed, and good coverage of European and Asian languages. Throughput per GPU is good thanks to low active parameters.
Strengths
- One model for chat, reasoning and coding
- Only 6.5B active parameters for fast decode
- Compact MLA KV cache
- Native function calling, JSON output and vision
- Apache 2.0
Watch out for
- 119B total weights need about 240GB at bf16, or a single large GPU at NVFP4
- MLA and MoE need recent engine versions
- Reasoning mode adds latency
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 2 | 70 / 72 GB | 194 |
| NVIDIA A100 80GB PCIe | 1 | 69 / 72 GB | 174 |
| NVIDIA A100 80GB SXM | 1 | 69 / 72 GB | 181 |
| NVIDIA H100 PCIe 80GB | 1 | 69 / 72 GB | 179 |
| NVIDIA H100 SXM 80GB | 1 | 69 / 72 GB | 253 |
| NVIDIA H100 NVL 94GB | 1 | 69 / 84 GB | 278 |
| NVIDIA L4 24GB | 4 | 73 / 86 GB | 91 |
| NVIDIA A10 24GB | 4 | 73 / 86 GB | 144 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H200 SXM 141GB | 1 | 125 / 126 GB | 185 |
| NVIDIA H200 NVL 141GB | 1 | 125 / 126 GB | 185 |
| NVIDIA A100 40GB PCIe | 4 | 129 / 144 GB | 154 |
| NVIDIA A100 80GB PCIe | 2 | 126 / 144 GB | 131 |
| NVIDIA A100 80GB SXM | 2 | 126 / 144 GB | 136 |
| NVIDIA H100 PCIe 80GB | 2 | 126 / 144 GB | 135 |
| NVIDIA H100 SXM 80GB | 2 | 126 / 144 GB | 188 |
| NVIDIA B200 180GB | 1 | 125 / 162 GB | 260 |
Serving and licence
vLLM with --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral; SGLang also supported. Use the NVFP4 checkpoint on Blackwell and the EAGLE head for speculative decoding.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Mistral Small 4 119B A6B (2603) deployment
Preloaded with Mistral Small 4 119B A6B (2603). Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Mistral Small 4 119B A6B (2603) need?
The weights alone take about 238 GB in BF16, 119 GB in FP8 and 66.6 GB in 4-bit. On top of that each request holds a KV cache of about 0.09 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 1.89 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Mistral Small 4 119B A6B (2603)?
For a single user at 8K context in 4-bit, 2 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 194 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Mistral Small 4 119B A6B (2603) allowed for commercial use?
Mistral Small 4 119B A6B (2603) is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Mistral Small 4 119B A6B (2603) on-prem?
vLLM with --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral; SGLang also supported. Use the NVFP4 checkpoint on Blackwell and the EAGLE head for speculative decoding.