On-prem models / Mistral AI
Run Magistral Small 1.2 (2509) on-prem
Mistral's 24B reasoning model built on Mistral Small 3.2 with SFT from Magistral Medium traces and RL. Version 1.2 adds a vision encoder and emits reasoning between [THINK] tags.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 48.0 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| FP8 | 24.0 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| INT4 (AWQ / GPTQ) | 13.4 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| GGUF Q4_K_M | 14.4 GB | 0.67 GB | 2.68 GB | 10.7 GB |
KV cache from the published config: 40 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Useful when ERP questions need multi-step reasoning, such as reconciling quantities or planning maintenance. Long reasoning traces hurt latency for interactive users. For most new projects Mistral Small 4 is the better fit.
Strengths
- Reasoning traces for math, planning and multi-step logic
- Vision input
- Fits a single GPU when quantized
- Broad multilingual support
Watch out for
- Quality may drop beyond 40K tokens
- Reasoning makes responses slower and longer
- Superseded by Mistral Small 4, which merges reasoning and instruct
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 1 | 27 / 28 GB | 41 |
| NVIDIA A100 40GB PCIe | 1 | 27 / 36 GB | 36 |
| NVIDIA L4 24GB | 2 | 28 / 43 GB | 13 |
| NVIDIA A10 24GB | 2 | 28 / 43 GB | 25 |
| NVIDIA L40S 48GB | 1 | 27 / 43 GB | 21 |
| NVIDIA RTX A6000 48GB | 1 | 27 / 43 GB | 18 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 27 / 43 GB | 23 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 28 / 43 GB | 41 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 54 / 57 GB | 46 |
| AMD Instinct MI210 64GB | 1 | 53 / 57 GB | 24 |
| NVIDIA A100 40GB PCIe | 2 | 54 / 72 GB | 40 |
| NVIDIA A100 80GB PCIe | 1 | 53 / 72 GB | 29 |
| NVIDIA A100 80GB SXM | 1 | 53 / 72 GB | 31 |
| NVIDIA H100 PCIe 80GB | 1 | 53 / 72 GB | 30 |
| NVIDIA H100 SXM 80GB | 1 | 53 / 72 GB | 49 |
| NVIDIA H100 NVL 94GB | 1 | 53 / 84 GB | 57 |
Serving and licence
vLLM with --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --reasoning-parser mistral --enable-auto-tool-choice; official GGUF for llama.cpp and Ollama.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Magistral Small 1.2 (2509) deployment
Preloaded with Magistral Small 1.2 (2509). Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Magistral Small 1.2 (2509) need?
The weights alone take about 48.0 GB in BF16, 24.0 GB in FP8 and 13.4 GB in 4-bit. On top of that each request holds a KV cache of about 0.67 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 13.4 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Magistral Small 1.2 (2509)?
For a single user at 8K context in FP8, 1 x NVIDIA GeForce RTX 5090 32GB (32 GB each) fits, at an estimated 41 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Magistral Small 1.2 (2509) allowed for commercial use?
Magistral Small 1.2 (2509) is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Magistral Small 1.2 (2509) on-prem?
vLLM with --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --reasoning-parser mistral --enable-auto-tool-choice; official GGUF for llama.cpp and Ollama.