On-prem models / Mistral AI
Run Mistral Small 3.2 24B Instruct 2506 on-prem
A minor update to Mistral Small 3.1 that improves instruction following, function calling and reduces repetitive output. It is a 24B dense model with a Pixtral vision encoder. Mistral notes it needs about 55GB of GPU RAM at bf16.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 48.0 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| FP8 | 24.0 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| INT4 (AWQ / GPTQ) | 13.4 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| GGUF Q4_K_M | 14.4 GB | 0.67 GB | 2.68 GB | 10.7 GB |
KV cache from the published config: 40 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A dependable, licence-clean choice for European plants: good tool calling, document image extraction and multilingual output. Text-to-SQL is adequate for simple queries with schema context.
Strengths
- Reliable function calling via vLLM
- Image input for document and screenshot reading
- Runs on one 24GB to 32GB GPU when quantized
- Strong European language support
Watch out for
- No reasoning mode
- Superseded by Mistral Small 4 for new deployments
- Full-attention KV cache is larger than hybrid designs at long context
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 1 | 27 / 28 GB | 41 |
| NVIDIA A100 40GB PCIe | 1 | 27 / 36 GB | 36 |
| NVIDIA L4 24GB | 2 | 28 / 43 GB | 13 |
| NVIDIA A10 24GB | 2 | 28 / 43 GB | 25 |
| NVIDIA L40S 48GB | 1 | 27 / 43 GB | 21 |
| NVIDIA RTX A6000 48GB | 1 | 27 / 43 GB | 18 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 27 / 43 GB | 23 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 28 / 43 GB | 41 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 54 / 57 GB | 46 |
| AMD Instinct MI210 64GB | 1 | 53 / 57 GB | 24 |
| NVIDIA A100 40GB PCIe | 2 | 54 / 72 GB | 40 |
| NVIDIA A100 80GB PCIe | 1 | 53 / 72 GB | 29 |
| NVIDIA A100 80GB SXM | 1 | 53 / 72 GB | 31 |
| NVIDIA H100 PCIe 80GB | 1 | 53 / 72 GB | 30 |
| NVIDIA H100 SXM 80GB | 1 | 53 / 72 GB | 49 |
| NVIDIA H100 NVL 94GB | 1 | 53 / 84 GB | 57 |
Serving and licence
vLLM 0.9.1 or newer with mistral_common 1.6.2 or newer, using --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice. llama.cpp and Ollama also support it.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Mistral Small 3.2 24B Instruct 2506 deployment
Preloaded with Mistral Small 3.2 24B Instruct 2506. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Mistral Small 3.2 24B Instruct 2506 need?
The weights alone take about 48.0 GB in BF16, 24.0 GB in FP8 and 13.4 GB in 4-bit. On top of that each request holds a KV cache of about 0.67 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 13.4 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Mistral Small 3.2 24B Instruct 2506?
For a single user at 8K context in FP8, 1 x NVIDIA GeForce RTX 5090 32GB (32 GB each) fits, at an estimated 41 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Mistral Small 3.2 24B Instruct 2506 allowed for commercial use?
Mistral Small 3.2 24B Instruct 2506 is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Mistral Small 3.2 24B Instruct 2506 on-prem?
vLLM 0.9.1 or newer with mistral_common 1.6.2 or newer, using --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice. llama.cpp and Ollama also support it.