On-prem models / Mistral AI
Run Devstral Small 2 24B Instruct 2512 on-prem
Mistral's agentic coding model for software-engineering tasks, released with Devstral 2 in December 2025. It is a 24B dense model with vision and a 256K context, small enough for a single RTX 4090 or a 32GB Mac per Mistral.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 48.0 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| FP8 | 24.0 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| INT4 (AWQ / GPTQ) | 13.4 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| GGUF Q4_K_M | 14.4 GB | 0.67 GB | 2.68 GB | 10.7 GB |
KV cache from the published config: 40 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Well suited to ERP developer teams writing customizations, reports, SQL and integration code (for example SyteLine or Baan scripts) on-prem. Not intended as an end-user ERP assistant.
Strengths
- Tuned for codebase exploration and multi-file edits in agent scaffolds
- Strong tool use
- Runs locally on one consumer GPU when quantized
- 256K context for large repositories
Watch out for
- Coding specialist; general chat and knowledge are secondary
- No reasoning mode
- Large KV cache at full 256K context
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 1 | 27 / 28 GB | 41 |
| NVIDIA A100 40GB PCIe | 1 | 27 / 36 GB | 36 |
| NVIDIA L4 24GB | 2 | 28 / 43 GB | 13 |
| NVIDIA A10 24GB | 2 | 28 / 43 GB | 25 |
| NVIDIA L40S 48GB | 1 | 27 / 43 GB | 21 |
| NVIDIA RTX A6000 48GB | 1 | 27 / 43 GB | 18 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 27 / 43 GB | 23 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 28 / 43 GB | 41 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 54 / 57 GB | 46 |
| AMD Instinct MI210 64GB | 1 | 53 / 57 GB | 24 |
| NVIDIA A100 40GB PCIe | 2 | 54 / 72 GB | 40 |
| NVIDIA A100 80GB PCIe | 1 | 53 / 72 GB | 29 |
| NVIDIA A100 80GB SXM | 1 | 53 / 72 GB | 31 |
| NVIDIA H100 PCIe 80GB | 1 | 53 / 72 GB | 30 |
| NVIDIA H100 SXM 80GB | 1 | 53 / 72 GB | 49 |
| NVIDIA H100 NVL 94GB | 1 | 53 / 84 GB | 57 |
Serving and licence
vLLM with mistral_common 1.8.6 or newer and the mistral tool-call parser; GGUF via llama.cpp, Ollama and LM Studio. Works with Mistral Vibe CLI and other coding agents.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Devstral Small 2 24B Instruct 2512 deployment
Preloaded with Devstral Small 2 24B Instruct 2512. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Devstral Small 2 24B Instruct 2512 need?
The weights alone take about 48.0 GB in BF16, 24.0 GB in FP8 and 13.4 GB in 4-bit. On top of that each request holds a KV cache of about 0.67 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 13.4 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Devstral Small 2 24B Instruct 2512?
For a single user at 8K context in FP8, 1 x NVIDIA GeForce RTX 5090 32GB (32 GB each) fits, at an estimated 41 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Devstral Small 2 24B Instruct 2512 allowed for commercial use?
Devstral Small 2 24B Instruct 2512 is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Devstral Small 2 24B Instruct 2512 on-prem?
vLLM with mistral_common 1.8.6 or newer and the mistral tool-call parser; GGUF via llama.cpp, Ollama and LM Studio. Works with Mistral Vibe CLI and other coding agents.