On-prem models / IBM
Run Granite 4.2 30B on-prem
The flagship reasoning model of IBM's Granite 4.2 family, released August 25, 2026, built on Granite 4.1 30B Base. It offers full thinking, non-thinking and low-effort modes and reasoning-augmented tool calling.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 60.0 GB | 2.15 GB | 8.59 GB | 34.4 GB |
| FP8 | 30.0 GB | 1.07 GB | 4.29 GB | 17.2 GB |
| INT4 (AWQ / GPTQ) | 16.8 GB | 1.07 GB | 4.29 GB | 17.2 GB |
| GGUF Q4_K_M | 18.0 GB | 1.07 GB | 4.29 GB | 17.2 GB |
KV cache from the published config: 64 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Good fit for regulated manufacturers that want an IBM-backed, Apache-licensed model for tool-calling agents and text-to-SQL with reasoning. Document extraction needs a separate OCR or vision model such as Granite Docling.
Strengths
- Strongest Granite for reasoning, code and tool calling
- Apache 2.0 with IBM enterprise support options
- Official FP8, NVFP4, MXFP4 and GGUF builds
- 12 tested languages
Watch out for
- Dense 30B with full attention means a larger KV cache than hybrid models
- Text only
- Less community adoption than Gemma or Mistral
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 33 / 36 GB | 29 |
| NVIDIA L4 24GB | 2 | 35 / 43 GB | 10 |
| NVIDIA A10 24GB | 2 | 35 / 43 GB | 20 |
| NVIDIA L40S 48GB | 1 | 33 / 43 GB | 17 |
| NVIDIA RTX A6000 48GB | 1 | 33 / 43 GB | 15 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 33 / 43 GB | 18 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 35 / 43 GB | 32 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 35 / 57 GB | 54 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H100 NVL 94GB | 1 | 75 / 84 GB | 42 |
| NVIDIA L4 24GB | 4 | 79 / 86 GB | 11 |
| NVIDIA A10 24GB | 4 | 79 / 86 GB | 21 |
| NVIDIA L40S 48GB | 2 | 76 / 86 GB | 17 |
| NVIDIA RTX A6000 48GB | 2 | 76 / 86 GB | 15 |
| NVIDIA RTX 6000 Ada 48GB | 2 | 76 / 86 GB | 19 |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96GB | 1 | 75 / 86 GB | 20 |
| NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB | 1 | 75 / 86 GB | 20 |
Serving and licence
vLLM with --enable-auto-tool-choice and a 131072 max length, SGLang, llama.cpp and Ollama via official GGUF; MLX builds for Apple silicon.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Granite 4.2 30B deployment
Preloaded with Granite 4.2 30B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Granite 4.2 30B need?
The weights alone take about 60.0 GB in BF16, 30.0 GB in FP8 and 16.8 GB in 4-bit. On top of that each request holds a KV cache of about 1.07 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 21.5 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Granite 4.2 30B?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 29 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Granite 4.2 30B allowed for commercial use?
Granite 4.2 30B is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Granite 4.2 30B on-prem?
vLLM with --enable-auto-tool-choice and a 131072 max length, SGLang, llama.cpp and Ollama via official GGUF; MLX builds for Apple silicon.