On-prem models / IBM
Run Granite 4.2 8B on-prem
IBM's mid-size Granite 4.2 model, released August 2026 with native thinking. It supports full thinking, non-thinking and low-effort modes and is tested in 12 languages. IBM publishes FP8, MXFP4, NVFP4, GGUF and MLX variants.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 16.0 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| FP8 | 8.00 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| INT4 (AWQ / GPTQ) | 4.48 GB | 0.67 GB | 2.68 GB | 10.7 GB |
| GGUF Q4_K_M | 4.80 GB | 0.67 GB | 2.68 GB | 10.7 GB |
KV cache from the published config: 40 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A practical small model for on-prem ERP assistants that call APIs, summarize work orders or draft emails, with a clean licence and IBM backing. For complex text-to-SQL over large schemas, step up to the 30B.
Strengths
- Enterprise-oriented with reasoning-augmented tool calling
- Apache 2.0 with official quantized builds
- Runs on a single small GPU or CPU
- 12 tested languages including German, Japanese and Czech
Watch out for
- Smaller knowledge base than 30B-class models
- Thinking mode adds tokens
- Text only
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 11 / 21 GB | 21 |
| NVIDIA A10 24GB | 1 | 11 / 21 GB | 40 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 11 / 21 GB | 65 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 11 / 28 GB | 108 |
| NVIDIA A100 40GB PCIe | 1 | 11 / 36 GB | 96 |
| NVIDIA L40S 48GB | 1 | 11 / 43 GB | 57 |
| NVIDIA RTX A6000 48GB | 1 | 11 / 43 GB | 51 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 11 / 43 GB | 62 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 2 | 38 / 43 GB | 14 |
| NVIDIA A10 24GB | 2 | 38 / 43 GB | 28 |
| NVIDIA L40S 48GB | 1 | 37 / 43 GB | 23 |
| NVIDIA RTX A6000 48GB | 1 | 37 / 43 GB | 20 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 37 / 43 GB | 25 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 38 / 43 GB | 45 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 38 / 57 GB | 74 |
| AMD Instinct MI210 64GB | 1 | 37 / 57 GB | 41 |
Serving and licence
Standard GraniteForCausalLM architecture; vLLM with --enable-auto-tool-choice, SGLang, llama.cpp and Ollama via official GGUF. Pass enable_thinking via chat_template_kwargs.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps.
Design a Granite 4.2 8B deployment
Preloaded with Granite 4.2 8B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Granite 4.2 8B need?
The weights alone take about 16.0 GB in BF16, 8.00 GB in FP8 and 4.48 GB in 4-bit. On top of that each request holds a KV cache of about 0.67 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 13.4 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Granite 4.2 8B?
For a single user at 8K context in FP8, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 21 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Granite 4.2 8B allowed for commercial use?
Granite 4.2 8B is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve Granite 4.2 8B on-prem?
Standard GraniteForCausalLM architecture; vLLM with --enable-auto-tool-choice, SGLang, llama.cpp and Ollama via official GGUF. Pass enable_thinking via chat_template_kwargs.