On-prem models / Google
Run Gemma 4 31B IT on-prem
The dense flagship of Gemma 4, released April 2, 2026 under Apache 2.0. It adds configurable thinking, native function calling and a 256K context. Google's card reports 85.2 percent on MMLU Pro and 84.3 percent on GPQA Diamond.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 61.4 GB | 2.18 GB | 6.21 GB | 22.3 GB |
| FP8 | 30.7 GB | 1.09 GB | 3.10 GB | 11.2 GB |
| INT4 (AWQ / GPTQ) | 17.2 GB | 1.09 GB | 3.10 GB | 11.2 GB |
| GGUF Q4_K_M | 18.4 GB | 1.09 GB | 3.10 GB | 11.2 GB |
KV cache from the published config: 60 layers, 16 KV heads x 256 dims, 50 sliding-window layers (1024 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A strong all-rounder for ERP work: text-to-SQL with schema context, extraction from scanned invoices and drawings, tool-calling agents and multilingual plants. Prefer it over the MoE when accuracy matters more than throughput.
Strengths
- Top-tier quality for a single-GPU dense model
- Strong document parsing (best OmniDocBench 1.5 score in the Gemma 4 family)
- Over 140 pretraining languages
- Apache 2.0 with official QAT int4 variants
Watch out for
- Dense 31B decode is slower than the 26B A4B MoE
- Needs about 62GB at bf16, so quantize for 24GB to 48GB GPUs
- Thinking mode increases token use
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 34 / 36 GB | 28 |
| NVIDIA L4 24GB | 2 | 35 / 43 GB | 10 |
| NVIDIA A10 24GB | 2 | 35 / 43 GB | 20 |
| NVIDIA L40S 48GB | 1 | 34 / 43 GB | 16 |
| NVIDIA RTX A6000 48GB | 1 | 34 / 43 GB | 14 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 34 / 43 GB | 18 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 35 / 43 GB | 32 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 35 / 57 GB | 53 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 2 | 69 / 72 GB | 29 |
| NVIDIA A100 80GB PCIe | 1 | 68 / 72 GB | 21 |
| NVIDIA A100 80GB SXM | 1 | 68 / 72 GB | 22 |
| NVIDIA H100 PCIe 80GB | 1 | 68 / 72 GB | 22 |
| NVIDIA H100 SXM 80GB | 1 | 68 / 72 GB | 36 |
| NVIDIA H100 NVL 94GB | 1 | 68 / 84 GB | 41 |
| NVIDIA L4 24GB | 4 | 72 / 86 GB | 11 |
| NVIDIA A10 24GB | 4 | 72 / 86 GB | 21 |
Serving and licence
vLLM, SGLang, llama.cpp and Ollama support Gemma 4; NVIDIA publishes an NVFP4 checkpoint. Requires Transformers 5.5 or newer.
Apache-2.0. Gemma 4 is released under Apache 2.0, so commercial use is unrestricted apart from standard notice requirements.
Design a Gemma 4 31B IT deployment
Preloaded with Gemma 4 31B IT. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Gemma 4 31B IT need?
The weights alone take about 61.4 GB in BF16, 30.7 GB in FP8 and 17.2 GB in 4-bit. On top of that each request holds a KV cache of about 1.09 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 21.8 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Gemma 4 31B IT?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 28 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Gemma 4 31B IT allowed for commercial use?
Gemma 4 31B IT is released under Apache-2.0. Gemma 4 is released under Apache 2.0, so commercial use is unrestricted apart from standard notice requirements. Check the licence text on the model card with your legal team before production use.
Which engine should serve Gemma 4 31B IT on-prem?
vLLM, SGLang, llama.cpp and Ollama support Gemma 4; NVIDIA publishes an NVFP4 checkpoint. Requires Transformers 5.5 or newer.