On-prem models / Google
Run Gemma 4 26B A4B IT on-prem
A 128-expert MoE (8 active plus 1 shared) with 3.8B active parameters, launched with Gemma 4 on April 2, 2026. Google says it runs almost as fast as a 4B model. It supports thinking, function calling and image input.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 50.4 GB | 0.55 GB | 1.55 GB | 5.58 GB |
| FP8 | 25.2 GB | 0.27 GB | 0.78 GB | 2.79 GB |
| INT4 (AWQ / GPTQ) | 14.1 GB | 0.27 GB | 0.78 GB | 2.79 GB |
| GGUF Q4_K_M | 15.1 GB | 0.27 GB | 0.78 GB | 2.79 GB |
KV cache from the published config: 30 layers, 8 KV heads x 256 dims, 25 sliding-window layers (1024 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
One of the best value picks for on-prem ERP assistants: fast enough for many users on one GPU, strong at tool calling, and able to read scanned documents. Validate generated SQL before execution.
Strengths
- Near-4B decode speed with much higher quality (Google card: 82.6 percent MMLU Pro)
- Tiny KV cache allows many concurrent users
- Native function calling and thinking
- Apache 2.0 with official QAT int4 checkpoints
Watch out for
- All 25B weights must still be resident in memory
- Lower than 31B dense on long-context retrieval per Google's MRCR v2 numbers
- No audio input on this size
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 1 | 27 / 28 GB | 155 |
| NVIDIA A100 40GB PCIe | 1 | 27 / 36 GB | 139 |
| NVIDIA L4 24GB | 2 | 29 / 43 GB | 54 |
| NVIDIA A10 24GB | 2 | 29 / 43 GB | 97 |
| NVIDIA L40S 48GB | 1 | 27 / 43 GB | 85 |
| NVIDIA RTX A6000 48GB | 1 | 27 / 43 GB | 77 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 27 / 43 GB | 93 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 29 / 43 GB | 143 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 36 / 36 GB | 67 |
| NVIDIA L4 24GB | 2 | 38 / 43 GB | 25 |
| NVIDIA A10 24GB | 2 | 38 / 43 GB | 47 |
| NVIDIA L40S 48GB | 1 | 36 / 43 GB | 39 |
| NVIDIA RTX A6000 48GB | 1 | 36 / 43 GB | 35 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 36 / 43 GB | 43 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 38 / 43 GB | 74 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 38 / 57 GB | 117 |
Serving and licence
Supported in vLLM, SGLang and llama.cpp with Gemma 4 releases; NVIDIA publishes an NVFP4 checkpoint and Google publishes QAT q4_0 GGUF and W4A16.
Apache-2.0. Gemma 4 is released under Apache 2.0, so commercial use is unrestricted apart from standard notice requirements.
Design a Gemma 4 26B A4B IT deployment
Preloaded with Gemma 4 26B A4B IT. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Gemma 4 26B A4B IT need?
The weights alone take about 50.4 GB in BF16, 25.2 GB in FP8 and 14.1 GB in 4-bit. On top of that each request holds a KV cache of about 0.27 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 5.45 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Gemma 4 26B A4B IT?
For a single user at 8K context in FP8, 1 x NVIDIA GeForce RTX 5090 32GB (32 GB each) fits, at an estimated 155 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Gemma 4 26B A4B IT allowed for commercial use?
Gemma 4 26B A4B IT is released under Apache-2.0. Gemma 4 is released under Apache 2.0, so commercial use is unrestricted apart from standard notice requirements. Check the licence text on the model card with your legal team before production use.
Which engine should serve Gemma 4 26B A4B IT on-prem?
Supported in vLLM, SGLang and llama.cpp with Gemma 4 releases; NVIDIA publishes an NVFP4 checkpoint and Google publishes QAT q4_0 GGUF and W4A16.