On-prem models / Google
Run Gemma 3 27B IT on-prem
Google DeepMind's 27B dense multimodal model from March 2025, trained on over 140 languages. Google also publishes QAT int4 checkpoints for single consumer GPU use.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 54.0 GB | 1.11 GB | 3.12 GB | 11.2 GB |
| FP8 | 27.0 GB | 0.55 GB | 1.56 GB | 5.59 GB |
| INT4 (AWQ / GPTQ) | 15.1 GB | 0.55 GB | 1.56 GB | 5.59 GB |
| GGUF Q4_K_M | 16.2 GB | 0.55 GB | 1.56 GB | 5.59 GB |
KV cache from the published config: 62 layers, 16 KV heads x 128 dims, 52 sliding-window layers (1024 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Reasonable for multilingual plants and invoice or label image extraction. Agentic tool use and SQL generation are weaker than newer models. For new projects prefer Gemma 4 31B or 26B A4B.
Strengths
- Strong multilingual coverage
- Image input with a SigLIP encoder
- Compact KV cache from 5:1 local-global attention
- Official QAT quantized variants
Watch out for
- Superseded by Gemma 4, which is Apache 2.0 and adds reasoning
- No dedicated tool-call tokens; function calling is prompt-based
- Custom licence rather than OSI open source
- No reasoning mode
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 30 / 36 GB | 32 |
| NVIDIA L4 24GB | 2 | 31 / 43 GB | 11 |
| NVIDIA A10 24GB | 2 | 31 / 43 GB | 22 |
| NVIDIA L40S 48GB | 1 | 30 / 43 GB | 18 |
| NVIDIA RTX A6000 48GB | 1 | 30 / 43 GB | 16 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 30 / 43 GB | 20 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 31 / 43 GB | 36 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 31 / 57 GB | 59 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 48 / 57 GB | 44 |
| AMD Instinct MI210 64GB | 1 | 47 / 57 GB | 24 |
| NVIDIA A100 40GB PCIe | 2 | 48 / 72 GB | 39 |
| NVIDIA A100 80GB PCIe | 1 | 47 / 72 GB | 29 |
| NVIDIA A100 80GB SXM | 1 | 47 / 72 GB | 30 |
| NVIDIA H100 PCIe 80GB | 1 | 47 / 72 GB | 30 |
| NVIDIA H100 SXM 80GB | 1 | 47 / 72 GB | 49 |
| NVIDIA H100 NVL 94GB | 1 | 47 / 84 GB | 56 |
Serving and licence
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (gemma3:27b) support it, including the QAT GGUF.
Gemma Terms of Use. Commercial use is allowed under Google's Gemma Terms of Use, subject to the Gemma Prohibited Use Policy, which must pass through to derivatives.
Design a Gemma 3 27B IT deployment
Preloaded with Gemma 3 27B IT. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Gemma 3 27B IT need?
The weights alone take about 54.0 GB in BF16, 27.0 GB in FP8 and 15.1 GB in 4-bit. On top of that each request holds a KV cache of about 0.55 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 11.1 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Gemma 3 27B IT?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 32 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Gemma 3 27B IT allowed for commercial use?
Gemma 3 27B IT is released under Gemma Terms of Use. Commercial use is allowed under Google's Gemma Terms of Use, subject to the Gemma Prohibited Use Policy, which must pass through to derivatives. Check the licence text on the model card with your legal team before production use.
Which engine should serve Gemma 3 27B IT on-prem?
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (gemma3:27b) support it, including the QAT GGUF.