On-prem models / Google
Run Gemma 4 12B IT on-prem
The encoder-free Unified member of Gemma 4, projecting image patches and audio directly into the LLM. It has configurable thinking, native function calling and a system role. Google publishes QAT int4 variants.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 23.9 GB | 0.87 GB | 2.48 GB | 8.93 GB |
| FP8 | 11.9 GB | 0.44 GB | 1.24 GB | 4.46 GB |
| INT4 (AWQ / GPTQ) | 6.69 GB | 0.44 GB | 1.24 GB | 4.46 GB |
| GGUF Q4_K_M | 7.17 GB | 0.44 GB | 1.24 GB | 4.46 GB |
KV cache from the published config: 48 layers, 8 KV heads x 256 dims, 40 sliding-window layers (1024 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A good single-GPU choice for multilingual plants: reads photos of labels, delivery notes and forms, and can transcribe audio notes. Fine for simple tool-calling and lookups; use a larger model for complex text-to-SQL.
Strengths
- Text, image and audio input in about 12B parameters
- Very small long-context KV cache
- Native function calling and thinking mode
- Apache 2.0 with official QAT int4 GGUF and W4A16 checkpoints
Watch out for
- Encoder-free design is new, so engine support may lag the other Gemma 4 sizes
- Weaker than 26B A4B and 31B on reasoning and coding per Google's card
- Thinking mode adds latency
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 14 / 21 GB | 14 |
| NVIDIA A10 24GB | 1 | 14 / 21 GB | 28 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 14 / 21 GB | 46 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 14 / 28 GB | 78 |
| NVIDIA A100 40GB PCIe | 1 | 14 / 36 GB | 68 |
| NVIDIA L40S 48GB | 1 | 14 / 43 GB | 40 |
| NVIDIA RTX A6000 48GB | 1 | 14 / 43 GB | 36 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 14 / 43 GB | 44 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 1 | 28 / 28 GB | 48 |
| NVIDIA A100 40GB PCIe | 1 | 28 / 36 GB | 42 |
| NVIDIA L4 24GB | 2 | 30 / 43 GB | 15 |
| NVIDIA A10 24GB | 2 | 30 / 43 GB | 29 |
| NVIDIA L40S 48GB | 1 | 28 / 43 GB | 24 |
| NVIDIA RTX A6000 48GB | 1 | 28 / 43 GB | 21 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 28 / 43 GB | 27 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 30 / 43 GB | 46 |
Serving and licence
Needs recent Transformers (5.x) and engine versions with gemma4_unified support; check vLLM and llama.cpp release notes. Official q4_0 GGUF works with llama.cpp and Ollama.
Apache-2.0. Gemma 4 is released under Apache 2.0, so commercial use is unrestricted apart from standard notice requirements.
Design a Gemma 4 12B IT deployment
Preloaded with Gemma 4 12B IT. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Gemma 4 12B IT need?
The weights alone take about 23.9 GB in BF16, 11.9 GB in FP8 and 6.69 GB in 4-bit. On top of that each request holds a KV cache of about 0.44 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 8.72 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Gemma 4 12B IT?
For a single user at 8K context in FP8, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 14 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Gemma 4 12B IT allowed for commercial use?
Gemma 4 12B IT is released under Apache-2.0. Gemma 4 is released under Apache 2.0, so commercial use is unrestricted apart from standard notice requirements. Check the licence text on the model card with your legal team before production use.
Which engine should serve Gemma 4 12B IT on-prem?
Needs recent Transformers (5.x) and engine versions with gemma4_unified support; check vLLM and llama.cpp release notes. Official q4_0 GGUF works with llama.cpp and Ollama.