GPUs for on-prem AI / Intel
Intel Gaudi 3 128GB (HL-325L OAM) for LLMs
128GB per accelerator with integrated Ethernet scale-out; serving relies on Intel's vLLM fork and Optimum Habana, so model and feature coverage is narrower than CUDA. MME BF16/FP8 per Intel technical paper; the PCIe HL-338 card is 600W. Price is an indicative 2026 street range for one unit, not a quote.
OAM | 24x 200GbE RoCE on-chip (1200 GB/s bidirectional) | Gaudi SynapseAI | indicative price $12K-20K
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
| Model | Size | 4-bit: cards | 4-bit: tok/s | FP8, 10 users: cards | FP8: tok/s per user |
|---|---|---|---|---|---|
| Step-3.7-Flash | 198B / 11B | 1 | 189 | 2 | 145 |
| Qwen3.8-Flash-Next | 180B / 6B | 1 | 282 | 2 | 208 |
| Mistral Medium 3.5 128B | 128B | 1 | 29 | 2 | 26 |
| Qwen3.5-122B-A10B | 125B / 10B | 1 | 204 | 2 | 163 |
| Ling-3.0-flash | 124B / 5.1B | 1 | 310 | 2 | 241 |
| NVIDIA Nemotron 3 Super 120B-A12B | 120B / 12B | 1 | 180 | 2 | 136 |
| Mistral Small 4 119B A6B (2603) | 119B / 6.5B | 1 | 269 | 2 | 212 |
| gpt-oss-120b | 117B / 5.1B | 1 | 306 | 2 | 225 |
| Command A (03-2025) | 111B | 1 | 34 | 2 | 31 |
| Llama 4 Scout 17B-16E Instruct | 109B / 17B | 1 | 134 | 2 | 92 |
| Qwen3-Coder-Next | 79.7B / 3B | 1 | 394 | 1 | 255 |
| Llama 3.3 70B Instruct | 70B | 1 | 51 | 1 | 25 |
| Qwen3.6-35B-A3B | 36B / 3B | 1 | 394 | 1 | 262 |
| Qwen3-32B | 32.8B | 1 | 100 | 1 | 47 |
| Olmo 3.1 32B Think | 32B | 1 | 102 | 1 | 53 |
| GLM-4.7-Flash | 31.2B / 3B | 1 | 388 | 1 | 214 |
| Gemma 4 31B IT | 30.7B | 1 | 104 | 1 | 49 |
| Qwen3-30B-A3B-Instruct-2507 | 30.5B / 3.3B | 1 | 365 | 1 | 167 |
| Granite 4.2 30B | 30B | 1 | 107 | 1 | 50 |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | 30B / 3B | 1 | 397 | 1 | 289 |
| Qwen3.8-27B | 27.8B | 1 | 117 | 1 | 65 |
| Gemma 3 27B IT | 27B | 1 | 118 | 1 | 62 |
| Gemma 4 26B A4B IT | 25.2B / 3.8B | 1 | 345 | 1 | 183 |
| Mistral Small 3.2 24B Instruct 2506 | 24B | 1 | 130 | 1 | 65 |
| Magistral Small 1.2 (2509) | 24B | 1 | 130 | 1 | 65 |
| Devstral Small 2 24B Instruct 2512 | 24B | 1 | 130 | 1 | 65 |
| gpt-oss-20b | 21B / 3.6B | 1 | 364 | 1 | 233 |
| Phi-4 | 14B | 1 | 191 | 1 | 85 |
| Gemma 4 12B IT | 11.95B | 1 | 215 | 1 | 113 |
| Qwen3.5-9B | 9.7B | 1 | 251 | 1 | 154 |
| Qwen3-VL-8B-Instruct | 8.8B | 1 | 259 | 1 | 121 |
| Granite 4.2 8B | 8B | 1 | 272 | 1 | 122 |
| Llama 3.1 8B Instruct | 8B | 1 | 274 | 1 | 132 |
| Qwen3.5-397B-A17B | 403B / 17B | 2 | 170 | 4 | 140 |
| Llama 4 Maverick 17B-128E Instruct | 400B / 17B | 2 | 173 | 4 | 127 |
| Trinity Large Thinking | 398B / 13B | 2 | 189 | 4 | 137 |
| GLM-5.3-Flash | 320B / 18B | 2 | 173 | 4 | 152 |
| MiMo-V2.6-Flash-RL | 309B / 15B | 2 | 189 | 4 | 159 |
| Hy3 | 295B / 21B | 2 | 138 | 4 | 92 |
| Command A+ (05-2026) | 218B / 25B | 2 | 145 | 2 | 86 |
| DeepSeek-V4.1-Flash-1017 | 748B / 16B | 4 | 213 | 8 | 184 |
| GLM-5.3 | 744B / 40B | 4 | 116 | 8 | 97 |
| Mistral Large 3 675B Instruct 2512 | 675B / 41B | 4 | 125 | 8 | 108 |
| DeepSeek-V3.2 | 671B / 37B | 4 | 131 | 8 | 112 |
| MiniMax-M3 | 428B / 23B | 4 | 161 | 4 | 113 |
| DeepSeek-V4-Pro-0813 | 1600B / 49B | 8 | 129 | 16 (multi-node) | 80 |
| MiMo-V2.6-Pro-RL | 1020B / 42B | 8 | 126 | 16 (multi-node) | 76 |
| Kimi-K2.6 | 1000B / 32B | 8 | 146 | 16 (multi-node) | 87 |
| Kimi-K2.7-Code | 1000B / 32B | 8 | 146 | 16 (multi-node) | 87 |
| Muse Spark Open 816B-A42B | 816B / 42B | 8 | 134 | 8 | 104 |
| Kimi-K3 | 2800B / 104B | 16 (multi-node) | 59 | 32 (multi-node) | 54 |
| Qwen3.8-2.4T-A95B | 2446B / 95B | 16 (multi-node) | 60 | 24 (multi-node) | 52 |
Design a deployment on Intel Gaudi 3 128GB (HL-325L OAM)
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one Intel Gaudi 3 128GB (HL-325L OAM)?
Of the models we track, Step-3.7-Flash (198B total, 11B active) fits on a single 128 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is Intel Gaudi 3 128GB (HL-325L OAM) for LLM inference?
Generation speed is mostly set by memory bandwidth: 3700 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 463 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is Intel Gaudi 3 128GB (HL-325L OAM) a good choice for on-prem AI?
128GB per accelerator with integrated Ethernet scale-out; serving relies on Intel's vLLM fork and Optimum Habana, so model and feature coverage is narrower than CUDA. MME BF16/FP8 per Intel technical paper; the PCIe HL-338 card is 600W. Price is an indicative 2026 street range for one unit, not a quote.