GPUs for on-prem AI / NVIDIA
NVIDIA GeForce RTX 5090 32GB for LLMs
Fastest consumer card for local LLMs (1.79 TB/s, FP4 at 1,676 TFLOPS dense) but limited to 32GB at 575W, with no NVLink. Dense rates with FP32 accumulate per NVIDIA whitepaper (FP8 is 838 with FP16 accumulate). Price is an indicative 2026 street range for one unit, not a quote.
workstation card | PCIe Gen5 only | CUDA | indicative price $2,500-3,500
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
| Model | Size | 4-bit: cards | 4-bit: tok/s | FP8, 10 users: cards | FP8: tok/s per user |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 36B / 3B | 1 | 275 | 2 | 202 |
| Qwen3-32B | 32.8B | 1 | 52 | 2 | 39 |
| Olmo 3.1 32B Think | 32B | 1 | 54 | 2 | 44 |
| GLM-4.7-Flash | 31.2B / 3B | 1 | 268 | 2 | 164 |
| Gemma 4 31B IT | 30.7B | 1 | 55 | 2 | 41 |
| Qwen3-30B-A3B-Instruct-2507 | 30.5B / 3.3B | 1 | 247 | 2 | 132 |
| Granite 4.2 30B | 30B | 1 | 57 | 2 | 42 |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | 30B / 3B | 1 | 278 | 2 | 209 |
| Qwen3.8-27B | 27.8B | 1 | 62 | 2 | 54 |
| Gemma 3 27B IT | 27B | 1 | 63 | 2 | 51 |
| Gemma 4 26B A4B IT | 25.2B / 3.8B | 1 | 228 | 2 | 151 |
| Mistral Small 3.2 24B Instruct 2506 | 24B | 1 | 70 | 2 | 55 |
| Magistral Small 1.2 (2509) | 24B | 1 | 70 | 2 | 55 |
| Devstral Small 2 24B Instruct 2512 | 24B | 1 | 70 | 2 | 55 |
| gpt-oss-20b | 21B / 3.6B | 1 | 246 | 1 | 138 |
| Phi-4 | 14B | 1 | 109 | 2 | 71 |
| Gemma 4 12B IT | 11.95B | 1 | 125 | 1 | 60 |
| Qwen3.5-9B | 9.7B | 1 | 151 | 1 | 85 |
| Qwen3-VL-8B-Instruct | 8.8B | 1 | 157 | 1 | 65 |
| Granite 4.2 8B | 8B | 1 | 167 | 1 | 65 |
| Llama 3.1 8B Instruct | 8B | 1 | 169 | 1 | 71 |
| Qwen3-Coder-Next | 79.7B / 3B | 2 | 274 | 4 | 204 |
| Llama 3.3 70B Instruct | 70B | 2 | 42 | 4 | 36 |
| Qwen3.8-Flash-Next | 180B / 6B | 4 | 214 | 8 (multi-node) | 123 |
| Mistral Medium 3.5 128B | 128B | 4 | 40 | 8 (multi-node) | 31 |
| Qwen3.5-122B-A10B | 125B / 10B | 4 | 182 | 8 (multi-node) | 112 |
| Ling-3.0-flash | 124B / 5.1B | 4 | 237 | 8 (multi-node) | 139 |
| NVIDIA Nemotron 3 Super 120B-A12B | 120B / 12B | 4 | 134 | 8 (multi-node) | 74 |
| Mistral Small 4 119B A6B (2603) | 119B / 6.5B | 4 | 233 | 8 (multi-node) | 145 |
| gpt-oss-120b | 117B / 5.1B | 4 | 249 | 8 (multi-node) | 148 |
| Command A (03-2025) | 111B | 4 | 47 | 8 (multi-node) | 39 |
| Llama 4 Scout 17B-16E Instruct | 109B / 17B | 4 | 142 | 8 (multi-node) | 86 |
| GLM-5.3-Flash | 320B / 18B | 8 (multi-node) | 119 | 16 (multi-node) | 105 |
| MiMo-V2.6-Flash-RL | 309B / 15B | 8 (multi-node) | 119 | 16 (multi-node) | 103 |
| Hy3 | 295B / 21B | 8 (multi-node) | 79 | 16 (multi-node) | 62 |
| Command A+ (05-2026) | 218B / 25B | 8 (multi-node) | 128 | 16 (multi-node) | 111 |
| Step-3.7-Flash | 198B / 11B | 8 (multi-node) | 130 | 8 (multi-node) | 110 |
| DeepSeek-V4.1-Flash-1017 | 748B / 16B | 16 (multi-node) | 129 | 32 (multi-node) | 118 |
| GLM-5.3 | 744B / 40B | 16 (multi-node) | 70 | 32 (multi-node) | 63 |
| Mistral Large 3 675B Instruct 2512 | 675B / 41B | 16 (multi-node) | 82 | 32 (multi-node) | 75 |
| DeepSeek-V3.2 | 671B / 37B | 16 (multi-node) | 84 | 32 (multi-node) | 77 |
| MiniMax-M3 | 428B / 23B | 16 (multi-node) | 92 | 24 (multi-node) | 78 |
| Qwen3.5-397B-A17B | 403B / 17B | 16 (multi-node) | 96 | 16 (multi-node) | 87 |
| Llama 4 Maverick 17B-128E Instruct | 400B / 17B | 16 (multi-node) | 112 | 16 (multi-node) | 93 |
| Trinity Large Thinking | 398B / 13B | 16 (multi-node) | 98 | 16 (multi-node) | 86 |
| MiMo-V2.6-Pro-RL | 1020B / 42B | 24 (multi-node) | 72 | 40 (multi-node) | 64 |
| Kimi-K2.6 | 1000B / 32B | 24 (multi-node) | 83 | 40 (multi-node) | 74 |
| Kimi-K2.7-Code | 1000B / 32B | 24 (multi-node) | 83 | 40 (multi-node) | 74 |
| Muse Spark Open 816B-A42B | 816B / 42B | 24 (multi-node) | 79 | 32 (multi-node) | 74 |
| DeepSeek-V4-Pro-0813 | 1600B / 49B | 40 (multi-node) | 76 | 64 (multi-node) | 74 |
| Qwen3.8-2.4T-A95B | 2446B / 95B | 56 (multi-node) | 53 | - | - |
| Kimi-K3 | 2800B / 104B | 64 (multi-node) | 52 | - | - |
Design a deployment on NVIDIA GeForce RTX 5090 32GB
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one NVIDIA GeForce RTX 5090 32GB?
Of the models we track, Qwen3.6-35B-A3B (36B total, 3B active) fits on a single 32 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is NVIDIA GeForce RTX 5090 32GB for LLM inference?
Generation speed is mostly set by memory bandwidth: 1792 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 224 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is NVIDIA GeForce RTX 5090 32GB a good choice for on-prem AI?
Fastest consumer card for local LLMs (1.79 TB/s, FP4 at 1,676 TFLOPS dense) but limited to 32GB at 575W, with no NVLink. Dense rates with FP32 accumulate per NVIDIA whitepaper (FP8 is 838 with FP16 accumulate). Price is an indicative 2026 street range for one unit, not a quote.