GPUs for on-prem AI / NVIDIA
NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB for LLMs
300W blower variant for 4-GPU workstations (384GB total). NVIDIA lists 3,511 AI TOPS FP4 with sparsity; BF16/FP8 here are derived from it at the same ratios, so treat as approximate. Price is an indicative 2026 street range for one unit, not a quote.
workstation card | PCIe Gen5 only | CUDA | indicative price $8,000-10K
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
| Model | Size | 4-bit: cards | 4-bit: tok/s | FP8, 10 users: cards | FP8: tok/s per user |
|---|---|---|---|---|---|
| Mistral Medium 3.5 128B | 128B | 1 | 15 | 2 | 13 |
| Qwen3.5-122B-A10B | 125B / 10B | 1 | 117 | 2 | 99 |
| Ling-3.0-flash | 124B / 5.1B | 1 | 197 | 2 | 164 |
| NVIDIA Nemotron 3 Super 120B-A12B | 120B / 12B | 1 | 101 | 2 | 85 |
| Mistral Small 4 119B A6B (2603) | 119B / 6.5B | 1 | 165 | 2 | 136 |
| gpt-oss-120b | 117B / 5.1B | 1 | 194 | 2 | 147 |
| Command A (03-2025) | 111B | 1 | 17 | 2 | 16 |
| Llama 4 Scout 17B-16E Instruct | 109B / 17B | 1 | 72 | 2 | 51 |
| Qwen3-Coder-Next | 79.7B / 3B | 1 | 274 | 1 | 154 |
| Llama 3.3 70B Instruct | 70B | 1 | 26 | 2 | 22 |
| Qwen3.6-35B-A3B | 36B / 3B | 1 | 275 | 1 | 159 |
| Qwen3-32B | 32.8B | 1 | 52 | 1 | 24 |
| Olmo 3.1 32B Think | 32B | 1 | 54 | 1 | 27 |
| GLM-4.7-Flash | 31.2B / 3B | 1 | 268 | 1 | 124 |
| Gemma 4 31B IT | 30.7B | 1 | 55 | 1 | 25 |
| Qwen3-30B-A3B-Instruct-2507 | 30.5B / 3.3B | 1 | 247 | 1 | 93 |
| Granite 4.2 30B | 30B | 1 | 57 | 1 | 25 |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | 30B / 3B | 1 | 278 | 1 | 180 |
| Qwen3.8-27B | 27.8B | 1 | 62 | 1 | 33 |
| Gemma 3 27B IT | 27B | 1 | 63 | 1 | 31 |
| Gemma 4 26B A4B IT | 25.2B / 3.8B | 1 | 228 | 1 | 104 |
| Mistral Small 3.2 24B Instruct 2506 | 24B | 1 | 70 | 1 | 33 |
| Magistral Small 1.2 (2509) | 24B | 1 | 70 | 1 | 33 |
| Devstral Small 2 24B Instruct 2512 | 24B | 1 | 70 | 1 | 33 |
| gpt-oss-20b | 21B / 3.6B | 1 | 246 | 1 | 138 |
| Phi-4 | 14B | 1 | 109 | 1 | 44 |
| Gemma 4 12B IT | 11.95B | 1 | 125 | 1 | 60 |
| Qwen3.5-9B | 9.7B | 1 | 151 | 1 | 85 |
| Qwen3-VL-8B-Instruct | 8.8B | 1 | 157 | 1 | 65 |
| Granite 4.2 8B | 8B | 1 | 167 | 1 | 65 |
| Llama 3.1 8B Instruct | 8B | 1 | 169 | 1 | 71 |
| Hy3 | 295B / 21B | 2 | 86 | 4 | 57 |
| Command A+ (05-2026) | 218B / 25B | 2 | 84 | 4 | 76 |
| Step-3.7-Flash | 198B / 11B | 2 | 148 | 4 | 122 |
| Qwen3.8-Flash-Next | 180B / 6B | 2 | 208 | 4 | 167 |
| MiniMax-M3 | 428B / 23B | 4 | 115 | 8 (multi-node) | 74 |
| Qwen3.5-397B-A17B | 403B / 17B | 4 | 135 | 8 (multi-node) | 86 |
| Llama 4 Maverick 17B-128E Instruct | 400B / 17B | 4 | 142 | 8 (multi-node) | 86 |
| Trinity Large Thinking | 398B / 13B | 4 | 148 | 8 (multi-node) | 84 |
| GLM-5.3-Flash | 320B / 18B | 4 | 143 | 4 | 100 |
| MiMo-V2.6-Flash-RL | 309B / 15B | 4 | 154 | 4 | 107 |
| MiMo-V2.6-Pro-RL | 1020B / 42B | 8 (multi-node) | 73 | 16 (multi-node) | 63 |
| Kimi-K2.6 | 1000B / 32B | 8 (multi-node) | 85 | 16 (multi-node) | 74 |
| Kimi-K2.7-Code | 1000B / 32B | 8 (multi-node) | 85 | 16 (multi-node) | 74 |
| Muse Spark Open 816B-A42B | 816B / 42B | 8 (multi-node) | 77 | 16 (multi-node) | 66 |
| DeepSeek-V4.1-Flash-1017 | 748B / 16B | 8 (multi-node) | 131 | 16 (multi-node) | 116 |
| GLM-5.3 | 744B / 40B | 8 (multi-node) | 69 | 16 (multi-node) | 59 |
| Mistral Large 3 675B Instruct 2512 | 675B / 41B | 8 (multi-node) | 79 | 16 (multi-node) | 68 |
| DeepSeek-V3.2 | 671B / 37B | 8 (multi-node) | 82 | 8 (multi-node) | 63 |
| DeepSeek-V4-Pro-0813 | 1600B / 49B | 16 (multi-node) | 79 | 24 (multi-node) | 67 |
| Kimi-K3 | 2800B / 104B | 24 (multi-node) | 50 | 40 (multi-node) | 45 |
| Qwen3.8-2.4T-A95B | 2446B / 95B | 24 (multi-node) | 51 | 32 (multi-node) | 47 |
Design a deployment on NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB?
Of the models we track, Mistral Medium 3.5 128B (128B) fits on a single 96 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB for LLM inference?
Generation speed is mostly set by memory bandwidth: 1792 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 224 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB a good choice for on-prem AI?
300W blower variant for 4-GPU workstations (384GB total). NVIDIA lists 3,511 AI TOPS FP4 with sparsity; BF16/FP8 here are derived from it at the same ratios, so treat as approximate. Price is an indicative 2026 street range for one unit, not a quote.