GPUs for on-prem AI / AMD
AMD Ryzen AI Max+ 395 128GB system for LLMs
Mini PCs (Framework Desktop, HP Z2 Mini G1a and others) with 128GB shared memory, a configurable share of which goes to the Radeon 8060S iGPU; good for single-user 70B quantized models. Bandwidth derived from 256-bit LPDDR5x-8000; AMD publishes no GPU BF16 figure; 45-120W cTDP. Most users run llama.cpp (Vulkan or ROCm). Price is an indicative 2026 street range for one unit, not a quote.
desktop system | PCIe Gen4 only | ROCm | indicative price $2,000-3,000
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
| Model | Size | 4-bit: cards | 4-bit: tok/s | FP8, 10 users: cards | FP8: tok/s per user |
|---|---|---|---|---|---|
| Step-3.7-Flash | 198B / 11B | 1 | 18 | 2 (multi-node) | 14 |
| Qwen3.8-Flash-Next | 180B / 6B | 1 | 32 | 2 (multi-node) | 26 |
| Mistral Medium 3.5 128B | 128B | 1 | 2 | 2 (multi-node) | 2 |
| Qwen3.5-122B-A10B | 125B / 10B | 1 | 20 | 2 (multi-node) | 17 |
| Ling-3.0-flash | 124B / 5.1B | 1 | 38 | 2 (multi-node) | 33 |
| NVIDIA Nemotron 3 Super 120B-A12B | 120B / 12B | 1 | 17 | 2 (multi-node) | 15 |
| Mistral Small 4 119B A6B (2603) | 119B / 6.5B | 1 | 30 | 2 (multi-node) | 25 |
| gpt-oss-120b | 117B / 5.1B | 1 | 37 | 2 (multi-node) | 28 |
| Command A (03-2025) | 111B | 1 | 2 | 2 (multi-node) | 2 |
| Llama 4 Scout 17B-16E Instruct | 109B / 17B | 1 | 11 | 2 (multi-node) | 8 |
| Qwen3-Coder-Next | 79.7B / 3B | 1 | 60 | 1 | 27 |
| Llama 3.3 70B Instruct | 70B | 1 | 4 | 1 | 2 |
| Qwen3.6-35B-A3B | 36B / 3B | 1 | 61 | 1 | 29 |
| Qwen3-32B | 32.8B | 1 | 8 | 1 | 3 |
| Olmo 3.1 32B Think | 32B | 1 | 8 | 1 | 4 |
| GLM-4.7-Flash | 31.2B / 3B | 1 | 58 | 1 | 21 |
| Gemma 4 31B IT | 30.7B | 1 | 8 | 1 | 4 |
| Qwen3-30B-A3B-Instruct-2507 | 30.5B / 3.3B | 1 | 52 | 1 | 15 |
| Granite 4.2 30B | 30B | 1 | 9 | 1 | 4 |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | 30B / 3B | 1 | 62 | 1 | 34 |
| Qwen3.8-27B | 27.8B | 1 | 10 | 1 | 5 |
| Gemma 3 27B IT | 27B | 1 | 10 | 1 | 5 |
| Gemma 4 26B A4B IT | 25.2B / 3.8B | 1 | 46 | 1 | 17 |
| Mistral Small 3.2 24B Instruct 2506 | 24B | 1 | 11 | 1 | 5 |
| Magistral Small 1.2 (2509) | 24B | 1 | 11 | 1 | 5 |
| Devstral Small 2 24B Instruct 2512 | 24B | 1 | 11 | 1 | 5 |
| gpt-oss-20b | 21B / 3.6B | 1 | 51 | 1 | 24 |
| Phi-4 | 14B | 1 | 18 | 1 | 7 |
| Gemma 4 12B IT | 11.95B | 1 | 21 | 1 | 9 |
| Qwen3.5-9B | 9.7B | 1 | 27 | 1 | 14 |
| Qwen3-VL-8B-Instruct | 8.8B | 1 | 28 | 1 | 10 |
| Granite 4.2 8B | 8B | 1 | 30 | 1 | 10 |
| Llama 3.1 8B Instruct | 8B | 1 | 31 | 1 | 11 |
| Qwen3.5-397B-A17B | 403B / 17B | 2 (multi-node) | 19 | 4 (multi-node) | 19 |
| Llama 4 Maverick 17B-128E Instruct | 400B / 17B | 2 (multi-node) | 19 | 4 (multi-node) | 14 |
| Trinity Large Thinking | 398B / 13B | 2 (multi-node) | 23 | 4 (multi-node) | 18 |
| GLM-5.3-Flash | 320B / 18B | 2 (multi-node) | 19 | 3 (multi-node) | 15 |
| MiMo-V2.6-Flash-RL | 309B / 15B | 2 (multi-node) | 22 | 3 (multi-node) | 16 |
| Hy3 | 295B / 21B | 2 (multi-node) | 15 | 3 (multi-node) | 8 |
| Command A+ (05-2026) | 218B / 25B | 2 (multi-node) | 14 | 2 (multi-node) | 7 |
| MiniMax-M3 | 428B / 23B | 3 (multi-node) | 19 | 4 (multi-node) | 13 |
| DeepSeek-V4.1-Flash-1017 | 748B / 16B | 4 (multi-node) | 36 | 7 (multi-node) | 32 |
| GLM-5.3 | 744B / 40B | 4 (multi-node) | 15 | 7 (multi-node) | 13 |
| Mistral Large 3 675B Instruct 2512 | 675B / 41B | 4 (multi-node) | 15 | 6 (multi-node) | 11 |
| DeepSeek-V3.2 | 671B / 37B | 4 (multi-node) | 17 | 6 (multi-node) | 12 |
| Kimi-K2.6 | 1000B / 32B | 5 (multi-node) | 21 | 9 (multi-node) | 16 |
| Kimi-K2.7-Code | 1000B / 32B | 5 (multi-node) | 21 | 9 (multi-node) | 16 |
| Muse Spark Open 816B-A42B | 816B / 42B | 5 (multi-node) | 17 | 8 (multi-node) | 14 |
| MiMo-V2.6-Pro-RL | 1020B / 42B | 6 (multi-node) | 19 | 10 (multi-node) | 14 |
| DeepSeek-V4-Pro-0813 | 1600B / 49B | 8 (multi-node) | 22 | 15 (multi-node) | 17 |
| Qwen3.8-2.4T-A95B | 2446B / 95B | 13 (multi-node) | 15 | 22 (multi-node) | 13 |
| Kimi-K3 | 2800B / 104B | 14 (multi-node) | 14 | 25 (multi-node) | 14 |
Design a deployment on AMD Ryzen AI Max+ 395 128GB system
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one AMD Ryzen AI Max+ 395 128GB system?
Of the models we track, Step-3.7-Flash (198B total, 11B active) fits on a single 128 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is AMD Ryzen AI Max+ 395 128GB system for LLM inference?
Generation speed is mostly set by memory bandwidth: 256 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 32 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is AMD Ryzen AI Max+ 395 128GB system a good choice for on-prem AI?
Mini PCs (Framework Desktop, HP Z2 Mini G1a and others) with 128GB shared memory, a configurable share of which goes to the Radeon 8060S iGPU; good for single-user 70B quantized models. Bandwidth derived from 256-bit LPDDR5x-8000; AMD publishes no GPU BF16 figure; 45-120W cTDP. Most users run llama.cpp (Vulkan or ROCm). Price is an indicative 2026 street range for one unit, not a quote.