GPUs for on-prem AI / AMD
AMD Instinct MI355X 288GB for LLMs
CDNA 4 part (launched June 2025) adding MXFP4/MXFP6 (10.1 PFLOPS MXFP4); 1400W, usually liquid cooled. AMD figures are dense (2x with sparsity). Sold in 8-GPU platforms; no unit price published.
OAM | Infinity Fabric 7 links x 153 GB/s | ROCm
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
Design a deployment on AMD Instinct MI355X 288GB
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one AMD Instinct MI355X 288GB?
Of the models we track, MiniMax-M3 (428B total, 23B active) fits on a single 288 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is AMD Instinct MI355X 288GB for LLM inference?
Generation speed is mostly set by memory bandwidth: 8000 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 1000 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is AMD Instinct MI355X 288GB a good choice for on-prem AI?
CDNA 4 part (launched June 2025) adding MXFP4/MXFP6 (10.1 PFLOPS MXFP4); 1400W, usually liquid cooled. AMD figures are dense (2x with sparsity). Sold in 8-GPU platforms; no unit price published.