GPUs for on-prem AI / AMD
AMD Instinct MI325X 256GB for LLMs
Same compute as MI300X with 256GB HBM3E and 6 TB/s, for larger models per node (2 TB per 8 GPUs). Same ROCm engine caveats as MI300X. Sold mainly in 8-GPU platforms; no unit price published.
OAM | Infinity Fabric 8 links x 128 GB/s | ROCm
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
Design a deployment on AMD Instinct MI325X 256GB
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one AMD Instinct MI325X 256GB?
Of the models we track, Qwen3.5-397B-A17B (403B total, 17B active) fits on a single 256 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is AMD Instinct MI325X 256GB for LLM inference?
Generation speed is mostly set by memory bandwidth: 6000 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 750 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is AMD Instinct MI325X 256GB a good choice for on-prem AI?
Same compute as MI300X with 256GB HBM3E and 6 TB/s, for larger models per node (2 TB per 8 GPUs). Same ROCm engine caveats as MI300X. Sold mainly in 8-GPU platforms; no unit price published.