GPUs for on-prem AI / AMD
AMD Instinct MI300X 192GB for LLMs
192GB per GPU fits a 70B model at BF16 on one device; vLLM and SGLang support ROCm, but some CUDA-only engines and kernels are missing. AMD figures are dense (2x with sparsity). Price is an indicative 2026 street range for one unit, not a quote.
OAM | Infinity Fabric 8 links x 128 GB/s | ROCm | indicative price $12K-20K
Which models fit, and how many cards
Planning estimates. 4-bit column: one user at 8K context. FP8 column: 10 users at once at 16K context.
Design a deployment on AMD Instinct MI300X 192GB
Preloaded with this hardware. Add your ERP, documents and users.
Questions
What is the largest model that fits on one AMD Instinct MI300X 192GB?
Of the models we track, Hy3 (295B total, 21B active) fits on a single 192 GB card in 4-bit with one user at 8K context. More users or longer context need memory for the KV cache, so leave headroom.
How fast is AMD Instinct MI300X 192GB for LLM inference?
Generation speed is mostly set by memory bandwidth: 5300 GB/s here. Each token reads the active weights once, so a model with 8 GB of active weights tops out near 663 tokens per second per stream before efficiency losses; batching many users raises total throughput.
Is AMD Instinct MI300X 192GB a good choice for on-prem AI?
192GB per GPU fits a 70B model at BF16 on one device; vLLM and SGLang support ROCm, but some CUDA-only engines and kernels are missing. AMD figures are dense (2x with sparsity). Price is an indicative 2026 street range for one unit, not a quote.