GPUs and systems for on-prem LLMs
Memory decides which models fit; bandwidth decides how fast they answer. Numbers from vendor datasheets, dense (not sparsity) TFLOPS. Open any card to see which models it runs, or size a full deployment.
Data center
| Hardware | Memory | Bandwidth | BF16 dense | FP8 | Power | Indicative price |
|---|---|---|---|---|---|---|
| NVIDIA L4 24GB | 24 GB GDDR6 | 300 GB/s | 121 TF | yes | 72 W | $2,000-3,000 |
| NVIDIA A10 24GB | 24 GB GDDR6 | 600 GB/s | 125 TF | no | 150 W | $1,500-3,000 |
| NVIDIA L40S 48GB | 48 GB GDDR6 | 864 GB/s | 362 TF | yes | 350 W | $7,500-10K |
| NVIDIA A100 40GB PCIe | 40 GB HBM2 | 1,555 GB/s | 312 TF | no | 250 W | $4,000-8,000 |
| NVIDIA A100 80GB PCIe | 80 GB HBM2e | 1,935 GB/s | 312 TF | no | 300 W | $8,000-15K |
| NVIDIA A100 80GB SXM | 80 GB HBM2e | 2,039 GB/s | 312 TF | no | 400 W | $8,000-15K |
| NVIDIA H100 PCIe 80GB | 80 GB HBM2e | 2,000 GB/s | 756 TF | yes | 350 W | $20K-28K |
| NVIDIA H100 SXM 80GB | 80 GB HBM3 | 3,350 GB/s | 989 TF | yes | 700 W | $22K-30K |
| NVIDIA H100 NVL 94GB | 94 GB HBM3 | 3,900 GB/s | 835 TF | yes | 400 W | $25K-32K |
| NVIDIA H200 SXM 141GB | 141 GB HBM3e | 4,800 GB/s | 989 TF | yes | 700 W | $30K-40K |
| NVIDIA H200 NVL 141GB | 141 GB HBM3e | 4,800 GB/s | 835 TF | yes | 600 W | $28K-38K |
| NVIDIA B200 180GB | 180 GB HBM3e | 8,000 GB/s | 2,250 TF | yes | 1000 W | $35K-50K |
| NVIDIA B300 (Blackwell Ultra) 270GB | 270 GB HBM3e | 8,000 GB/s | 2,250 TF | yes | 1100 W | n/a |
| NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB | 96 GB GDDR7 | 1,597 GB/s | 500 TF | yes | 600 W | $8,500-11K |
| AMD Instinct MI210 64GB | 64 GB HBM2e | 1,600 GB/s | 181 TF | no | 300 W | $4,000-8,000 |
| AMD Instinct MI300X 192GB | 192 GB HBM3 | 5,300 GB/s | 1,307 TF | yes | 750 W | $12K-20K |
| AMD Instinct MI325X 256GB | 256 GB HBM3E | 6,000 GB/s | 1,307 TF | yes | 1000 W | n/a |
| AMD Instinct MI355X 288GB | 288 GB HBM3E | 8,000 GB/s | 2,500 TF | yes | 1400 W | n/a |
| Intel Gaudi 3 128GB (HL-325L OAM) | 128 GB HBM2e | 3,700 GB/s | 1,678 TF | yes | 900 W | $12K-20K |
Workstation
| Hardware | Memory | Bandwidth | BF16 dense | FP8 | Power | Indicative price |
|---|---|---|---|---|---|---|
| NVIDIA RTX A6000 48GB | 48 GB GDDR6 | 768 GB/s | 155 TF | no | 300 W | $3,500-5,000 |
| NVIDIA RTX 6000 Ada 48GB | 48 GB GDDR6 | 960 GB/s | 364 TF | yes | 300 W | $6,500-8,000 |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96GB | 96 GB GDDR7 | 1,792 GB/s | 500 TF | yes | 600 W | $8,000-10K |
| NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB | 96 GB GDDR7 | 1,792 GB/s | 439 TF | yes | 300 W | $8,000-10K |
Consumer
| Hardware | Memory | Bandwidth | BF16 dense | FP8 | Power | Indicative price |
|---|---|---|---|---|---|---|
| NVIDIA GeForce RTX 4090 24GB | 24 GB GDDR6X | 1,008 GB/s | 165 TF | yes | 450 W | $1,600-2,500 |
| NVIDIA GeForce RTX 5090 32GB | 32 GB GDDR7 | 1,792 GB/s | 209 TF | yes | 575 W | $2,500-3,500 |
Small systems
| Hardware | Memory | Bandwidth | BF16 dense | FP8 | Power | Indicative price |
|---|---|---|---|---|---|---|
| NVIDIA DGX Spark (GB10) 128GB | 128 GB LPDDR5x unified | 273 GB/s | n/a | no | 240 W | $4,000-4,700 |
| Apple Mac Studio M3 Ultra 512GB | 512 GB unified memory | 819 GB/s | n/a | no | 480 W | n/a |
| Apple Mac Studio M5 Ultra 512GB | 512 GB unified memory | 1,200 GB/s | n/a | no | 480 W | n/a |
| AMD Ryzen AI Max+ 395 128GB system | 128 GB LPDDR5x-8000 unified | 256 GB/s | n/a | no | 120 W | $2,000-3,000 |