On-prem models / Meta
Run Llama 4 Maverick 17B-128E Instruct on-prem
Meta's larger Llama 4 model with 128 routed experts, 17B active and 400B total parameters. It accepts images natively. Meta also publishes an official FP8 checkpoint for single-node serving.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 800 GB | 1.61 GB | 2.82 GB | 7.65 GB |
| FP8 | 400 GB | 0.81 GB | 1.41 GB | 3.83 GB |
| INT4 (AWQ / GPTQ) | 224 GB | 0.81 GB | 1.41 GB | 3.83 GB |
| GGUF Q4_K_M | 240 GB | 0.81 GB | 1.41 GB | 3.83 GB |
KV cache from the published config: 48 layers, 8 KV heads x 128 dims, 36 sliding-window layers (8192 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Capable for multilingual document understanding and general assistants, but the hardware footprint is hard to justify against newer models for ERP agents. Choose it only if already standardized on Llama.
Strengths
- Official FP8 checkpoint fits one 8x H100 node
- Native image input
- Low active parameters for decode speed
- Long context with compact chunked-attention KV cache
Watch out for
- 400B total weights need a full multi-GPU node
- Superseded on quality by newer open MoE models
- No reasoning mode
- Same mixed community reception as Scout
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI325X 256GB | 1 | 227 / 230 GB | 193 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 228 / 230 GB | 173 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 227 / 243 GB | 235 |
| NVIDIA H200 SXM 141GB | 2 | 228 / 253 GB | 199 |
| NVIDIA H200 NVL 141GB | 2 | 228 / 253 GB | 199 |
| AMD Instinct MI355X 288GB | 1 | 227 / 259 GB | 235 |
| NVIDIA A100 40GB PCIe | 8 | 237 / 288 GB | 151 |
| NVIDIA A100 80GB PCIe | 4 | 231 / 288 GB | 147 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 8 | 433 / 460 GB | 86 |
| AMD Instinct MI325X 256GB | 2 | 424 / 460 GB | 108 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 427 / 460 GB | 108 |
| Apple Mac Studio M3 Ultra 512GB | 1 | 422 / 460 GB | 11 |
| Apple Mac Studio M5 Ultra 512GB | 1 | 422 / 460 GB | 16 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 2 | 424 / 486 GB | 132 |
| NVIDIA H200 SXM 141GB | 4 | 427 / 507 GB | 126 |
| NVIDIA H200 NVL 141GB | 4 | 427 / 507 GB | 126 |
Serving and licence
vLLM, SGLang and TensorRT-LLM support it; use meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 on Hopper. llama.cpp supports GGUF quantizations.
Llama 4 Community License. Commercial use is allowed, but companies with over 700 million monthly active users need a separate licence from Meta, and attribution plus the Acceptable Use Policy apply.
Design a Llama 4 Maverick 17B-128E Instruct deployment
Preloaded with Llama 4 Maverick 17B-128E Instruct. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Llama 4 Maverick 17B-128E Instruct need?
The weights alone take about 800 GB in BF16, 400 GB in FP8 and 224 GB in 4-bit. On top of that each request holds a KV cache of about 0.81 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 16.1 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Llama 4 Maverick 17B-128E Instruct?
For a single user at 8K context in 4-bit, 1 x AMD Instinct MI325X 256GB (256 GB each) fits, at an estimated 193 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Llama 4 Maverick 17B-128E Instruct allowed for commercial use?
Llama 4 Maverick 17B-128E Instruct is released under Llama 4 Community License. Commercial use is allowed, but companies with over 700 million monthly active users need a separate licence from Meta, and attribution plus the Acceptable Use Policy apply. Check the licence text on the model card with your legal team before production use.
Which engine should serve Llama 4 Maverick 17B-128E Instruct on-prem?
vLLM, SGLang and TensorRT-LLM support it; use meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 on Hopper. llama.cpp supports GGUF quantizations.