On-prem models / Meta
Run Llama 3.3 70B Instruct on-prem
A text-only 70B dense model released in December 2024. It is the last dense flagship in the Llama 3 line and supports multiple tool-use formats and eight languages. It is still widely deployed on-prem as a general assistant.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 140 GB | 2.68 GB | 10.7 GB | 42.9 GB |
| FP8 | 70.0 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| INT4 (AWQ / GPTQ) | 39.2 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| GGUF Q4_K_M | 42.0 GB | 1.34 GB | 5.37 GB | 21.5 GB |
KV cache from the published config: 80 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Solid for document summarization, email drafting and moderate text-to-SQL with good schema prompts. For new deployments, MoE models such as gpt-oss-120b or Gemma 4 26B A4B usually give similar or better quality at much lower serving cost. Multilingual coverage is limited to eight languages.
Strengths
- Mature, predictable behaviour and broad tooling support
- Strong general instruction following for its generation
- Runs on 2x 80GB GPUs at FP8 or 1x 80GB at 4-bit
- Many enterprise fine-tunes available
Watch out for
- Dense 70B is slow and costly per token compared with modern MoE models of similar quality
- Large KV cache limits concurrent long-context users
- No reasoning mode or vision
- Outperformed by 2026 open models on coding and agentic benchmarks
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L40S 48GB | 1 | 43 / 43 GB | 13 |
| NVIDIA RTX A6000 48GB | 1 | 43 / 43 GB | 11 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 43 / 43 GB | 14 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 44 / 57 GB | 42 |
| AMD Instinct MI210 64GB | 1 | 43 / 57 GB | 23 |
| NVIDIA A100 40GB PCIe | 2 | 44 / 72 GB | 37 |
| NVIDIA A100 80GB PCIe | 1 | 43 / 72 GB | 28 |
| NVIDIA A100 80GB SXM | 1 | 43 / 72 GB | 29 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H200 SXM 141GB | 1 | 126 / 126 GB | 28 |
| NVIDIA H200 NVL 141GB | 1 | 126 / 126 GB | 28 |
| NVIDIA A100 40GB PCIe | 4 | 130 / 144 GB | 28 |
| NVIDIA A100 80GB PCIe | 2 | 127 / 144 GB | 20 |
| NVIDIA A100 80GB SXM | 2 | 127 / 144 GB | 21 |
| NVIDIA H100 PCIe 80GB | 2 | 127 / 144 GB | 21 |
| NVIDIA H100 SXM 80GB | 2 | 127 / 144 GB | 33 |
| NVIDIA B200 180GB | 1 | 126 / 162 GB | 46 |
Serving and licence
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (llama3.3:70b) support it. Common deployments use FP8 on Hopper or AWQ/GPTQ 4-bit.
Llama 3.3 Community License. Commercial use is allowed, but companies with over 700 million monthly active users need a separate licence from Meta, and attribution plus the Acceptable Use Policy apply.
Design a Llama 3.3 70B Instruct deployment
Preloaded with Llama 3.3 70B Instruct. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Llama 3.3 70B Instruct need?
The weights alone take about 140 GB in BF16, 70.0 GB in FP8 and 39.2 GB in 4-bit. On top of that each request holds a KV cache of about 1.34 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 26.8 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Llama 3.3 70B Instruct?
For a single user at 8K context in 4-bit, 1 x NVIDIA L40S 48GB (48 GB each) fits, at an estimated 13 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Llama 3.3 70B Instruct allowed for commercial use?
Llama 3.3 70B Instruct is released under Llama 3.3 Community License. Commercial use is allowed, but companies with over 700 million monthly active users need a separate licence from Meta, and attribution plus the Acceptable Use Policy apply. Check the licence text on the model card with your legal team before production use.
Which engine should serve Llama 3.3 70B Instruct on-prem?
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (llama3.3:70b) support it. Common deployments use FP8 on Hopper or AWQ/GPTQ 4-bit.