On-prem models / Meta
Run Llama 3.1 8B Instruct on-prem
Meta's 8B dense instruction model from the Llama 3.1 release. It supports 128K context, eight languages and built-in tool-calling prompt formats. It remains one of the most fine-tuned and widely deployed small models.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 16.0 GB | 1.07 GB | 4.29 GB | 17.2 GB |
| FP8 | 8.00 GB | 0.54 GB | 2.15 GB | 8.59 GB |
| INT4 (AWQ / GPTQ) | 4.48 GB | 0.54 GB | 2.15 GB | 8.59 GB |
| GGUF Q4_K_M | 4.80 GB | 0.54 GB | 2.15 GB | 8.59 GB |
KV cache from the published config: 32 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Best used as a fine-tuning base for narrow ERP tasks such as classifying tickets, normalizing part descriptions or routing requests. Zero-shot text-to-SQL over large ERP schemas is weak. Fine for simple tool calls with a small tool set.
Strengths
- Huge ecosystem of fine-tunes, quantizations and tutorials
- Runs on a single 16GB GPU at 8-bit or on CPU at 4-bit
- Supported by every major inference engine
- Built-in tool-use prompt format
Watch out for
- Clearly behind 2025 and 2026 small models on reasoning and coding
- No reasoning mode
- Gated repo on Hugging Face; licence is not OSI open source
- Full-attention KV cache grows fast at long context
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 11 / 21 GB | 21 |
| NVIDIA A10 24GB | 1 | 11 / 21 GB | 41 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 11 / 21 GB | 66 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 11 / 28 GB | 109 |
| NVIDIA A100 40GB PCIe | 1 | 11 / 36 GB | 96 |
| NVIDIA L40S 48GB | 1 | 11 / 43 GB | 57 |
| NVIDIA RTX A6000 48GB | 1 | 11 / 43 GB | 51 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 11 / 43 GB | 63 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 31 / 36 GB | 46 |
| NVIDIA L4 24GB | 2 | 33 / 43 GB | 16 |
| NVIDIA A10 24GB | 2 | 33 / 43 GB | 32 |
| NVIDIA L40S 48GB | 1 | 31 / 43 GB | 26 |
| NVIDIA RTX A6000 48GB | 1 | 31 / 43 GB | 23 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 31 / 43 GB | 29 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 33 / 43 GB | 51 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 33 / 57 GB | 83 |
Serving and licence
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (llama3.1:8b) all support it fully.
Llama 3.1 Community License. Commercial use is allowed, but companies with over 700 million monthly active users need a separate licence from Meta, and attribution plus the Acceptable Use Policy apply.
Design a Llama 3.1 8B Instruct deployment
Preloaded with Llama 3.1 8B Instruct. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Llama 3.1 8B Instruct need?
The weights alone take about 16.0 GB in BF16, 8.00 GB in FP8 and 4.48 GB in 4-bit. On top of that each request holds a KV cache of about 0.54 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 10.7 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Llama 3.1 8B Instruct?
For a single user at 8K context in FP8, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 21 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Llama 3.1 8B Instruct allowed for commercial use?
Llama 3.1 8B Instruct is released under Llama 3.1 Community License. Commercial use is allowed, but companies with over 700 million monthly active users need a separate licence from Meta, and attribution plus the Acceptable Use Policy apply. Check the licence text on the model card with your legal team before production use.
Which engine should serve Llama 3.1 8B Instruct on-prem?
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (llama3.1:8b) all support it fully.