On-prem models / Microsoft
Run Phi-4 on-prem
Microsoft's 14B dense model trained heavily on synthetic textbook-style data plus filtered web and academic sources. It targets reasoning, math and coding in a small footprint, primarily in English.
Memory needed
| Precision | Weights | KV per request, 8K |
|---|---|---|
| BF16 (full precision) | 28.0 GB | 1.68 GB |
| FP8 | 14.0 GB | 0.84 GB |
| INT4 (AWQ / GPTQ) | 7.84 GB | 0.84 GB |
| GGUF Q4_K_M | 8.40 GB | 0.84 GB |
KV cache from the published config: 40 layers, 10 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Fits narrow English-language tasks such as classification, short extraction from text or fine-tuned SQL generation for a fixed schema. The 16K context and lack of tool calling limit agentic ERP use.
Strengths
- Strong math and logic for 14B
- MIT licence
- Runs on a single 16GB to 24GB GPU when quantized
- Small, simple dense architecture supported everywhere
Watch out for
- Only 16K context
- Multilingual data is about 8 percent of training, so non-English quality is limited
- No native tool-calling format in the base release
- Code training focused on common Python packages
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 17 / 21 GB | 12 |
| NVIDIA A10 24GB | 1 | 17 / 21 GB | 24 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 17 / 21 GB | 39 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 17 / 28 GB | 67 |
| NVIDIA A100 40GB PCIe | 1 | 17 / 36 GB | 59 |
| NVIDIA L40S 48GB | 1 | 17 / 43 GB | 34 |
| NVIDIA RTX A6000 48GB | 1 | 17 / 43 GB | 30 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 17 / 43 GB | 38 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 51 / 57 GB | 54 |
| AMD Instinct MI210 64GB | 1 | 50 / 57 GB | 29 |
| NVIDIA A100 40GB PCIe | 2 | 51 / 72 GB | 48 |
| NVIDIA A100 80GB PCIe | 1 | 50 / 72 GB | 35 |
| NVIDIA A100 80GB SXM | 1 | 50 / 72 GB | 37 |
| NVIDIA H100 PCIe 80GB | 1 | 50 / 72 GB | 36 |
| NVIDIA H100 SXM 80GB | 1 | 50 / 72 GB | 59 |
| NVIDIA H100 NVL 94GB | 1 | 50 / 84 GB | 67 |
Serving and licence
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (phi4) support it.
MIT. MIT permits commercial use, modification and redistribution with no restrictions beyond the notice.
Design a Phi-4 deployment
Preloaded with Phi-4. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Phi-4 need?
The weights alone take about 28.0 GB in BF16, 14.0 GB in FP8 and 7.84 GB in 4-bit. On top of that each request holds a KV cache of about 0.84 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 16.8 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Phi-4?
For a single user at 8K context in FP8, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 12 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Phi-4 allowed for commercial use?
Phi-4 is released under MIT. MIT permits commercial use, modification and redistribution with no restrictions beyond the notice. Check the licence text on the model card with your legal team before production use.
Which engine should serve Phi-4 on-prem?
vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama (phi4) support it.