On-prem models / OpenAI
Run gpt-oss-120b on-prem
OpenAI's larger open-weight MoE model with 128 experts (4 active per token). MoE weights were post-trained in MXFP4 so the model fits on a single 80GB GPU. It uses the harmony response format and offers low, medium and high reasoning effort.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 234 GB | 0.31 GB | 1.21 GB | 4.84 GB |
| FP8 | 117 GB | 0.15 GB | 0.61 GB | 2.42 GB |
| INT4 (AWQ / GPTQ) | 65.5 GB | 0.15 GB | 0.61 GB | 2.42 GB |
| GGUF Q4_K_M | 70.2 GB | 0.15 GB | 0.61 GB | 2.42 GB |
KV cache from the published config: 36 layers, 8 KV heads x 64 dims, 18 sliding-window layers (128 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A strong default for on-prem ERP agents: reliable function calling and structured outputs suit tool-calling agents and text-to-SQL when the schema is supplied in context. Document extraction works from OCR text only since there is no vision. Multilingual quality is reasonable but English-centric.
Strengths
- Fits a single 80GB H100 or MI300X at native MXFP4
- Configurable reasoning effort set in the system prompt
- Native function calling and Structured Outputs
- Very small KV cache thanks to 8 KV heads, 64-dim heads and half the layers being 128-token sliding window
- Apache 2.0 licence
Watch out for
- Must be prompted in the harmony format or output quality breaks
- Text only, no vision
- Low active parameter count can show in breadth of world knowledge versus dense 70B-class models
- Raw chain of thought is not intended to be shown to end users
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 2 | 69 / 72 GB | 220 |
| NVIDIA A100 80GB PCIe | 1 | 68 / 72 GB | 205 |
| NVIDIA A100 80GB SXM | 1 | 68 / 72 GB | 213 |
| NVIDIA H100 PCIe 80GB | 1 | 68 / 72 GB | 210 |
| NVIDIA H100 SXM 80GB | 1 | 68 / 72 GB | 290 |
| NVIDIA H100 NVL 94GB | 1 | 68 / 84 GB | 315 |
| NVIDIA L4 24GB | 4 | 72 / 86 GB | 107 |
| NVIDIA A10 24GB | 4 | 72 / 86 GB | 163 |
20 people at once, 16K context, MXFP4
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 2 | 72 / 72 GB | 145 |
| NVIDIA A100 80GB PCIe | 1 | 70 / 72 GB | 121 |
| NVIDIA A100 80GB SXM | 1 | 70 / 72 GB | 127 |
| NVIDIA H100 PCIe 80GB | 1 | 70 / 72 GB | 125 |
| NVIDIA H100 SXM 80GB | 1 | 70 / 72 GB | 186 |
| NVIDIA H100 NVL 94GB | 1 | 70 / 84 GB | 207 |
| NVIDIA L4 24GB | 4 | 75 / 86 GB | 64 |
| NVIDIA A10 24GB | 4 | 75 / 86 GB | 107 |
Serving and licence
vLLM (gpt-oss support from 0.10.1), SGLang, TensorRT-LLM, llama.cpp and Ollama (gpt-oss:120b) all support it. MXFP4 kernels are fastest on Hopper and Blackwell; older GPUs dequantize.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue or user caps.
Design a gpt-oss-120b deployment
Preloaded with gpt-oss-120b. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does gpt-oss-120b need?
The weights alone take about 234 GB in BF16, 117 GB in FP8 and 65.5 GB in 4-bit. On top of that each request holds a KV cache of about 0.15 GB at 8K tokens of context (MXFP4 cache), so 20 people generating at once need roughly 3.07 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs gpt-oss-120b?
For a single user at 8K context in 4-bit, 2 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 220 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is gpt-oss-120b allowed for commercial use?
gpt-oss-120b is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue or user caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve gpt-oss-120b on-prem?
vLLM (gpt-oss support from 0.10.1), SGLang, TensorRT-LLM, llama.cpp and Ollama (gpt-oss:120b) all support it. MXFP4 kernels are fastest on Hopper and Blackwell; older GPUs dequantize.