On-prem models / OpenAI
Run gpt-oss-20b on-prem
The smaller gpt-oss model, a 32-expert MoE with 4 experts active per token. OpenAI states it runs within 16GB of memory at native MXFP4. It shares the harmony format, reasoning levels and tool support of the 120b model.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 42.0 GB | 0.20 GB | 0.81 GB | 3.22 GB |
| FP8 | 21.0 GB | 0.10 GB | 0.40 GB | 1.61 GB |
| INT4 (AWQ / GPTQ) | 11.8 GB | 0.10 GB | 0.40 GB | 1.61 GB |
| GGUF Q4_K_M | 12.6 GB | 0.10 GB | 0.40 GB | 1.61 GB |
KV cache from the published config: 24 layers, 8 KV heads x 64 dims, 12 sliding-window layers (128 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Good for edge or single-GPU plant deployments that need a capable tool-calling agent, such as looking up orders or stock through APIs. For complex text-to-SQL over large ERP schemas expect more errors than the 120b; use schema retrieval and query validation. Document extraction needs a separate OCR step.
Strengths
- Runs within 16GB of memory per OpenAI
- Fast decode with 3.6B active parameters
- Native function calling and configurable reasoning effort
- Fine-tunable on consumer hardware per the model card
Watch out for
- Harmony format required
- Text only
- Shallower knowledge than larger models; verify facts and SQL
- Reasoning at high effort adds latency and tokens
GPUs that run it
One user, 8K context, MXFP4
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 13 / 21 GB | 62 |
| NVIDIA A10 24GB | 1 | 13 / 21 GB | 114 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 13 / 21 GB | 172 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 13 / 28 GB | 254 |
| NVIDIA A100 40GB PCIe | 1 | 13 / 36 GB | 232 |
| NVIDIA L40S 48GB | 1 | 13 / 43 GB | 153 |
| NVIDIA RTX A6000 48GB | 1 | 13 / 43 GB | 139 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 13 / 43 GB | 165 |
20 people at once, 16K context, MXFP4
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L4 24GB | 1 | 17 / 21 GB | 32 |
| NVIDIA A10 24GB | 1 | 17 / 21 GB | 61 |
| NVIDIA GeForce RTX 4090 24GB | 1 | 17 / 21 GB | 97 |
| NVIDIA GeForce RTX 5090 32GB | 1 | 17 / 28 GB | 154 |
| NVIDIA A100 40GB PCIe | 1 | 17 / 36 GB | 138 |
| NVIDIA L40S 48GB | 1 | 17 / 43 GB | 85 |
| NVIDIA RTX A6000 48GB | 1 | 17 / 43 GB | 76 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 17 / 43 GB | 93 |
Serving and licence
vLLM, SGLang, TensorRT-LLM, llama.cpp, Ollama (gpt-oss:20b) and LM Studio support it. Fits a single 16GB to 24GB GPU.
Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue or user caps.
Design a gpt-oss-20b deployment
Preloaded with gpt-oss-20b. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does gpt-oss-20b need?
The weights alone take about 42.0 GB in BF16, 21.0 GB in FP8 and 11.8 GB in 4-bit. On top of that each request holds a KV cache of about 0.10 GB at 8K tokens of context (MXFP4 cache), so 20 people generating at once need roughly 2.04 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs gpt-oss-20b?
For a single user at 8K context in MXFP4, 1 x NVIDIA L4 24GB (24 GB each) fits, at an estimated 62 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is gpt-oss-20b allowed for commercial use?
gpt-oss-20b is released under Apache-2.0. Apache 2.0 permits commercial use, modification and redistribution with no revenue or user caps. Check the licence text on the model card with your legal team before production use.
Which engine should serve gpt-oss-20b on-prem?
vLLM, SGLang, TensorRT-LLM, llama.cpp, Ollama (gpt-oss:20b) and LM Studio support it. Fits a single 16GB to 24GB GPU.