gpt-oss On-Prem Deployment: A Practical Enterprise Guide
gpt-oss is OpenAI's first open-weight model release, shipped in two mixture-of-experts sizes, 20B and 120B total parameters, under the Apache 2.0 license, a notable departure from OpenAI's usual API-only distribution model. Both sizes are reasoning-capable with an adjustable reasoning effort setting, letting the same deployment trade latency for answer quality on a per-request basis, similar in spirit to Qwen3's hybrid thinking mode but implemented as a graduated effort parameter rather than a binary toggle. Because it is Apache 2.0 and built by OpenAI, it draws particular interest from enterprises that have used OpenAI's hosted API and want an on-prem equivalent with comparable behavior. This guide covers the architecture and license, hardware sizing at each quantization, serving stack setup, fine-tuning options, regulated-industry considerations, and when Netray recommends gpt-oss over the other options in this guide series.
What gpt-oss Is: Architecture, Sizes, and License
Both gpt-oss sizes are sparse mixture-of-experts transformers with a reasoning effort parameter that adjusts how much internal deliberation the model performs before answering, exposed as a simple low/medium/high setting rather than a separate reasoning-mode checkpoint. The 20B model targets single-GPU and workstation-class deployment; the 120B model targets multi-GPU server deployment and is positioned closer to frontier-model quality on reasoning and coding benchmarks. Both ship under Apache 2.0, OpenAI's first fully open-source license release, with no usage threshold, no field-of-use restriction, and full commercial and fine-tuning rights, which is a materially different posture than the Llama 4 Community License or Mistral Research License.
- 20B: MoE architecture, single-GPU-friendly, targets workstation and single-server deployment
- 120B: MoE architecture, multi-GPU deployment, targets frontier-adjacent reasoning and coding quality
- Adjustable reasoning effort (low/medium/high) trades latency for answer quality per request
- Apache 2.0 license: no usage threshold, no field-of-use restriction, full commercial and fine-tuning rights
Hardware Requirements at Different Quantizations
The 20B model at FP16 needs roughly 40GB, fitting a single H100 80GB comfortably; at INT4/AWQ that drops to roughly 10-12GB, fitting an RTX 4090/5090 workstation card, which makes it genuinely deployable outside a dedicated server room. The 120B model at FP8 needs roughly 120GB, fitting on a single H200 or two H100 80GB in tensor parallel; at INT4 that drops to roughly 60GB, fitting a single H100 80GB with room for moderate concurrency. High reasoning effort settings increase output token count meaningfully, similar to a dedicated reasoning model, so budget KV cache and throughput headroom based on the reasoning effort level you expect production traffic to actually use, not just the base model size.
- 20B at INT4: approximately 10-12GB, single RTX 4090/5090 workstation card
- 20B at FP16: approximately 40GB, single H100 80GB
- 120B at FP8: approximately 120GB, single H200 or 2x H100 80GB
- 120B at INT4: approximately 60GB, single H100 80GB with moderate concurrency headroom
Serving Stack Setup
vLLM added gpt-oss support quickly given the model's high-profile release and handles the MoE architecture and reasoning-effort parameter natively. SGLang is a strong alternative for structured-output-heavy workloads. For the 20B model specifically, llama.cpp/Ollama support makes local workstation and edge deployment practical, which is a meaningful advantage for teams wanting to prototype against gpt-oss on a laptop before committing to server hardware. Because reasoning effort is a request-level parameter rather than a separate model, your serving layer needs to expose that control to calling applications rather than hardcoding a single reasoning mode at deployment time.
Fine-Tuning and Customization Options
Both gpt-oss sizes support LoRA and QLoRA fine-tuning through Axolotl, Unsloth, and Hugging Face TRL, with the 20B model being a particularly practical fine-tuning target given it fits on a single workstation GPU for both inference and, with enough memory headroom or gradient checkpointing, light fine-tuning work. As with other reasoning-capable models, DPO is useful for tuning reasoning verbosity and preventing the model from defaulting to high reasoning effort on tasks that do not need it, which is a real cost concern in production since higher reasoning effort directly increases token generation and GPU-hours per request.
Security and License Considerations for Regulated Industries
Apache 2.0 licensing removes the legal review bottleneck that Llama 4 and Mistral Large both introduce, making gpt-oss one of the more straightforward models in this guide series to clear through procurement for regulated-industry use. On-prem deployment resolves data residency entirely for clients who specifically wanted OpenAI-quality reasoning behavior without sending data to OpenAI's hosted API, which is a common motivation for evaluating gpt-oss among clients already familiar with OpenAI's commercial products. As with any reasoning-capable model, apply access controls and retention policy to logged reasoning traces, not just final outputs, since intermediate reasoning steps can surface sensitive information even when the final answer is appropriately redacted.
When Netray Recommends gpt-oss vs Alternatives
We recommend the 20B gpt-oss model when a client wants a genuinely workstation-deployable model with adjustable reasoning depth and values Apache 2.0's license simplicity, and the 120B model when the client needs stronger reasoning and coding quality with a moderate multi-GPU footprint. Against DeepSeek R1, gpt-oss offers a smaller footprint at comparable reasoning capability for many tasks, though R1's distilled family still wins for teams wanting the smallest possible footprint. Against Qwen3, the choice often comes down to specific benchmark performance on the client's own evaluation set rather than a general rule, which is exactly why we run a structured comparison before recommending either.
Frequently Asked Questions
What sizes does gpt-oss come in and what is the license?
gpt-oss ships in two mixture-of-experts sizes, 20B and 120B total parameters, both under the Apache 2.0 license. This gives full commercial use, fine-tuning, and redistribution rights with no usage threshold or field-of-use restriction, making it one of the more straightforward models in the current open-weight landscape to clear through legal and procurement review.
Can gpt-oss run on a single workstation GPU?
Yes, the 20B model needs roughly 10-12GB of GPU memory at INT4/AWQ quantization, which fits a single RTX 4090 or 5090 workstation card. This makes it practical to prototype and even run light production workloads on a workstation rather than a dedicated server, particularly for teams wanting to evaluate gpt-oss before committing to server-class hardware.
What does the reasoning effort setting in gpt-oss do?
Reasoning effort is a request-level parameter, typically set to low, medium, or high, that controls how much internal deliberation the model performs before producing its final answer. Higher settings improve accuracy on complex tasks but increase output token count and latency, so production deployments should tune the default effort level to match the actual complexity of incoming requests rather than always using the highest setting.
How does gpt-oss compare to DeepSeek R1 for on-prem deployment?
gpt-oss generally offers a smaller GPU footprint at the 20B and 120B sizes compared to R1's full 671B model, though R1's distilled variants (down to 32B and smaller) can match or beat gpt-oss on footprint depending on the size chosen. The right choice depends on your specific workload; benchmark both against your own tasks rather than relying on published leaderboard rankings, since the two models trade places depending on the task type.
Key Takeaways
- 1What gpt-oss Is: Architecture, Sizes, and License: Both gpt-oss sizes are sparse mixture-of-experts transformers with a reasoning effort parameter that adjusts how much internal deliberation the model performs before answering, exposed as a simple low/medium/high setting rather than a separate reasoning-mode checkpoint. The 20B model targets single-GPU and workstation-class deployment; the 120B model targets multi-GPU server deployment and is positioned closer to frontier-model quality on reasoning and coding benchmarks.
- 2Hardware Requirements at Different Quantizations: The 20B model at FP16 needs roughly 40GB, fitting a single H100 80GB comfortably; at INT4/AWQ that drops to roughly 10-12GB, fitting an RTX 4090/5090 workstation card, which makes it genuinely deployable outside a dedicated server room. The 120B model at FP8 needs roughly 120GB, fitting on a single H200 or two H100 80GB in tensor parallel; at INT4 that drops to roughly 60GB, fitting a single H100 80GB with room for moderate concurrency.
- 3Serving Stack Setup: vLLM added gpt-oss support quickly given the model's high-profile release and handles the MoE architecture and reasoning-effort parameter natively. SGLang is a strong alternative for structured-output-heavy workloads.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
gpt-oss Hardware Sizing Calculator
Size VRAM, GPU count, and capital cost for OpenAI's Apache 2.0 licensed gpt-oss-20b or gpt-oss-120b, both natively quantized to MXFP4 for single-GPU deployment.
Free ToolGLM-4.5 On-Prem Sizing Calculator
Size VRAM, GPU count, and capital cost for GLM-4.5 or the smaller GLM-4.5-Air, both mixture-of-experts models tuned for agentic and coding workloads.
Free ToolKimi K2 Deployment Cost Calculator
Estimate the multi-GPU cluster cost required to self-host Kimi K2, a roughly 1 trillion parameter mixture-of-experts model with only 32B active per token.
Terms used in this article
Considering gpt-oss as an on-prem alternative to a hosted API? Netray will benchmark it against your actual tasks and size the deployment before you commit hardware budget.
Related Resources
Gemma 3 Enterprise Deployment: An On-Prem Guide
Deploy Gemma 3 on-prem: 1B-27B sizes, multimodal support, Gemma license terms, GPU sizing, serving setup, fine-tuning, and when it fits enterprise workloads.
AI & AutomationDeepSeek R1 On-Prem Deployment: An Enterprise Guide
Deploy DeepSeek R1 on-prem: 671B MoE architecture, MIT license, distilled model sizing, GPU requirements, serving stack, and when reasoning models fit.
AI & AutomationReasoning Models in the Enterprise: When the Extra Cost Pays Off
When enterprise tasks justify reasoning models like R1 and QwQ: thinking budgets, latency and cost overhead, and how to decide against standard LLMs.