On-prem models / NVIDIA
Run NVIDIA Nemotron 3 Super 120B-A12B on-prem
A LatentMoE hybrid of Mamba-2, 512-expert MoE and attention layers with multi-token prediction, released March 11, 2026. It was pretrained with NVFP4 on over 25T tokens and is aimed at collaborative agents and high-volume workloads such as IT ticket automation.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 240 GB | 0.07 GB | 0.27 GB | 1.07 GB |
| FP8 | 120 GB | 0.03 GB | 0.13 GB | 0.54 GB |
| INT4 (AWQ / GPTQ) | 67.2 GB | 0.03 GB | 0.13 GB | 0.54 GB |
| GGUF Q4_K_M | 72.0 GB | 0.03 GB | 0.13 GB | 0.54 GB |
KV cache from the published config: 88 layers, 2 KV heads x 128 dims, 8 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Well matched to on-prem ERP agent fleets, ticket triage and RAG over long ERP documentation, with high throughput per GPU. Check language coverage for non-European plants.
Strengths
- Tiny KV cache from 8 attention layers with 2 KV heads
- 1M-token context
- Built-in MTP head for faster generation
- NVFP4 checkpoint runs on a single B200 or DGX Spark
Watch out for
- Seven supported languages only
- Hybrid architecture needs specific engine versions and flags
- Text only
- NVIDIA licence is custom, not OSI open source
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 2 | 71 / 72 GB | 116 |
| NVIDIA A100 80GB PCIe | 1 | 69 / 72 GB | 108 |
| NVIDIA A100 80GB SXM | 1 | 69 / 72 GB | 113 |
| NVIDIA H100 PCIe 80GB | 1 | 69 / 72 GB | 111 |
| NVIDIA H100 SXM 80GB | 1 | 69 / 72 GB | 168 |
| NVIDIA H100 NVL 94GB | 1 | 69 / 84 GB | 187 |
| NVIDIA L4 24GB | 4 | 74 / 86 GB | 51 |
| NVIDIA A10 24GB | 4 | 74 / 86 GB | 81 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H200 SXM 141GB | 1 | 123 / 126 GB | 136 |
| NVIDIA H200 NVL 141GB | 1 | 123 / 126 GB | 136 |
| NVIDIA A100 40GB PCIe | 4 | 128 / 144 GB | 97 |
| NVIDIA A100 80GB PCIe | 2 | 125 / 144 GB | 88 |
| NVIDIA A100 80GB SXM | 2 | 125 / 144 GB | 91 |
| NVIDIA H100 PCIe 80GB | 2 | 125 / 144 GB | 90 |
| NVIDIA H100 SXM 80GB | 2 | 125 / 144 GB | 126 |
| NVIDIA B200 180GB | 1 | 123 / 162 GB | 199 |
Serving and licence
vLLM 0.18.1 per NVIDIA's card with --kv-cache-dtype fp8 --mamba-ssm-cache-dtype float32; SGLang and TensorRT-LLM supported; NVFP4 and FP8 checkpoints available.
NVIDIA Nemotron Open Model License. NVIDIA's open model licence allows commercial use and NVIDIA states the model is ready for commercial use; review its terms on attribution and redistribution.
Design a NVIDIA Nemotron 3 Super 120B-A12B deployment
Preloaded with NVIDIA Nemotron 3 Super 120B-A12B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does NVIDIA Nemotron 3 Super 120B-A12B need?
The weights alone take about 240 GB in BF16, 120 GB in FP8 and 67.2 GB in 4-bit. On top of that each request holds a KV cache of about 0.03 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 0.67 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs NVIDIA Nemotron 3 Super 120B-A12B?
For a single user at 8K context in 4-bit, 2 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 116 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is NVIDIA Nemotron 3 Super 120B-A12B allowed for commercial use?
NVIDIA Nemotron 3 Super 120B-A12B is released under NVIDIA Nemotron Open Model License. NVIDIA's open model licence allows commercial use and NVIDIA states the model is ready for commercial use; review its terms on attribution and redistribution. Check the licence text on the model card with your legal team before production use.
Which engine should serve NVIDIA Nemotron 3 Super 120B-A12B on-prem?
vLLM 0.18.1 per NVIDIA's card with --kv-cache-dtype fp8 --mamba-ssm-cache-dtype float32; SGLang and TensorRT-LLM supported; NVFP4 and FP8 checkpoints available.