On-prem models / NVIDIA
Run NVIDIA Nemotron 3.5 Lightning 30B-A3B on-prem
A hybrid Mamba-2, MoE and attention model with 3B active parameters, released August 11, 2026 with multi-token prediction layers. The BF16 release is the reference for fine-tuning; NVIDIA recommends the NVFP4 release for deployment.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 60.0 GB | 0.05 GB | 0.20 GB | 0.81 GB |
| FP8 | 30.0 GB | 0.03 GB | 0.10 GB | 0.40 GB |
| INT4 (AWQ / GPTQ) | 16.8 GB | 0.03 GB | 0.10 GB | 0.40 GB |
| GGUF Q4_K_M | 18.0 GB | 0.03 GB | 0.10 GB | 0.40 GB |
KV cache from the published config: 52 layers, 2 KV heads x 128 dims, 6 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Strong for high-volume ERP agents and RAG over long manuals or logs, with good tool calling at low cost per query. Language coverage misses many plant languages such as Chinese, Hindi or Polish.
Strengths
- Very high throughput and long context on modest GPUs
- Minimal KV cache from only 6 attention layers
- Built-in MTP and speculative decoding recipes
- Permissive licence
Watch out for
- Hybrid Mamba architecture needs specific engine flags and recent versions
- Supported languages limited to English, Spanish, French, German, Italian and Japanese
- Text only
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 32 / 36 GB | 172 |
| NVIDIA L4 24GB | 2 | 34 / 43 GB | 67 |
| NVIDIA A10 24GB | 2 | 34 / 43 GB | 114 |
| NVIDIA L40S 48GB | 1 | 32 / 43 GB | 108 |
| NVIDIA RTX A6000 48GB | 1 | 32 / 43 GB | 98 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 32 / 43 GB | 118 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 34 / 43 GB | 160 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 34 / 57 GB | 217 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 33 / 36 GB | 153 |
| NVIDIA L4 24GB | 2 | 35 / 43 GB | 59 |
| NVIDIA A10 24GB | 2 | 35 / 43 GB | 102 |
| NVIDIA L40S 48GB | 1 | 33 / 43 GB | 95 |
| NVIDIA RTX A6000 48GB | 1 | 33 / 43 GB | 86 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 33 / 43 GB | 104 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 35 / 43 GB | 146 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 35 / 57 GB | 201 |
Serving and licence
vLLM 0.27.1 (vllm/vllm-openai:v0.27.1) with --mamba-backend flashinfer --mamba-ssm-cache-dtype float16; SGLang supported; GGUF via llama.cpp. Use the NVFP4 checkpoint on Blackwell or DGX Spark.
OpenMDW-1.1. OpenMDW 1.1 is a permissive open model licence and NVIDIA states the model is ready for commercial use.
Design a NVIDIA Nemotron 3.5 Lightning 30B-A3B deployment
Preloaded with NVIDIA Nemotron 3.5 Lightning 30B-A3B. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does NVIDIA Nemotron 3.5 Lightning 30B-A3B need?
The weights alone take about 60.0 GB in BF16, 30.0 GB in FP8 and 16.8 GB in 4-bit. On top of that each request holds a KV cache of about 0.03 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 0.50 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs NVIDIA Nemotron 3.5 Lightning 30B-A3B?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 172 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is NVIDIA Nemotron 3.5 Lightning 30B-A3B allowed for commercial use?
NVIDIA Nemotron 3.5 Lightning 30B-A3B is released under OpenMDW-1.1. OpenMDW 1.1 is a permissive open model licence and NVIDIA states the model is ready for commercial use. Check the licence text on the model card with your legal team before production use.
Which engine should serve NVIDIA Nemotron 3.5 Lightning 30B-A3B on-prem?
vLLM 0.27.1 (vllm/vllm-openai:v0.27.1) with --mamba-backend flashinfer --mamba-ssm-cache-dtype float16; SGLang supported; GGUF via llama.cpp. Use the NVFP4 checkpoint on Blackwell or DGX Spark.