On-prem models / Moonshot AI
Run Kimi-K3 on-prem
Moonshot's 2.8T-parameter open-weight flagship (104B active, 16 of 896 experts) built on Kimi Delta Attention and Attention Residuals, with native vision and 1M context. Trained with MXFP4 quantisation-aware training.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 5600 GB | 0.23 GB | 0.91 GB | 3.62 GB |
| FP8 | 2800 GB | 0.11 GB | 0.45 GB | 1.81 GB |
| INT4 (AWQ / GPTQ) | 1568 GB | 0.11 GB | 0.45 GB | 1.81 GB |
| GGUF Q4_K_M | 1680 GB | 0.11 GB | 0.45 GB | 1.81 GB |
KV cache from the published config: 93 layers, 96 KV heads x 192 dims, 24 full-attention layers. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Top-tier capability for complex agent workflows (planning, reconciliation, multi-system tool chains), but sized for hyperscale or large group IT. Typical manufacturers will get more value per dollar from Kimi K2.6-class or mid-size Qwen/GLM models.
Strengths
- Frontier-level open model for long-horizon coding and agentic knowledge work
- Hybrid KDA/MLA keeps 1M context affordable
- Native MXFP4 weights (about 1.4 TB) instead of BF16
- Native image and video understanding
Watch out for
- Still needs a large multi-GPU cluster (MXFP4 support best on Blackwell/MI350-class hardware)
- Thinking is always on; multi-turn requests must pass back reasoning_content
- Licence has MaaS and attribution clauses
- Very new architecture; engine support limited to recent vLLM/SGLang
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI325X 256GB | 8 | 1581 / 1843 GB | 96 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 8 | 1581 / 1944 GB | 104 |
| AMD Instinct MI355X 288GB | 8 | 1581 / 2073 GB | 104 |
20 people at once, 16K context, MXFP4
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI325X 256GB | 8 | 1501 / 1843 GB | 96 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 8 | 1501 / 1944 GB | 104 |
| AMD Instinct MI355X 288GB | 8 | 1501 / 2073 GB | 104 |
Serving and licence
vLLM recipe and SGLang cookbook (Kimi-K3). Use reasoning_effort low/high/max (default max). No llama.cpp path at this scale.
Kimi K3 License. Commercial use allowed; name display required above 100M MAU or US$20M monthly revenue, and Model-as-a-Service businesses above US$20M revenue need a separate agreement (internal use exempt).
Design a Kimi-K3 deployment
Preloaded with Kimi-K3. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Kimi-K3 need?
The weights alone take about 5600 GB in BF16, 2800 GB in FP8 and 1568 GB in 4-bit. On top of that each request holds a KV cache of about 0.11 GB at 8K tokens of context (MXFP4 cache), so 20 people generating at once need roughly 2.26 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Kimi-K3?
For a single user at 8K context in 4-bit, 8 x AMD Instinct MI325X 256GB (256 GB each) fits, at an estimated 96 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Kimi-K3 allowed for commercial use?
Kimi-K3 is released under Kimi K3 License. Commercial use allowed; name display required above 100M MAU or US$20M monthly revenue, and Model-as-a-Service businesses above US$20M revenue need a separate agreement (internal use exempt). Check the licence text on the model card with your legal team before production use.
Which engine should serve Kimi-K3 on-prem?
vLLM recipe and SGLang cookbook (Kimi-K3). Use reasoning_effort low/high/max (default max). No llama.cpp path at this scale.