On-prem models / Moonshot AI
Run Kimi-K2.6 on-prem
1T-parameter MoE (32B active) native multimodal agent model with the DeepSeek-V3-style MLA architecture and a 400M MoonViT encoder. Released with native INT4 weights from quantisation-aware training.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 2000 GB | 0.58 GB | 2.30 GB | 9.21 GB |
| FP8 | 1000 GB | 0.29 GB | 1.15 GB | 4.61 GB |
| INT4 (AWQ / GPTQ) | 560 GB | 0.29 GB | 1.15 GB | 4.61 GB |
| GGUF Q4_K_M | 600 GB | 0.29 GB | 1.15 GB | 4.61 GB |
KV cache from the published config: 61 layers, MLA latent 512 + 64 RoPE dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A capable on-prem agent brain for enterprises that can afford one 8-GPU server: multi-step ERP workflows, document understanding and SQL with tool feedback. Overkill for single-task extraction.
Strengths
- Strong long-horizon coding and agent-swarm orchestration
- Native INT4 (about 0.6 TB) runs on one 8x H200 or 8x B200 node
- Thinking and Instant modes
- Well supported by vLLM, SGLang and KTransformers
Watch out for
- Still needs a full 8-GPU server
- MLA cache on all 61 layers: long context costs more than hybrid models
- Attribution clause at very large scale
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 80GB PCIe | 8 | 573 / 576 GB | 119 |
| NVIDIA A100 80GB SXM | 8 | 573 / 576 GB | 121 |
| NVIDIA H100 PCIe 80GB | 8 | 573 / 576 GB | 121 |
| NVIDIA H100 SXM 80GB | 8 | 573 / 576 GB | 142 |
| NVIDIA B200 180GB | 4 | 567 / 648 GB | 185 |
| NVIDIA H100 NVL 94GB | 8 | 573 / 676 GB | 148 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB | 8 | 573 / 691 GB | 110 |
| AMD Instinct MI300X 192GB | 4 | 567 / 691 GB | 162 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI355X 288GB | 4 | 1018 / 1036 GB | 142 |
| NVIDIA B200 180GB | 8 | 1024 / 1296 GB | 147 |
| AMD Instinct MI300X 192GB | 8 | 1024 / 1382 GB | 131 |
| AMD Instinct MI325X 256GB | 8 | 1024 / 1843 GB | 136 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 8 | 1024 / 1944 GB | 147 |
Serving and licence
vLLM, SGLang and KTransformers listed by Moonshot; transformers >= 4.57.1. Instant mode via chat_template_kwargs thinking false. KTransformers enables CPU/GPU hybrid serving on smaller GPU counts.
Modified MIT. MIT terms plus one condition: products over 100M MAU or US$20M monthly revenue must display 'Kimi K2.6' in the UI.
Design a Kimi-K2.6 deployment
Preloaded with Kimi-K2.6. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Kimi-K2.6 need?
The weights alone take about 2000 GB in BF16, 1000 GB in FP8 and 560 GB in 4-bit. On top of that each request holds a KV cache of about 0.29 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 5.76 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Kimi-K2.6?
For a single user at 8K context in 4-bit, 8 x NVIDIA A100 80GB PCIe (80 GB each) fits, at an estimated 119 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Kimi-K2.6 allowed for commercial use?
Kimi-K2.6 is released under Modified MIT. MIT terms plus one condition: products over 100M MAU or US$20M monthly revenue must display 'Kimi K2.6' in the UI. Check the licence text on the model card with your legal team before production use.
Which engine should serve Kimi-K2.6 on-prem?
vLLM, SGLang and KTransformers listed by Moonshot; transformers >= 4.57.1. Instant mode via chat_template_kwargs thinking false. KTransformers enables CPU/GPU hybrid serving on smaller GPU counts.