On-prem models / Tencent Hunyuan
Run Hy3 on-prem
Tencent's 295B MoE (21B active) text model released after the Hy3 Preview, with scaled-up RL focused on tool-call stability and multi-turn context tracking. Offers no_think, low and high reasoning effort.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 590 GB | 2.68 GB | 10.7 GB | 42.9 GB |
| FP8 | 295 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| INT4 (AWQ / GPTQ) | 165 GB | 1.34 GB | 5.37 GB | 21.5 GB |
| GGUF Q4_K_M | 177 GB | 1.34 GB | 5.37 GB | 21.5 GB |
KV cache from the published config: 80 layers, 8 KV heads x 128 dims. 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Its focus on reliable tool calls and format adherence suits ERP agents and structured JSON output, and Apache 2.0 is easy for legal. Needs a full 8-GPU server; use the FP8 repo.
Strengths
- Apache 2.0
- Card emphasises production-grade tool-call and output-format stability
- Defaults to direct answers (no_think), good for latency
- Official FP8 repo and fine-tuning/RL guides
Watch out for
- About 590 GB in BF16; 8 large-memory GPUs (H20-3e class) recommended
- Full GQA KV on 80 layers makes long context memory-heavy
- Text only
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA L40S 48GB | 4 | 173 / 172 GB | 71 |
| NVIDIA RTX A6000 48GB | 4 | 173 / 172 GB | 65 |
| NVIDIA RTX 6000 Ada 48GB | 4 | 173 / 172 GB | 76 |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96GB | 2 | 170 / 172 GB | 86 |
| NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB | 2 | 170 / 172 GB | 86 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB | 2 | 170 / 172 GB | 78 |
| AMD Instinct MI300X 192GB | 1 | 169 / 172 GB | 149 |
| AMD Instinct MI210 64GB | 4 | 173 / 230 GB | 102 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 8 | 361 / 460 GB | 59 |
| AMD Instinct MI325X 256GB | 2 | 352 / 460 GB | 76 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 355 / 460 GB | 75 |
| Apple Mac Studio M3 Ultra 512GB | 1 | 351 / 460 GB | 7 |
| Apple Mac Studio M5 Ultra 512GB | 1 | 351 / 460 GB | 11 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 2 | 352 / 486 GB | 94 |
| NVIDIA H200 SXM 141GB | 4 | 355 / 507 GB | 88 |
| NVIDIA H200 NVL 141GB | 4 | 355 / 507 GB | 88 |
Serving and licence
vLLM (build from source or recent release; --tool-call-parser hy_v3, MTP enabled) and SGLang (--tool-call-parser hunyuan, transformers>=5.6.0); recipes on vLLM and SGLang sites.
Apache-2.0. Apache 2.0 (per the repository LICENSE): commercial use, modification and redistribution allowed with notice.
Design a Hy3 deployment
Preloaded with Hy3. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Hy3 need?
The weights alone take about 590 GB in BF16, 295 GB in FP8 and 165 GB in 4-bit. On top of that each request holds a KV cache of about 1.34 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 26.8 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Hy3?
For a single user at 8K context in 4-bit, 4 x NVIDIA L40S 48GB (48 GB each) fits, at an estimated 71 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Hy3 allowed for commercial use?
Hy3 is released under Apache-2.0. Apache 2.0 (per the repository LICENSE): commercial use, modification and redistribution allowed with notice. Check the licence text on the model card with your legal team before production use.
Which engine should serve Hy3 on-prem?
vLLM (build from source or recent release; --tool-call-parser hy_v3, MTP enabled) and SGLang (--tool-call-parser hunyuan, transformers>=5.6.0); recipes on vLLM and SGLang sites.