On-prem models / Arcee AI
Run Trinity Large Thinking on-prem
A US-built 398B sparse MoE with 256 experts (4 active plus 1 shared) and about 13B active parameters, post-trained for extended reasoning and agentic RL. Reasoning traces must be kept in context across turns for tool calling to work reliably.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 796 GB | 1.26 GB | 2.77 GB | 8.81 GB |
| FP8 | 398 GB | 0.63 GB | 1.38 GB | 4.40 GB |
| INT4 (AWQ / GPTQ) | 223 GB | 0.63 GB | 1.38 GB | 4.40 GB |
| GGUF Q4_K_M | 239 GB | 0.63 GB | 1.38 GB | 4.40 GB |
KV cache from the published config: 60 layers, 8 KV heads x 128 dims, 45 sliding-window layers (4096 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A licence-clean, US-origin option for heavy agentic ERP workflows if hardware allows. Requires careful client handling of reasoning content; test tool-calling stability on your own ERP APIs.
Strengths
- Agentic-first post-training; card reports 94.7 percent on tau2-Bench
- Low active parameters for its size
- Permissive licence
- Official FP8, NVFP4, W4A16 and GGUF builds
Watch out for
- 398B total weights need a multi-GPU node
- Dropping reasoning content from history can cause malformed tool calls
- Smaller ecosystem and track record than major labs
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 4 | 230 / 230 GB | 141 |
| AMD Instinct MI325X 256GB | 1 | 226 / 230 GB | 228 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 227 / 230 GB | 189 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 226 / 243 GB | 273 |
| NVIDIA H200 SXM 141GB | 2 | 227 / 253 GB | 213 |
| NVIDIA H200 NVL 141GB | 2 | 227 / 253 GB | 213 |
| AMD Instinct MI355X 288GB | 1 | 226 / 259 GB | 273 |
| NVIDIA A100 40GB PCIe | 8 | 236 / 288 GB | 146 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI210 64GB | 8 | 428 / 460 GB | 94 |
| AMD Instinct MI325X 256GB | 2 | 419 / 460 GB | 125 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 4 | 422 / 460 GB | 118 |
| Apple Mac Studio M3 Ultra 512GB | 1 | 418 / 460 GB | 14 |
| Apple Mac Studio M5 Ultra 512GB | 1 | 418 / 460 GB | 20 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 2 | 419 / 486 GB | 150 |
| NVIDIA H200 SXM 141GB | 4 | 422 / 507 GB | 135 |
| NVIDIA H200 NVL 141GB | 4 | 422 / 507 GB | 135 |
Serving and licence
vLLM 0.11.1 or newer with reasoning and tool-call parsers (see card); FP8-Block, NVFP4 and GGUF checkpoints available.
OpenMDW-1.1. OpenMDW 1.1 is a permissive open model licence that allows commercial use.
Design a Trinity Large Thinking deployment
Preloaded with Trinity Large Thinking. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Trinity Large Thinking need?
The weights alone take about 796 GB in BF16, 398 GB in FP8 and 223 GB in 4-bit. On top of that each request holds a KV cache of about 0.63 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 12.6 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Trinity Large Thinking?
For a single user at 8K context in 4-bit, 4 x AMD Instinct MI210 64GB (64 GB each) fits, at an estimated 141 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Trinity Large Thinking allowed for commercial use?
Trinity Large Thinking is released under OpenMDW-1.1. OpenMDW 1.1 is a permissive open model licence that allows commercial use. Check the licence text on the model card with your legal team before production use.
Which engine should serve Trinity Large Thinking on-prem?
vLLM 0.11.1 or newer with reasoning and tool-call parsers (see card); FP8-Block, NVFP4 and GGUF checkpoints available.