On-prem models / Cohere
Run Command A+ (05-2026) on-prem
Cohere's sparse MoE with 128 experts (8 active plus a shared expert), 25B active and 218B total parameters, optimized for agentic, multilingual and reasoning-heavy enterprise work. It supports vision input, grounded citations and conversational tool use. Cohere ships BF16, FP8 and W4A4 checkpoints.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 436 GB | 0.67 GB | 1.48 GB | 4.70 GB |
| FP8 | 218 GB | 0.34 GB | 0.74 GB | 2.35 GB |
| INT4 (AWQ / GPTQ) | 122 GB | 0.34 GB | 0.74 GB | 2.35 GB |
| GGUF Q4_K_M | 131 GB | 0.34 GB | 0.74 GB | 2.35 GB |
KV cache from the published config: 32 layers, 8 KV heads x 128 dims, 24 sliding-window layers (4096 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
A strong choice for multilingual manufacturers needing RAG over ERP documentation with citations, plus tool-calling agents. Vision helps with scanned documents. Hardware needs are similar to other 200B-class MoE models.
Strengths
- Grounded answers with span-level citations to tool or document sources
- Very broad language coverage (over 40 languages listed)
- Enterprise RAG and tool-use focus
- Apache 2.0 with official FP8 and W4A4 builds
Watch out for
- 218B total weights need a multi-GPU node at BF16
- Response parsing needs Cohere's melody library and vLLM 0.21 or newer
- Smaller community ecosystem than Llama or Gemma
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA H200 SXM 141GB | 1 | 124 / 126 GB | 123 |
| NVIDIA H200 NVL 141GB | 1 | 124 / 126 GB | 123 |
| NVIDIA A100 40GB PCIe | 4 | 129 / 144 GB | 114 |
| NVIDIA A100 80GB PCIe | 2 | 126 / 144 GB | 89 |
| NVIDIA A100 80GB SXM | 2 | 126 / 144 GB | 93 |
| NVIDIA H100 PCIe 80GB | 2 | 126 / 144 GB | 91 |
| NVIDIA H100 SXM 80GB | 2 | 126 / 144 GB | 135 |
| NVIDIA B200 180GB | 1 | 124 / 162 GB | 183 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| AMD Instinct MI325X 256GB | 1 | 229 / 230 GB | 75 |
| Intel Gaudi 3 128GB (HL-325L OAM) | 2 | 231 / 230 GB | 78 |
| NVIDIA B300 (Blackwell Ultra) 270GB | 1 | 229 / 243 GB | 97 |
| NVIDIA H200 SXM 141GB | 2 | 231 / 253 GB | 97 |
| NVIDIA H200 NVL 141GB | 2 | 231 / 253 GB | 97 |
| AMD Instinct MI355X 288GB | 1 | 229 / 259 GB | 97 |
| NVIDIA A100 40GB PCIe | 8 | 240 / 288 GB | 95 |
| NVIDIA A100 80GB PCIe | 4 | 234 / 288 GB | 74 |
Serving and licence
vLLM 0.21.0 or newer with --tool-call-parser cohere_command4 --reasoning-parser cohere_command4 --enable-auto-tool-choice and cohere-melody installed; Cohere's example uses -tp 4 on B200.
Apache-2.0. Unlike earlier Command releases, Command A+ is Apache 2.0, so commercial use is permitted.
Design a Command A+ (05-2026) deployment
Preloaded with Command A+ (05-2026). Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Command A+ (05-2026) need?
The weights alone take about 436 GB in BF16, 218 GB in FP8 and 122 GB in 4-bit. On top of that each request holds a KV cache of about 0.34 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 6.71 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Command A+ (05-2026)?
For a single user at 8K context in 4-bit, 1 x NVIDIA H200 SXM 141GB (141 GB each) fits, at an estimated 123 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Command A+ (05-2026) allowed for commercial use?
Command A+ (05-2026) is released under Apache-2.0. Unlike earlier Command releases, Command A+ is Apache 2.0, so commercial use is permitted. Check the licence text on the model card with your legal team before production use.
Which engine should serve Command A+ (05-2026) on-prem?
vLLM 0.21.0 or newer with --tool-call-parser cohere_command4 --reasoning-parser cohere_command4 --enable-auto-tool-choice and cohere-melody installed; Cohere's example uses -tp 4 on B200.