On-prem models / Cohere
Run Command A (03-2025) on-prem
Cohere's 111B dense open-weights research release optimized for enterprise RAG, tool use and 23 languages. It supports grounded generation with citations.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K | KV per request, 128K |
|---|---|---|---|---|
| BF16 (full precision) | 222 GB | 1.34 GB | 2.95 GB | 9.40 GB |
| FP8 | 111 GB | 0.67 GB | 1.48 GB | 4.70 GB |
| INT4 (AWQ / GPTQ) | 62.2 GB | 0.67 GB | 1.48 GB | 4.70 GB |
| GGUF Q4_K_M | 66.6 GB | 0.67 GB | 1.48 GB | 4.70 GB |
KV cache from the published config: 64 layers, 8 KV heads x 128 dims, 48 sliding-window layers (4096 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Only suitable for internal evaluation or research under its non-commercial licence. For production ERP use, choose Command A+ (Apache 2.0) instead.
Strengths
- Strong RAG with citations and conversational tool use
- 23 languages
- Compact KV cache relative to its size thanks to sliding-window layers
Watch out for
- Non-commercial licence makes it unsuitable for production business use
- Dense 111B is expensive to serve
- Gated download
- Superseded by Command A+
GPUs that run it
One user, 8K context, 4-bit
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 2 | 66 / 72 GB | 25 |
| NVIDIA A100 80GB PCIe | 1 | 65 / 72 GB | 18 |
| NVIDIA A100 80GB SXM | 1 | 65 / 72 GB | 19 |
| NVIDIA H100 PCIe 80GB | 1 | 65 / 72 GB | 19 |
| NVIDIA H100 SXM 80GB | 1 | 65 / 72 GB | 31 |
| NVIDIA H100 NVL 94GB | 1 | 65 / 84 GB | 35 |
| NVIDIA L4 24GB | 4 | 69 / 86 GB | 9 |
| NVIDIA A10 24GB | 4 | 69 / 86 GB | 18 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 4 | 136 / 144 GB | 23 |
| NVIDIA A100 80GB PCIe | 2 | 133 / 144 GB | 16 |
| NVIDIA A100 80GB SXM | 2 | 133 / 144 GB | 17 |
| NVIDIA H100 PCIe 80GB | 2 | 133 / 144 GB | 17 |
| NVIDIA H100 SXM 80GB | 2 | 133 / 144 GB | 27 |
| NVIDIA B200 180GB | 1 | 132 / 162 GB | 36 |
| NVIDIA H100 NVL 94GB | 2 | 133 / 169 GB | 31 |
| NVIDIA L4 24GB | 8 | 142 / 172 GB | 9 |
Serving and licence
vLLM and llama.cpp support the Cohere2 architecture; GGUF builds from bartowski and unsloth.
CC-BY-NC-4.0. Non-commercial only under CC-BY-NC 4.0 plus Cohere's acceptable use policy; commercial use needs a licence from Cohere or the Apache-licensed Command A+.
Design a Command A (03-2025) deployment
Preloaded with Command A (03-2025). Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Command A (03-2025) need?
The weights alone take about 222 GB in BF16, 111 GB in FP8 and 62.2 GB in 4-bit. On top of that each request holds a KV cache of about 0.67 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 13.4 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Command A (03-2025)?
For a single user at 8K context in 4-bit, 2 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 25 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Command A (03-2025) allowed for commercial use?
Command A (03-2025) is released under CC-BY-NC-4.0. Non-commercial only under CC-BY-NC 4.0 plus Cohere's acceptable use policy; commercial use needs a licence from Cohere or the Apache-licensed Command A+. Check the licence text on the model card with your legal team before production use.
Which engine should serve Command A (03-2025) on-prem?
vLLM and llama.cpp support the Cohere2 architecture; GGUF builds from bartowski and unsloth.