On-prem models / Allen Institute for AI (Ai2)
Run Olmo 3.1 32B Think on-prem
A fully open 32B reasoning model from Ai2, with released training data (Dolma 3, Dolci), code, checkpoints and recipes. The Think variant produces long chain-of-thought for math and coding.
Memory needed
| Precision | Weights | KV per request, 8K | KV per request, 32K |
|---|---|---|---|
| BF16 (full precision) | 64.0 GB | 1.34 GB | 2.95 GB |
| FP8 | 32.0 GB | 0.67 GB | 1.48 GB |
| INT4 (AWQ / GPTQ) | 17.9 GB | 0.67 GB | 1.48 GB |
| GGUF Q4_K_M | 19.2 GB | 0.67 GB | 1.48 GB |
KV cache from the published config: 64 layers, 8 KV heads x 128 dims, 48 sliding-window layers (4096 tokens). 16-bit cache for BF16 rows, 8-bit otherwise.
Fit for ERP and manufacturing work
Best where data provenance and auditability matter, for example regulated manufacturers fine-tuning on their own data. As an ERP agent it is weaker than Gemma 4 or Mistral Small 4 because of limited tool-calling and language support.
Strengths
- Fully open data and training pipeline for auditability
- Apache 2.0
- Compact KV cache from 3:1 sliding-window attention
- Intermediate checkpoints for research and fine-tuning
Watch out for
- Model card does not document tool calling
- English-focused
- 64K context is shorter than most 2026 peers
- Long reasoning traces add latency
GPUs that run it
One user, 8K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA A100 40GB PCIe | 1 | 35 / 36 GB | 27 |
| NVIDIA L4 24GB | 2 | 36 / 43 GB | 10 |
| NVIDIA A10 24GB | 2 | 36 / 43 GB | 19 |
| NVIDIA L40S 48GB | 1 | 35 / 43 GB | 16 |
| NVIDIA RTX A6000 48GB | 1 | 35 / 43 GB | 14 |
| NVIDIA RTX 6000 Ada 48GB | 1 | 35 / 43 GB | 17 |
| NVIDIA GeForce RTX 4090 24GB | 2 | 36 / 43 GB | 31 |
| NVIDIA GeForce RTX 5090 32GB | 2 | 36 / 57 GB | 51 |
20 people at once, 16K context, FP8
| Hardware | Count | Memory used | Tok/s per user |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 32GB | 2 | 54 / 57 GB | 38 |
| AMD Instinct MI210 64GB | 1 | 53 / 57 GB | 20 |
| NVIDIA A100 40GB PCIe | 2 | 54 / 72 GB | 33 |
| NVIDIA A100 80GB PCIe | 1 | 53 / 72 GB | 25 |
| NVIDIA A100 80GB SXM | 1 | 53 / 72 GB | 26 |
| NVIDIA H100 PCIe 80GB | 1 | 53 / 72 GB | 25 |
| NVIDIA H100 SXM 80GB | 1 | 53 / 72 GB | 41 |
| NVIDIA H100 NVL 94GB | 1 | 53 / 84 GB | 48 |
Serving and licence
Transformers 4.57 or newer; vLLM and SGLang support Olmo3ForCausalLM; GGUF and MLX community builds exist.
Apache-2.0. Apache 2.0 permits commercial use; Ai2 states it is intended for research and educational use under its Responsible Use Guidelines.
Design a Olmo 3.1 32B Think deployment
Preloaded with Olmo 3.1 32B Think. Add your ERP and documents, change GPUs or users, and export the diagram.
Questions
How much VRAM does Olmo 3.1 32B Think need?
The weights alone take about 64.0 GB in BF16, 32.0 GB in FP8 and 17.9 GB in 4-bit. On top of that each request holds a KV cache of about 0.67 GB at 8K tokens of context (FP8 cache), so 20 people generating at once need roughly 13.4 GB more, plus a few GB of runtime overhead.
What is the smallest GPU setup that runs Olmo 3.1 32B Think?
For a single user at 8K context in FP8, 1 x NVIDIA A100 40GB PCIe (40 GB each) fits, at an estimated 27 tokens per second. For a team, size for concurrent requests and context; the builder above does that.
Is Olmo 3.1 32B Think allowed for commercial use?
Olmo 3.1 32B Think is released under Apache-2.0. Apache 2.0 permits commercial use; Ai2 states it is intended for research and educational use under its Responsible Use Guidelines. Check the licence text on the model card with your legal team before production use.
Which engine should serve Olmo 3.1 32B Think on-prem?
Transformers 4.57 or newer; vLLM and SGLang support Olmo3ForCausalLM; GGUF and MLX community builds exist.