Mistral Large On-Prem Deployment: A Practical Enterprise Guide
Mistral Large is Mistral AI's flagship dense model at 123B parameters, positioned as a European alternative to the large MoE models dominating the open-weight landscape, with particular strength in multilingual tasks and function calling. It sits under the Mistral Research License, which is more restrictive than Apache 2.0 or MIT: free for research and testing, but commercial production use requires a separate commercial license from Mistral, a detail that catches teams off guard when they move from pilot to production. Mistral also ships Mistral Small and Devstral under Apache 2.0, which matters when license simplicity outweighs the flagship's extra capability. This guide covers the architecture and license terms, hardware sizing by quantization, serving stack setup, fine-tuning options, regulated-industry considerations, and when Netray recommends Mistral Large over an MoE alternative.
What Mistral Large Is: Architecture, Sizes, and License
Mistral Large 2, at 123B parameters, is a dense transformer rather than a mixture-of-experts model, which means every parameter activates on every token: compute cost scales directly with the full parameter count, unlike the sparse MoE models where active parameters are a fraction of total. This makes Mistral Large computationally heavier per token than an MoE model of similar total size, but it also makes memory and compute sizing far more predictable, since there is no expert-routing variability to account for. The license split matters: Mistral Large itself is under the Mistral Research License, requiring a commercial agreement for production use, while Mistral Small and the coding-focused Devstral model are Apache 2.0 and free for unrestricted commercial deployment.
- Mistral Large 2: 123B dense parameters, strong multilingual and function-calling performance
- Dense architecture means full parameter count activates per token, unlike sparse MoE models
- Mistral Research License on Large: free for research, requires a commercial license for production use
- Mistral Small and Devstral: Apache 2.0, unrestricted commercial use, smaller parameter counts
Hardware Requirements at Different Quantizations
At FP16, Mistral Large's 123B parameters need roughly 246GB of GPU memory, requiring 3-4x H100 80GB in tensor parallel. At FP8, that drops to roughly 123GB, fitting on 2x H100 80GB. At INT4/AWQ, it drops further to roughly 65-70GB, fitting on a single H100 80GB with headroom for moderate concurrency, or a single H200 with substantial headroom. Because it is dense rather than MoE, throughput per GPU is lower than an MoE model with similar total parameters but far fewer active parameters, so factor that into your concurrent-user sizing rather than comparing weight memory alone. Mistral Small, by contrast, runs comfortably on a single RTX 4090 or 5090 at INT4, making it the practical choice for edge or low-concurrency deployments where the Research License complexity is not worth navigating.
- FP16: approximately 246GB, 3-4x H100 80GB in tensor parallel
- FP8: approximately 123GB, 2x H100 80GB
- INT4/AWQ: approximately 65-70GB, single H100 80GB or single H200
- Dense architecture means lower throughput per GPU than an MoE model of similar memory footprint
Serving Stack Setup
vLLM and TensorRT-LLM both have mature Mistral Large support, and because it is a dense model without MoE routing complexity, the serving setup is comparatively straightforward: standard tensor parallelism across GPUs, no expert-parallel scheduling to configure. TensorRT-LLM is a reasonable choice here specifically because dense models are more stable targets for compiled-engine optimization than MoE models, and the throughput gain over vLLM is often worth the extra build complexity for high-volume production deployments. For Mistral Small and Devstral, llama.cpp/Ollama handles edge and workstation deployment cleanly given the smaller footprint.
Fine-Tuning and Customization Options
Mistral Large's dense architecture makes it a more forgiving fine-tuning target than an MoE model of similar size, since there is no expert-routing stability to worry about during training. LoRA and QLoRA on Mistral Large are well supported by Axolotl, LLaMA-Factory, and Unsloth, and full SFT is realistic on a modest multi-GPU cluster given the dense 123B parameter count. Note that fine-tuning terms are covered under the same Research License as base inference: confirm your commercial agreement explicitly covers derivative fine-tuned weights before deploying a fine-tune to production, since this is a common gap in initial license negotiations.
Security and License Considerations for Regulated Industries
The Mistral Research License is the central consideration here: production use of Mistral Large, including internal enterprise deployment behind your own firewall, requires a commercial license from Mistral, not just an accepted click-through. Budget legal and procurement time for this negotiation before committing engineering effort, since it is a materially different process than downloading Apache 2.0 or MIT weights and deploying immediately. Once licensed, on-prem deployment resolves data residency cleanly: Mistral is a French company subject to EU data protection norms, which some European regulated clients find preferable on data governance grounds independent of the technical deployment model. For ITAR and CMMC environments, the licensing negotiation itself needs to explicitly address air-gapped and on-prem terms, since some vendor commercial licenses assume connectivity for telemetry or usage reporting that an air-gapped environment cannot provide.
When Netray Recommends Mistral Large vs Alternatives
We recommend Mistral Large when a client has strong multilingual requirements, particularly European languages, or needs robust native function-calling for agentic workflows, and is prepared to navigate the commercial license process. Where license simplicity matters more than the multilingual edge, we typically recommend Qwen3 or DeepSeek V3, both Apache 2.0 or MIT licensed with comparable general capability. For lighter deployments where Apache 2.0 licensing is a hard requirement but Mistral's model family fits the use case, Mistral Small or Devstral often replace the need for Large entirely.
Frequently Asked Questions
Is Mistral Large free for commercial use?
No. Mistral Large is released under the Mistral Research License, which permits free use for research and testing but requires a separate commercial license from Mistral for production deployment, including internal enterprise use behind your own firewall. Mistral Small and Devstral, by contrast, are Apache 2.0 and free for unrestricted commercial use, which is worth considering if the license negotiation is a blocker.
How much GPU memory does Mistral Large need on-prem?
At INT4/AWQ quantization, Mistral Large's 123B dense parameters need roughly 65-70GB of GPU memory, fitting a single H100 80GB card with moderate concurrency headroom. At FP16, that grows to roughly 246GB, requiring three to four H100 80GB cards in tensor parallel, since it is a dense model where every parameter activates per token.
Why is Mistral Large slower per GPU than a similarly sized MoE model?
Mistral Large is a dense model, meaning all 123B parameters activate on every token, while an MoE model of similar total parameter count only activates a fraction of its weights per token. This gives dense models lower throughput per GPU for the same memory footprint, though it also makes their sizing and latency more predictable since there is no expert-routing variability.
Can I fine-tune Mistral Large and deploy it in production?
Yes, but confirm your commercial license agreement with Mistral explicitly covers derivative fine-tuned weights, not just base inference. This is a common gap in initial license negotiations. Technically, LoRA, QLoRA, and full SFT are all well supported for Mistral Large through Axolotl, LLaMA-Factory, and Unsloth, and the dense architecture is generally a more stable fine-tuning target than an MoE model.
Key Takeaways
- 1What Mistral Large Is: Architecture, Sizes, and License: Mistral Large 2, at 123B parameters, is a dense transformer rather than a mixture-of-experts model, which means every parameter activates on every token: compute cost scales directly with the full parameter count, unlike the sparse MoE models where active parameters are a fraction of total. This makes Mistral Large computationally heavier per token than an MoE model of similar total size, but it also makes memory and compute sizing far more predictable, since there is no expert-routing variability to account for.
- 2Hardware Requirements at Different Quantizations: At FP16, Mistral Large's 123B parameters need roughly 246GB of GPU memory, requiring 3-4x H100 80GB in tensor parallel. At FP8, that drops to roughly 123GB, fitting on 2x H100 80GB.
- 3Serving Stack Setup: vLLM and TensorRT-LLM both have mature Mistral Large support, and because it is a dense model without MoE routing complexity, the serving setup is comparatively straightforward: standard tensor parallelism across GPUs, no expert-parallel scheduling to configure. TensorRT-LLM is a reasonable choice here specifically because dense models are more stable targets for compiled-engine optimization than MoE models, and the throughput gain over vLLM is often worth the extra build complexity for high-volume production deployments.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
gpt-oss Hardware Sizing Calculator
Size VRAM, GPU count, and capital cost for OpenAI's Apache 2.0 licensed gpt-oss-20b or gpt-oss-120b, both natively quantized to MXFP4 for single-GPU deployment.
Free ToolGLM-4.5 On-Prem Sizing Calculator
Size VRAM, GPU count, and capital cost for GLM-4.5 or the smaller GLM-4.5-Air, both mixture-of-experts models tuned for agentic and coding workloads.
Free ToolMistral Large On-Prem Sizing Calculator
Estimate VRAM, GPU count, and capital cost to self-host Mistral Large's 123B dense parameter model, with predictable memory scaling and no MoE routing complexity.
Terms used in this article
Evaluating Mistral Large against a more permissively licensed alternative? Netray will map the license, hardware, and workload fit before you start a commercial license negotiation.
Related Resources
Gemma 3 Enterprise Deployment: An On-Prem Guide
Deploy Gemma 3 on-prem: 1B-27B sizes, multimodal support, Gemma license terms, GPU sizing, serving setup, fine-tuning, and when it fits enterprise workloads.
AI & AutomationQwen3 Enterprise Deployment: The On-Prem Guide
Deploy Qwen3 on-prem: MoE and dense sizes from 0.6B to 235B, Apache 2.0 license, GPU sizing by quantization, serving setup, fine-tuning, and when to use it.
AI & AutomationOpen Model License Compliance for Enterprises
Open-model license compliance: Llama community license thresholds, Apache 2.0 and MIT obligations, and acceptable use for enterprise deployments.