Mistral Large On-Prem Deployment: A Practical Enterprise Guide

What Mistral Large Is: Architecture, Sizes, and License

Mistral Large 2, at 123B parameters, is a dense transformer rather than a mixture-of-experts model, which means every parameter activates on every token: compute cost scales directly with the full parameter count, unlike the sparse MoE models where active parameters are a fraction of total. This makes Mistral Large computationally heavier per token than an MoE model of similar total size, but it also makes memory and compute sizing far more predictable, since there is no expert-routing variability to account for. 0 and free for unrestricted commercial deployment.

• Mistral Large 2: 123B dense parameters, strong multilingual and function-calling performance
• Dense architecture means full parameter count activates per token, unlike sparse MoE models
• Mistral Research License on Large: free for research, requires a commercial license for production use
• Mistral Small and Devstral: Apache 2.0, unrestricted commercial use, smaller parameter counts

Hardware Requirements at Different Quantizations

At FP16, Mistral Large's 123B parameters need roughly 246GB of GPU memory, requiring 3-4x H100 80GB in tensor parallel. At FP8, that drops to roughly 123GB, fitting on 2x H100 80GB. At INT4/AWQ, it drops further to roughly 65-70GB, fitting on a single H100 80GB with headroom for moderate concurrency, or a single H200 with substantial headroom.

Because it is dense rather than MoE, throughput per GPU is lower than an MoE model with similar total parameters but far fewer active parameters, so factor that into your concurrent-user sizing rather than comparing weight memory alone. Mistral Small, by contrast, runs comfortably on a single RTX 4090 or 5090 at INT4, making it the practical choice for edge or low-concurrency deployments where the Research License complexity is not worth navigating.

• FP16: approximately 246GB, 3-4x H100 80GB in tensor parallel
• FP8: approximately 123GB, 2x H100 80GB
• INT4/AWQ: approximately 65-70GB, single H100 80GB or single H200
• Dense architecture means lower throughput per GPU than an MoE model of similar memory footprint

Serving Stack Setup

vLLM and TensorRT-LLM both have mature Mistral Large support, and because it is a dense model without MoE routing complexity, the serving setup is comparatively straightforward: standard tensor parallelism across GPUs, no expert-parallel scheduling to configure. TensorRT-LLM is a reasonable choice here specifically because dense models are more stable targets for compiled-engine optimization than MoE models, and the throughput gain over vLLM is often worth the extra build complexity for high-volume production deployments. cpp/Ollama handles edge and workstation deployment cleanly given the smaller footprint.

Fine-Tuning and Customization Options

Mistral Large's dense architecture makes it a more forgiving fine-tuning target than an MoE model of similar size, since there is no expert-routing stability to worry about during training. LoRA and QLoRA on Mistral Large are well supported by Axolotl, LLaMA-Factory, and Unsloth, and full SFT is realistic on a modest multi-GPU cluster given the dense 123B parameter count. Note that fine-tuning terms are covered under the same Research License as base inference: confirm your commercial agreement explicitly covers derivative fine-tuned weights before deploying a fine-tune to production, since this is a common gap in initial license negotiations.

Security and License Considerations for Regulated Industries

The Mistral Research License is the central consideration here: production use of Mistral Large, including internal enterprise deployment behind your own firewall, requires a commercial license from Mistral, not just an accepted click-through. 0 or MIT weights and deploying immediately. Once licensed, on-prem deployment resolves data residency cleanly: Mistral is a French company subject to EU data protection norms, which some European regulated clients find preferable on data governance grounds independent of the technical deployment model.

For ITAR and CMMC environments, the licensing negotiation itself needs to explicitly address air-gapped and on-prem terms, since some vendor commercial licenses assume connectivity for telemetry or usage reporting that an air-gapped environment cannot provide.

When Netray Recommends Mistral Large vs Alternatives

We recommend Mistral Large when a client has strong multilingual requirements, particularly European languages, or needs robust native function-calling for agentic workflows, and is prepared to navigate the commercial license process. 0 or MIT licensed with comparable general capability. 0 licensing is a hard requirement but Mistral's model family fits the use case, Mistral Small or Devstral often replace the need for Large entirely.