# Mistral Large On-Prem Sizing Calculator: 123B Dense Model Hardware Plan

> Source: https://netray.co/tools/mistral-large-on-prem-sizing-calculator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Estimate VRAM, GPU count, and capital cost to self-host Mistral Large's 123B dense parameter model, with predictable memory scaling and no MoE routing complexity.

## Category

on-prem-ai

## Tool

**Kind:** calculator

**Inputs:**

#### Weight precision

**Key:** quant_bytes

**Type:** select

**Default:** 1

**Unit:** bytes/param

**Help:** FP8 is the practical production default on Hopper and Blackwell GPUs; FP16 remains common on older A100 fleets.

**Options:**

##### FP16 / BF16 (2 bytes/param)

**Value:** 2

##### FP8 (1 byte/param)

**Value:** 1

##### INT4 / AWQ (0.5 bytes/param)

**Value:** 0.5

#### Serving context length

**Key:** context_tokens

**Type:** slider

**Default:** 32768

**Min:** 4096

**Max:** 131072

**Step:** 4096

**Unit:** tokens

**Help:** Mistral Large 2 supports up to 128K context natively.

#### Concurrent active users

**Key:** concurrent_users

**Type:** number

**Default:** 25

**Min:** 1

**Max:** 300

**Step:** 1

**Unit:** users

**Help:** Simultaneous in-flight requests during peak hours.

#### GPU type

**Key:** gpu_vram_gb

**Type:** select

**Default:** 80

**Unit:** GB VRAM

**Help:** VRAM per GPU; enter a matching price below.

**Options:**

##### A100 80GB SXM

**Value:** 80

##### H100 80GB SXM

**Value:** 80

##### H200 141GB SXM

**Value:** 141

#### Price per GPU

**Key:** gpu_price

**Type:** number

**Default:** 28000

**Min:** 5000

**Max:** 70000

**Step:** 500

**Unit:** $

**Help:** H100 80GB runs roughly $25,000-$32,000; a legacy A100 80GB fleet is often cheaper on the used market.

#### Cloud rental rate per equivalent GPU

**Key:** cloud_hourly_rate

**Type:** number

**Default:** 3

**Min:** 0.5

**Max:** 10

**Step:** 0.1

**Unit:** $/hr

**Help:** On-demand H100/A100 rates typically run $2-$6 per hour.

**Outputs:**

#### Model weights footprint

**Key:** model_weights_gb

**Expr:** 123 * quant_bytes

**Format:** number

**Description:** Mistral Large's 123B dense parameters at your chosen precision; every parameter activates on every token.

#### KV cache at peak concurrency

**Key:** kv_cache_gb

**Expr:** concurrent_users * context_tokens / 1000 * 0.45

**Format:** number

**Description:** Attention cache growth across concurrent sessions using standard grouped-query attention.

#### Total VRAM required

**Key:** total_vram_gb

**Expr:** model_weights_gb + kv_cache_gb + model_weights_gb * 0.05

**Format:** number

**Description:** Weights plus KV cache plus a 5% allowance for runtime overhead.

**Primary:** true

#### GPUs required

**Key:** gpus_required

**Expr:** ceil(total_vram_gb / gpu_vram_gb)

**Format:** number

**Description:** Minimum accelerator count for one serving replica at your target concurrency.

#### Hardware capital cost

**Key:** hardware_capex

**Expr:** gpus_required * gpu_price

**Format:** currency

**Description:** One-time GPU spend at your entered price per card.

**Primary:** true

#### Equivalent monthly cloud rental

**Key:** monthly_cloud_cost

**Expr:** gpus_required * cloud_hourly_rate * 730

**Format:** currency

**Description:** What the same GPU count would cost per month rented on-demand, for capex payback comparison.

**Disclaimer:** Planning estimates only. Real memory use depends on serving engine and batch size. Benchmark on representative traffic before finalizing a purchase order.

## Content

**Intro:** This free Mistral Large on-prem sizing calculator estimates GPU count, VRAM, and capital cost for self-hosting Mistral Large 2, a 123B dense parameter model, and it is built for platform teams who want predictable hardware sizing without mixture-of-experts complexity. Enter precision, context length, and concurrency, and the tool returns weight footprint, KV cache, total VRAM, GPU count, and cost against an equivalent cloud rental. Mistral Large is a dense model, meaning every parameter activates on every token and memory scales linearly and predictably, a meaningful planning advantage over the large MoE models now common in the open-weight landscape.

**Sections:**

#### How Mistral Large runs on hardware you own

**Body:** Mistral Large 2 is a 123B-parameter dense transformer, which means sizing is straightforward compared to mixture-of-experts models: there is no gap between active and total parameters, so the weight footprint you calculate is exactly the compute footprint too. At FP8, weights need roughly 123GB, which fits comfortably on two 80GB GPUs with room for KV cache and reasonable concurrency. This predictability is a genuine operational advantage for teams that have been burned by underestimating MoE memory requirements, and it makes Mistral Large a common choice when the priority is simple, defensible capacity planning over squeezing out maximum benchmark performance.

**Bullets:**

- 123B dense parameters at FP8 need roughly 123GB for weights, fitting on 2 GPUs at 80GB each.
- No active-versus-total parameter gap; memory and compute scale together predictably.
- 128K native context support makes it a strong fit for long-document ERP and engineering use cases.
- Mistral is headquartered in the EU, which matters for European data-sovereignty requirements.

#### Quantization tradeoffs for Mistral Large

**Body:** FP8 offers the best balance of quality and memory for Mistral Large on Hopper and Blackwell GPUs, with minimal measurable degradation on standard benchmarks. INT4/AWQ compresses further to roughly 62GB, potentially fitting a single 80GB GPU with modest concurrency headroom, which is attractive for smaller deployments or edge sites, though expect a somewhat larger quality hit on complex instruction-following and code generation than you would see on a smaller model, since Mistral Large's strength lies partly in nuanced instruction adherence that aggressive quantization can blunt. FP16 remains relevant for teams running an existing A100 fleet without FP8 tensor core support.

**Bullets:**

- FP8 is the recommended default on H100/H200/B200 hardware for near-lossless quality.
- INT4/AWQ shrinks the model to fit a single 80GB GPU but costs more on complex instruction-following tasks.
- FP16 remains the practical choice on legacy A100 fleets without native FP8 acceleration.
- Validate quantized checkpoints against instruction-following and structured-output tasks specifically.

#### Serving Mistral Large with vLLM and SGLang

**Body:** Mistral Large has straightforward tensor-parallel support in both vLLM and SGLang, without the expert-parallel complexity mixture-of-experts models require. This makes it one of the simpler large models to stand up a production serving stack for: two to four GPUs with standard tensor parallelism, continuous batching for throughput, and no need to reason about expert routing or cross-expert interconnect traffic. For teams new to self-hosting large open-weight models, Mistral Large's dense architecture is often a gentler starting point than an MoE flagship of similar benchmark quality, precisely because the operational model matches intuitions built from smaller dense models.

**Bullets:**

- Standard tensor parallelism across 2-4 GPUs, no expert-parallel configuration required.
- Continuous batching in vLLM or SGLang delivers strong aggregate throughput for dense architectures.
- Simpler operational model than MoE flagships makes it a good first large-model deployment.
- 128K context support benefits from FlashAttention-class kernels for efficient long-context serving.

#### When to fine-tune Mistral Large instead of prompting

**Body:** Fine-tune Mistral Large when you need it to consistently follow a complex structured output schema, adopt a specific technical writing style, or handle domain terminology that generic prompting struggles to enforce reliably at scale. LoRA and QLoRA are practical on 123B given standard multi-GPU training infrastructure, and Mistral's own fine-tuning tooling is well documented. A common enterprise pattern is fine-tuning for document generation tasks, engineering specification summarization, or structured ERP report generation, where consistency of format matters as much as raw language quality.

#### How Netray deploys and customizes Mistral Large on-prem

**Body:** Netray deploys Mistral Large for manufacturers who need European data-sovereignty guarantees alongside on-prem control, a combination that makes Mistral a frequent choice for customers with EU operations or European customer flow-down requirements. We handle tensor-parallel serving configuration, quantization validation against your document types, and fine-tuning for structured ERP and engineering-document generation tasks. Integration with SyteLine, LN, and M3 completes the picture so the model works directly against your operational data. Engagements typically start with a two-week sizing and benchmark phase.

**Cta:** Get a benchmarked Mistral Large deployment plan sized to your concurrency, context, and sovereignty requirements.

**Faq:**

#### Is Mistral Large easier to size than a mixture-of-experts model?

**Answer:** Yes, meaningfully so. As a dense model, every one of Mistral Large's 123B parameters activates on every token, so there is no gap between the memory footprint and the compute footprint to reason about. Sizing is a straightforward calculation of parameters times bytes per parameter, without the added complexity of accounting for total experts versus active experts that mixture-of-experts models require.

#### Can Mistral Large run on a single GPU?

**Answer:** Only at aggressive quantization. At INT4/AWQ, weights compress to roughly 62GB, which fits a single 80GB GPU with modest headroom for KV cache and concurrency. At FP8 or FP16, plan for at least two GPUs to hold the full 123GB or 246GB weight footprint respectively, plus room for KV cache and runtime overhead.

#### Why do European manufacturers often choose Mistral Large specifically?

**Answer:** Mistral AI is headquartered in France and subject to EU jurisdiction, which matters for customers with strict European data-sovereignty requirements or defense-sector customer flow-down clauses that specify non-US model providers. Combined with on-prem deployment, this satisfies data residency requirements that a US-based model provider, even open-weight, cannot fully address for some regulated European contracts.

#### How does Mistral Large compare to Llama 3.3 70B for on-prem deployment?

**Answer:** Mistral Large is larger (123B versus 70B) and generally scores higher on complex reasoning and instruction-following benchmarks, but needs roughly 1.5-2x the GPU count at equivalent precision. Llama 3.3 70B is the more economical choice when task complexity does not require Mistral Large's extra capacity; benchmark both against your evaluation set, since the gap on real enterprise tasks is often smaller than published leaderboards suggest.

## Search terms

Mistral Large on-prem, Mistral Large GPU requirements, Mistral Large hardware calculator, self-hosted Mistral Large, Mistral Large VRAM, Mistral Large quantization, Mistral Large fine-tuning, European sovereign AI Mistral

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json