Deploying Llama 4 On-Prem: A Practical Enterprise Guide
Llama 4 is Meta's first natively multimodal, mixture-of-experts open-weight release, shipped as two production models: Scout, a 109B-total-parameter MoE with 17B active parameters across 16 experts, and Maverick, a 400B-total-parameter MoE also running 17B active parameters but spread across 128 experts. Both support very long context windows and native image understanding, which makes them attractive for engineering drawing review and long-document analysis workloads that used to require a separate vision pipeline. On-prem deployment matters here for two reasons: the license carries real restrictions that a legal team needs to sign off on before production use, and the MoE architecture changes GPU sizing math in ways that trip up teams used to dense models. This guide covers the architecture and license, hardware sizing at each quantization level, the serving stack, fine-tuning options, and when Netray recommends Llama 4 over Qwen3, DeepSeek, or Mistral for a regulated manufacturing client.
What Llama 4 Is: Architecture, Sizes, and License
Scout and Maverick are both sparse mixture-of-experts transformers, meaning every token only activates a subset of the total parameters, which keeps inference compute closer to a 17B dense model even though the weights on disk are far larger. Scout targets single high-memory GPU or small multi-GPU deployments; Maverick targets multi-GPU nodes and competes with larger dense models on reasoning and coding benchmarks. Both ship under the Llama 4 Community License, not an OSI-approved open source license: it permits commercial use and fine-tuning, but organizations with more than 700 million monthly active users need a separate license from Meta, and the license includes an acceptable use policy and an attribution requirement on derivative models. Read the license text yourself before committing engineering time; do not rely on a summary, including this one.
- Scout: 109B total parameters, 17B active, 16 experts, single-GPU-friendly at quantized precision
- Maverick: 400B total parameters, 17B active, 128 experts, needs multi-GPU or multi-node serving
- Llama 4 Community License: commercial use allowed, 700M MAU threshold, attribution and acceptable use terms apply
- Native multimodality (image plus text) removes the need for a bolted-on vision encoder in document workflows
Hardware Requirements at Different Quantizations
Because Llama 4 is MoE, you still need enough GPU memory to hold every expert, even though only a fraction activates per token; memory sizing follows total parameters, not active parameters. Scout at FP8 needs roughly 110GB of GPU memory for weights alone, which fits on a single H200 (141GB) or two H100 80GB cards with tensor parallelism; at INT4/AWQ that drops to around 55-60GB, fitting on a single H100 80GB with headroom for KV cache. Maverick at FP8 needs roughly 400GB, requiring a multi-GPU node such as 4x H100 80GB or 2x H200; at INT4 it drops to around 200GB, feasible on 2x H100 80GB or a single 8x A100 40GB legacy node with careful tensor parallelism. Add 15-25% headroom for KV cache at your target context length and concurrency, more if you are running long-document or multi-image workloads.
- Scout FP8: approximately 110GB, one H200 or two H100 80GB in tensor parallel
- Scout INT4/AWQ: approximately 55-60GB, one H100 80GB with room for KV cache
- Maverick FP8: approximately 400GB, 4x H100 80GB or 2x H200 minimum
- Maverick INT4: approximately 200GB, 2x H100 80GB or an 8x A100 40GB legacy fleet
Serving Stack: vLLM, SGLang, and TensorRT-LLM
vLLM has the most mature Llama 4 support as of 2026, with native MoE expert-parallel scheduling, FP8 and AWQ quantization, and continuous batching that handles the mixed text-and-image request pattern reasonably well. SGLang is a strong alternative when your workload is dominated by structured output and repeated prompt prefixes, since its RadixAttention caching can meaningfully cut latency on document-heavy pipelines. TensorRT-LLM gives the best raw throughput on NVIDIA hardware but costs more engineering time to build and maintain the compiled engine, and every model or quantization change means a rebuild; reserve it for stable, high-volume production paths rather than a first deployment. For most enterprise pilots we start on vLLM, benchmark against your actual traffic pattern, and only move to TensorRT-LLM once volume justifies the engineering overhead.
Fine-Tuning and Customization Options
Full fine-tuning of Scout or Maverick is rarely practical outside a well-funded ML team; the MoE routing layer makes full SFT expensive and easy to destabilize if the learning rate and expert load balancing are not tuned carefully. LoRA and QLoRA adapters targeting the attention and a subset of expert layers are the practical path for domain adaptation, and tools like Axolotl and LLaMA-Factory now support MoE-aware LoRA for the Llama 4 family. DPO works well for tone and refusal-behavior alignment once you have a base fine-tune. Plan for meaningfully more GPU-hours than a same-active-parameter dense model, because gradient computation still touches the routing mechanism even when using parameter-efficient methods, and validate on a held-out set that stresses your document types before promoting a fine-tune to production.
Security and License Considerations for Regulated Industries
For aerospace, defense, and export-controlled manufacturing environments, the Llama 4 Community License's acceptable use policy and attribution clause need explicit legal review before the model touches ITAR-controlled technical data, even when the deployment is fully on-prem and air-gapped. On-prem deployment does resolve the data residency question entirely: weights run on your own GPUs, no inference request leaves your network, and there is no third-party API logging or retention to negotiate. Keep a documented model card with the exact weight checksum, license version, and quantization method used in production, since audit and compliance reviews increasingly ask for that provenance chain alongside the usual access control and logging evidence.
When Netray Recommends Llama 4 vs Alternatives
We reach for Llama 4 Scout when a client needs native multimodal document understanding, such as reading engineering drawings alongside spec text, on a single-GPU or dual-GPU footprint, and Maverick when the same workload needs stronger reasoning at higher volume and the client already has a multi-GPU node. Where the license restrictions are a blocker, or the workload is pure text reasoning at large scale, we typically recommend Qwen3 or DeepSeek V3 instead, both under more permissive licenses. Netray runs a structured hardware sizing and license fit assessment before recommending any specific model, using our own benchmark corpus rather than published leaderboard numbers, because leaderboard rankings rarely predict performance on your actual document mix.
Frequently Asked Questions
Can Llama 4 run on a single GPU on-prem?
Llama 4 Scout can run on a single GPU on-prem if it is a high-memory card: an H200 at FP8 (roughly 110GB) or an H100 80GB at INT4/AWQ (roughly 55-60GB), with headroom left for KV cache. Maverick, at 400B total parameters, needs a multi-GPU node even at INT4 quantization, typically two H100 80GB cards or better for production-grade concurrency.
Is Llama 4 open source?
No, not in the OSI sense. Llama 4 ships under the Llama 4 Community License, which permits commercial use and fine-tuning but adds a 700 million monthly active user threshold requiring a separate license from Meta, plus an acceptable use policy and attribution requirements. Legal teams in regulated industries should review the license text directly before production deployment, especially where export-controlled data is involved.
How much GPU memory does Llama 4 Maverick need?
Maverick's 400B total parameters mean roughly 400GB of GPU memory at FP8 for weights alone, requiring four H100 80GB cards or two H200 cards in tensor parallel. At INT4 quantization that drops to approximately 200GB, feasible on two H100 80GB cards. Add 15 to 25 percent for KV cache depending on context length and concurrent request volume.
Should I fine-tune Llama 4 or use retrieval-augmented generation instead?
For most enterprise knowledge tasks, start with RAG: it is cheaper, faster to iterate, and keeps the model's general reasoning intact. Reach for LoRA or QLoRA fine-tuning when you need consistent domain-specific tone, terminology, or output format that RAG cannot reliably enforce through prompting alone, and validate any fine-tune on a held-out set before production promotion.
Key Takeaways
- 1What Llama 4 Is: Architecture, Sizes, and License: Scout and Maverick are both sparse mixture-of-experts transformers, meaning every token only activates a subset of the total parameters, which keeps inference compute closer to a 17B dense model even though the weights on disk are far larger. Scout targets single high-memory GPU or small multi-GPU deployments; Maverick targets multi-GPU nodes and competes with larger dense models on reasoning and coding benchmarks.
- 2Hardware Requirements at Different Quantizations: Because Llama 4 is MoE, you still need enough GPU memory to hold every expert, even though only a fraction activates per token; memory sizing follows total parameters, not active parameters. Scout at FP8 needs roughly 110GB of GPU memory for weights alone, which fits on a single H200 (141GB) or two H100 80GB cards with tensor parallelism; at INT4/AWQ that drops to around 55-60GB, fitting on a single H100 80GB with headroom for KV cache.
- 3Serving Stack: vLLM, SGLang, and TensorRT-LLM: vLLM has the most mature Llama 4 support as of 2026, with native MoE expert-parallel scheduling, FP8 and AWQ quantization, and continuous batching that handles the mixed text-and-image request pattern reasonably well. SGLang is a strong alternative when your workload is dominated by structured output and repeated prompt prefixes, since its RadixAttention caching can meaningfully cut latency on document-heavy pipelines.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Llama 4 Hardware Requirements Calculator
Estimate VRAM, GPU count, and capital cost to run Llama 4 Scout or Maverick on your own hardware, accounting for full mixture-of-experts weight loading and KV cache growth.
Free Toolgpt-oss Hardware Sizing Calculator
Size VRAM, GPU count, and capital cost for OpenAI's Apache 2.0 licensed gpt-oss-20b or gpt-oss-120b, both natively quantized to MXFP4 for single-GPU deployment.
Free ToolGLM-4.5 On-Prem Sizing Calculator
Size VRAM, GPU count, and capital cost for GLM-4.5 or the smaller GLM-4.5-Air, both mixture-of-experts models tuned for agentic and coding workloads.
Terms used in this article
Considering Llama 4 for an on-prem deployment? Netray will run a sizing and license fit assessment against your actual workload before you commit GPU budget.
Related Resources
Qwen3 Enterprise Deployment: The On-Prem Guide
Deploy Qwen3 on-prem: MoE and dense sizes from 0.6B to 235B, Apache 2.0 license, GPU sizing by quantization, serving setup, fine-tuning, and when to use it.
AI & AutomationDeepSeek V3 Enterprise Deployment Guide
Deploy DeepSeek V3 on-prem: 671B MoE architecture, MIT license, GPU sizing by quantization, vLLM serving setup, fine-tuning options, and when to use it.
AI & AutomationHow Netray Evaluates Every New AI Model in 48 Hours
Netray's repeatable framework for evaluating any new open-weight AI model release within 48 hours: license, hardware fit, benchmarks, and deployment risk.