# Open-Weight Model Selector: Find Your Right LLM Size Class > Source: https://netray.co/tools/open-weight-model-selector > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class. ## Category on-prem-ai ## Tool **Kind:** assessment **Questions:** #### What GPU budget or capacity can you realistically dedicate to model serving? **Options:** ##### A single 24GB workstation card at most **Score:** 0 ##### One server-class GPU (48-80GB) **Score:** 1 ##### 2-4 server-class GPUs **Score:** 2 ##### 4-8+ GPUs or approved budget for a dedicated cluster **Score:** 3 #### How complex are the tasks the model must handle well? **Help:** Be honest about the hardest 20% of tasks, not the average. **Options:** ##### Classification, extraction, and simple summarization **Score:** 0 ##### Document Q&A and drafting over retrieved context (RAG) **Score:** 1 ##### Multi-step reasoning, analysis, and code assistance **Score:** 2 ##### Complex agentic workflows with tool use and long chains of reasoning **Score:** 3 #### What are your latency expectations for a typical response? **Options:** ##### Batch or overnight processing is fine **Score:** 0 ##### Under 30 seconds is acceptable **Score:** 1 ##### Interactive: first token within 2-3 seconds **Score:** 2 ##### Snappy interactive at high concurrency across many users **Score:** 3 #### How many people or processes will use the model in production? **Options:** ##### A handful of pilot users **Score:** 0 ##### One team or department (10-50 users) **Score:** 1 ##### Hundreds of users or several automated pipelines **Score:** 2 ##### Enterprise-wide rollout plus high-volume automation **Score:** 3 #### Do you plan to fine-tune the model on your own data? **Help:** Fine-tuning smaller models often beats prompting larger ones for narrow domains. **Options:** ##### No fine-tuning planned; prompting only **Score:** 0 ##### Maybe later; starting with prompting and RAG **Score:** 1 ##### Yes, LoRA-style tuning on domain documents or transcripts **Score:** 2 ##### Yes, ongoing tuning pipeline with curated training data and evals **Score:** 3 #### How will you evaluate whether a model is good enough for your tasks? **Options:** ##### Informal vibes testing by a few people **Score:** 0 ##### A small shared set of example prompts **Score:** 1 ##### A written gold set of test cases with pass criteria **Score:** 2 ##### Automated evaluation harness run against every candidate model **Score:** 3 #### What is your team's experience operating self-hosted model serving? **Options:** ##### None; this would be our first deployment **Score:** 0 ##### Experimented with Ollama or similar on workstations **Score:** 1 ##### Run a pilot vLLM or TGI deployment on a server **Score:** 2 ##### Operate production inference with monitoring and upgrades today **Score:** 3 #### How long are the contexts the model must handle per request? **Options:** ##### Short prompts under 4K tokens **Score:** 0 ##### Moderate: 4K-16K tokens with retrieved snippets **Score:** 1 ##### Long: 16K-64K tokens for full documents **Score:** 2 ##### Very long: 64K+ tokens across multiple large documents **Score:** 3 #### How sensitive is the data the model will process? **Help:** Higher sensitivity strengthens the case for fully self-hosted open weights. **Options:** ##### Public or low-sensitivity internal data **Score:** 0 ##### Confidential business data **Score:** 1 ##### Regulated data (CUI, export-controlled, PII at scale) **Score:** 2 ##### Classified or ITAR data in isolated environments **Score:** 3 #### What licensing constraints matter for your deployment? **Options:** ##### Have not reviewed model licenses at all **Score:** 0 ##### Aware of licenses but no legal review yet **Score:** 1 ##### Legal has approved specific permissive licenses (Apache 2.0 etc.) **Score:** 2 ##### Full license review process including acceptable-use terms and redistribution **Score:** 3 **Bands:** #### Start Small: 4B-8B Model Class **Min:** 0 **Max:** 39 **Verdict:** Your hardware budget, task profile, or operational maturity points to the small model class. That is not a consolation prize - modern 8B models handle classification, extraction, and grounded RAG answering remarkably well, run on a single modest GPU, and let your team build serving skills cheaply before scaling up. **Recommendations:** - Deploy an 8B-class instruct model (Llama or Qwen family) quantized to INT8 on one GPU - Pair it with solid RAG retrieval - grounding lifts small-model quality more than size does - Build a written gold set of 30-50 test cases now so future upgrades are measurable - Use Ollama or vLLM on a single node; defer cluster complexity until demand proves out - Route the rare hard task to a compliant API tier rather than oversizing hardware #### The Sweet Spot: 14B-32B Model Class **Min:** 40 **Max:** 69 **Verdict:** Your profile fits the mid-size class where most enterprise deployments land in 2026. Models in the 14B-32B range deliver strong reasoning and document handling on one or two server GPUs, and quantized 32B models now rival much larger models of a generation ago. Your constraint is more likely evaluation rigor than raw capability. **Recommendations:** - Benchmark a 14B and a 32B candidate head-to-head on your own gold set before committing - Serve INT8 or INT4 quantized on 1-2 80GB-class GPUs with vLLM continuous batching - Invest in LoRA fine-tuning on domain data - a tuned 14B often beats a stock 70B on narrow tasks - Define latency SLOs and load test at 2x expected concurrency before launch - Plan a quarterly model refresh cadence; this class improves fastest between releases #### Frontier Open Weights: 70B+ Class **Min:** 70 **Max:** 100 **Verdict:** Your workload complexity, scale, and operational maturity justify frontier open-weight models - dense 70B-class or large mixture-of-experts architectures. You have the hardware capacity and evaluation discipline to exploit them, and your data sensitivity profile makes self-hosting at this tier strategically sound rather than merely economical. **Recommendations:** - Evaluate a dense 70B and a large MoE candidate; MoE gives frontier quality at lower per-token compute - Serve with tensor parallelism across 2-4 GPUs at FP8/INT8; validate interconnect bandwidth first - Implement tiered routing: small model for easy traffic, frontier weights for the hard 20% - Stand up automated evals in CI so model refreshes ship with regression gates - Budget KV cache generously for your long-context and agentic workloads - it will dominate VRAM ## Content **Intro:** This assessment matches your organization to the right open-weight LLM size class - small (4B-8B), mid-size (14B-32B), or frontier (70B+) - for teams planning a self-hosted deployment. Model selection is not about grabbing the biggest weights your hardware can hold; it is a trade among task complexity, latency, concurrency, fine-tuning plans, and operational maturity. Ten questions score those factors and place you in a size class with concrete deployment recommendations, including which specific levers - quantization, fine-tuning, routing - matter most for your band. **Sections:** #### Why size class is the right first decision **Body:** The open-weight ecosystem moves fast - specific model names age in months - but size classes are stable planning units because they map directly to hardware, latency, and cost tiers. An 8B model runs on one modest GPU and answers fast; a 32B model needs a server card or two and reasons noticeably better; a 70B+ model demands tensor parallelism across multiple GPUs and delivers near-frontier quality. Choosing class first lets you size hardware, budget, and operations honestly, then pick the best current model within the class at deployment time - which is how Netray runs every selection engagement. #### What the questions are really measuring **Body:** The assessment weighs four forces that push your size class up or down. Scores are deliberately balanced so that hardware alone cannot carry you into the frontier band - capability without evaluation discipline and operational maturity produces expensive disappointments. **Bullets:** - Demand side: task complexity, context length, and user scale push toward larger classes - Supply side: GPU budget and serving experience cap what you can operate well - Quality control: evaluation rigor determines whether any choice can be validated - Strategic fit: data sensitivity and licensing shape how much self-hosting matters #### Interpreting your band - and the common overshoot **Body:** The most frequent selection error we see is overshooting: teams deploy a 70B model for workloads a tuned 14B handles indistinguishably, then pay for it in hardware, latency, and operational drag every day after. If you scored mid-band, trust it - a quantized 32B with good retrieval covers the large majority of enterprise document and reasoning tasks in 2026. The reverse error matters too: if agentic workflows and long chains of tool use are genuinely central, undersizing produces failure loops that burn more tokens than a larger model would have. Let your gold-set results, not demos, make the final call. #### How Netray runs model selection for clients **Body:** Netray turns this band into a named model and a running deployment. We maintain current benchmarks across the Llama, Qwen, Mistral, and leading MoE families, build a gold evaluation set from your real prompts and documents, and test candidates head-to-head on your tasks - including quantized variants, because deployment precision is what you will actually run. The engagement ends with a deployed, monitored serving stack and a written selection rationale your architecture board and security team can audit. **Cta:** Find your size class, then let Netray benchmark specific models against your real tasks before you deploy. **Faq:** #### Why does the assessment recommend a size class instead of a specific model? **Answer:** Because specific open-weight leaders change every few months, while the hardware, latency, and cost profile of each size class stays stable. A recommendation naming one model would be stale before your procurement cycle finished. Class-first planning lets you budget hardware and operations now, then select the best current model in that class - validated on your own evaluation set - at deployment time. #### Can a fine-tuned small model really beat a larger general model? **Answer:** For narrow, well-defined tasks, yes - consistently. A 14B model LoRA-tuned on a few thousand quality examples from your domain routinely outperforms a stock 70B on that task, while costing a quarter as much to serve. The larger model retains the edge on open-ended reasoning and unfamiliar tasks. That is why the assessment asks about fine-tuning plans: a real tuning pipeline effectively buys you one size class of headroom. #### What about mixture-of-experts (MoE) models - which band are they? **Answer:** Large MoE models belong in the frontier band operationally: their full parameter count must fit in VRAM even though only a fraction activates per token, so memory needs are frontier-class while per-token compute is closer to mid-size. They are excellent value if you have the VRAM and multi-GPU serving experience. If you scored below the frontier band, dense mid-size models are the simpler, safer operational choice. ## Search terms open weight model selection, which llm to self host, llama vs qwen enterprise, small language model vs 70b, open source llm comparison enterprise, choose llm model size --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json