# NVIDIA GPU Selector: Find the Right GPU Tier for Your LLM Workload > Source: https://netray.co/tools/nvidia-gpu-selector-for-llm-workloads > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters. ## Category on-prem-ai ## Tool **Kind:** assessment **Questions:** #### What is your primary workload type? **Help:** Training and fine-tuning need more headroom than inference alone, and multi-node work needs fast interconnect. **Options:** ##### Inference only, low concurrency **Score:** 1 ##### Inference at moderate to high concurrency or long context **Score:** 4 ##### LoRA or QLoRA fine-tuning plus inference **Score:** 7 ##### Full fine-tuning, continued pretraining, or multi-node distributed training **Score:** 10 #### What is the largest active model size you need to run? **Help:** For mixture-of-experts models like DeepSeek or Kimi K2, use active parameters per token, not total parameters. **Options:** ##### Under 8B parameters **Score:** 1 ##### 8B to 34B parameters **Score:** 4 ##### 34B to 100B parameters, or MoE with over 20B active **Score:** 7 ##### Over 100B parameters, or frontier MoE class **Score:** 10 #### How many concurrent requests must one GPU node serve? **Options:** ##### Under 10 concurrent users **Score:** 1 ##### 10 to 50 concurrent users **Score:** 4 ##### 50 to 200 concurrent users **Score:** 7 ##### Over 200 concurrent users, or a shared multi-tenant platform **Score:** 10 #### How latency-sensitive is the workload? **Options:** ##### Batch or offline processing is acceptable **Score:** 1 ##### A few seconds of response time is fine **Score:** 4 ##### Sub-second interactive response is required **Score:** 7 ##### Real-time streaming at scale with strict SLAs **Score:** 10 #### What is your budget per GPU? **Options:** ##### Under $10,000 per GPU **Score:** 1 ##### $10,000 to $25,000 per GPU **Score:** 4 ##### $25,000 to $45,000 per GPU **Score:** 7 ##### $45,000 or more per GPU **Score:** 10 #### What power and cooling capacity does your facility have available? **Help:** This is the constraint most teams discover too late, after hardware has already been ordered. **Options:** ##### No dedicated capacity yet, edge or office environment **Score:** 1 ##### Air-cooled rack, up to about 15 kW **Score:** 4 ##### Air-cooled rack up to about 30 kW, or rear-door heat exchangers **Score:** 7 ##### Liquid-cooled rack capacity above 40 kW **Score:** 10 #### Do you need to scale beyond a single 8-GPU server? **Options:** ##### No, one server is sufficient **Score:** 1 ##### Possibly within the next 12 to 18 months **Score:** 4 ##### Yes, within the next 6 to 12 months **Score:** 7 ##### Yes, immediately, with InfiniBand or high-speed Ethernet fabric planned **Score:** 10 #### How much VRAM headroom do you need for context length and KV cache? **Options:** ##### Modest, 8K to 16K context with a handful of users **Score:** 1 ##### Moderate, 32K to 64K context with dozens of concurrent users **Score:** 4 ##### Large, 128K or more context with over a hundred concurrent users **Score:** 7 ##### Very large, 200K-plus context combined with high concurrency **Score:** 10 **Bands:** #### Entry tier: RTX 5090 or L40S class is enough **Min:** 0 **Max:** 25 **Verdict:** Your workload profile does not justify data-center-class accelerators. A single server with RTX 5090 or L40S GPUs, air cooled and standing under 15 kW, covers this comfortably and preserves budget for the platform engineering work that determines whether the pilot actually succeeds. Revisit this assessment if concurrency or model size grows. **Recommendations:** - Start with one to two RTX 5090 or L40S GPUs in a single server rather than pre-buying data-center-class hardware. - Use quantized 7B to 14B open-weight models such as Qwen3 or Gemma 3 to fit comfortably in available VRAM. - Standard air cooling and a single 20A circuit are sufficient at this tier; do not over-invest in facility upgrades yet. - Revisit this assessment every quarter as concurrency and model size requirements evolve. #### Mid tier: A100 80GB or L40S cluster fits your workload **Min:** 26 **Max:** 50 **Verdict:** You need real data-center-class capacity but not the newest generation. An A100 80GB or L40S cluster of two to four GPUs in a single node, air cooled with good containment, handles this workload profile at meaningfully lower capex than H100-class hardware while leaving room to grow. **Recommendations:** - Size a single 4-GPU A100 80GB or L40S node before considering multi-node scaling. - Budget for a 15-30 kW rack with proper containment or rear-door heat exchangers. - Use FP8 or INT4 quantization to stretch VRAM for models in the 14B to 34B range. - Plan a refresh evaluation in 18-24 months as newer-generation price per token improves. #### High-end tier: H100 or H200 SXM is the right target **Min:** 51 **Max:** 75 **Verdict:** Your concurrency, model size, and latency requirements point to H100 or H200 SXM class hardware. This tier delivers the memory bandwidth needed for responsive decode at meaningful concurrency and supports models up to roughly 70B dense or mid-size MoE without heroic quantization. **Recommendations:** - Standardize on an 8-GPU H100 or H200 SXM node with NVLink for tensor parallelism. - Plan for a 30-40 kW rack with liquid cooling or high-capacity rear-door heat exchangers. - Benchmark vLLM or SGLang on your actual prompts before finalizing GPU count. - Budget lead time of 8-16 weeks for allocation on high-demand SKUs. - Evaluate FP8 serving to raise effective throughput without added hardware. #### Flagship tier: B200 multi-node cluster with InfiniBand **Min:** 76 **Max:** 100 **Verdict:** Your requirements, large models, high concurrency, and multi-node training, call for a flagship B200 deployment with InfiniBand fabric. This is a significant capital and facility commitment and should be preceded by a formal architecture and procurement plan, not ordered directly from this assessment. **Recommendations:** - Commission a full reference architecture review before procurement, including networking topology. - Plan for liquid-cooled racks above 40 kW and confirm electrical service can support the full cluster. - Budget InfiniBand or 400Gb Ethernet fabric at roughly 15-25% of combined GPU and server cost. - Engage procurement 4-6 months ahead of need given lead times on flagship accelerators. - Validate multi-node scaling efficiency on a smaller pilot before committing to full cluster size. ## Content **Intro:** This free NVIDIA GPU selector scores your workload across eight dimensions, model size, concurrency, latency sensitivity, budget, facility power, and scaling needs, and recommends a GPU tier from entry-level RTX class through flagship B200 multi-node clusters. It is built for engineers and IT leaders who are early in planning an on-prem AI deployment and need a defensible starting point before requesting vendor quotes. Answer eight questions about your actual workload, not your aspirations, and the tool returns a scored recommendation with concrete next steps for that tier. **Sections:** #### Why GPU selection is not just about model size **Body:** Teams default to sizing GPUs from parameter count alone, but concurrency, latency SLA, and facility power constrain the decision just as strongly. A 13B model serving 300 concurrent interactive users needs more aggregate memory bandwidth than a 70B model answering ten batch queries an hour. Facility power is the constraint most often discovered after hardware is already on order: a flagship-tier recommendation is meaningless if the server room has 12 kW of spare capacity and a 20-week electrical upgrade timeline ahead of it. #### How the tiers map to real hardware in 2026 **Body:** Each tier in this assessment corresponds to a real, procurable configuration rather than an abstract score. Entry tier means workstation-class GPUs you can order this week; flagship tier means a multi-month procurement and facility project. Matching your score honestly to the right tier avoids the two most common mistakes: over-buying flagship hardware for a workload that never needed it, and under-buying entry hardware for a pilot that was always going to scale to production concurrency within a quarter. **Bullets:** - Entry tier: RTX 5090 or L40S, single server, standard office or closet power. - Mid tier: A100 80GB or L40S cluster, one node, light facility upgrade. - High-end tier: H100 or H200 SXM, 8-GPU node, liquid cooling or rear-door heat exchangers. - Flagship tier: B200 multi-node with InfiniBand, full data center or colocation build. #### What to do with your result **Body:** Treat the recommended tier as a starting point for a vendor conversation and a facility assessment, not a final purchase order. Validate the model size and quantization assumptions against your actual candidate models, confirm real electrical and cooling capacity with facilities before committing to a rack density, and benchmark a serving engine like vLLM or SGLang on your real prompts before finalizing GPU count. The single biggest source of over-provisioning we see is sizing for peak theoretical load instead of measured production traffic. #### How Netray turns this into a procured cluster **Body:** Netray takes teams from this kind of directional assessment through a validated reference architecture, vendor sourcing, and installed hardware for aerospace, defense, and electronics manufacturers running private AI. We benchmark your actual workload on loaner or cloud hardware before committing capital, then manage procurement, facility coordination, and burn-in testing. Engagements typically start with a workload and facility assessment that produces a costed architecture within two to three weeks. **Cta:** Get a validated GPU architecture and procurement plan built from your actual workload benchmarks, not a generic tier. **Faq:** #### How do I score a mixture-of-experts model like DeepSeek or Kimi K2? **Answer:** Use active parameters per token, not total parameters. DeepSeek V3 has 671B total parameters but only about 37B active per token, which behaves closer to a mid-size dense model for memory bandwidth purposes even though the full weight set still needs to fit in aggregate VRAM across the cluster. Score the model size question based on that active parameter count. #### Can I mix GPU tiers in one deployment? **Answer:** Yes, and it is often the right answer. Many enterprises route simple extraction and classification work to a smaller L40S or A100 tier while reserving H100 or H200 capacity for the hardest reasoning tasks or highest-concurrency interactive workloads. This assessment scores a single workload profile; run it again for each distinct workload if your portfolio is mixed. #### What if my facility power score is much lower than my other scores? **Answer:** Facility power is frequently the binding constraint even when budget and workload point to a higher tier. In that case, plan the facility upgrade as its own project with its own timeline, typically 8-20 weeks for electrical and cooling work, and consider colocation as a bridge that lets you deploy the recommended tier without waiting on your own building. #### How often should I re-run this assessment? **Answer:** Re-run it whenever concurrency, model size targets, or budget change materially, and at minimum before every major procurement cycle. GPU pricing and available generations shift roughly every 12-18 months, and a tier that was flagship two years ago is often mid-tier today, which changes both cost and availability assumptions. ## Search terms NVIDIA GPU selector, which GPU for LLM inference, H100 vs A100 vs B200, GPU sizing for LLM workloads, on-prem AI GPU selection, GPU tier assessment, LLM hardware selection --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json