The on-prem AI math, with the parts vendors skip.

Hardware by model class (planning numbers)

Exact sizing depends on quantization, context length, and concurrency, which is why we benchmark before you buy. But for planning:

• 7-9B models, quantized: one 24-48GB GPU (workstation or single server class) serves a departmental workload
• 30-70B models: 2 to 4 datacenter-class GPUs for real concurrency; one high-memory GPU for light use
• Frontier-scale open models (100B+): multi-GPU nodes; justified only when benchmarks show smaller tuned models genuinely fail your task
• Embedding + reranking for RAG: near-free by comparison; runs beside the LLM or on CPU

Self-hosting vs API: where break-even actually sits

API pricing is linear forever; hardware is a step function that then runs at the cost of electricity and ops. The crossover depends on token volume: light experimentation stays cheaper on APIs, while sustained production workloads (RAG over busy teams, coding assistants, batch document processing) typically cross within the first year, after which self-hosting runs at a fraction of the API bill.

The variable that dominates the math is not the GPU price; it is whether your workload is steady. Bursty, occasional use favors APIs and hybrid routing. All-day-every-day use favors owned hardware emphatically. Our sizing engagement produces this curve for your actual traffic instead of a generic chart.

The three line items teams forget

First: evaluation and data engineering. The model is free; making it good at your task is the project, and it costs engineering weeks, not GPU dollars. Second: day-2 operations, meaning patching, model refreshes, and drift monitoring, at roughly a fraction of an FTE once the runbooks exist (we train your team or run it with you). Third: the failed-pilot tax of skipping benchmarking and buying hardware for a model that was never the right one. It is the most expensive item on this page and entirely avoidable.

Where the savings compound

Beyond the per-token math: fine-tuned small models cut token counts by needing less prompting; owned infrastructure means experimentation is free at the margin, so teams actually iterate; and there is no per-seat pricing, so rolling the assistant to 400 people costs the same electricity as 40. For regulated industries, add the category that matters most: workloads that could never use an API at any price become possible at all.