LLM Inference Cost Per Million Tokens Calculator
This free LLM inference cost per million tokens calculator computes the number that actually determines whether self-hosting beats API pricing: your true cost per token after accounting for real utilization, not the theoretical best case at 100% GPU saturation. It is built for finance-facing engineers and IT directors who need to defend a self-hosted infrastructure decision. Enter GPU hourly cost, cluster throughput, and your honestly-estimated utilization rate, and the tool returns cost per million tokens, monthly infrastructure spend, and monthly capacity.
Your numbers
Cloud rental rate, or owned hardware amortized over its useful life plus power and cooling, per GPU.
Total GPUs dedicated to this serving deployment.
Total cluster tokens per second under continuous batching at your typical traffic mix.
Share of time the cluster is actually serving traffic near full throughput, versus idle or lightly loaded. This is the number teams overestimate most.
Your results
Estimates only. Real GPU amortization schedules, power costs, and traffic patterns vary by deployment. Compare against actual metered usage before presenting a cost-per-token figure to finance.
Get your full cost-per-token analysis
We will email you a personalized cost breakdown with utilization scenarios and a comparison against your current API spend, and a Netray infrastructure specialist will follow up with an optimization plan.
No spam. Your results stay private. Unsubscribe anytime.
Why utilization is the number that changes everything
GPU clusters cost the same per hour whether they serve one token or a million, which means cost per token is entirely a function of how much of that fixed capacity actually gets used. A cluster running at 90% utilization delivers roughly four times the effective throughput per dollar of one running at 22%, even though both have identical hardware and identical theoretical peak throughput. Most teams size a cluster for peak load, then run it at 30-60% average utilization because traffic is not smooth throughout the day, and this gap is precisely why self-hosted cost projections built on peak-throughput assumptions consistently disappoint.
- Fixed hourly cost divided by actual tokens served, not theoretical maximum, is the only honest unit economic.
- Bursty enterprise traffic commonly averages 30-60% utilization even when peak-sized correctly.
- Multi-tenant serving, batching multiple internal applications onto shared infrastructure, is the single biggest utilization lever.
- Autoscaling that shuts down idle replicas trades some latency for dramatically better utilization on variable workloads.
Reading your result against API pricing
Compare the cost-per-million-tokens figure directly against your current API spend per million tokens, but be honest about what each side includes. The self-hosted number here covers compute only; add engineering time for platform operations, security hardening, and model updates before the comparison is fair. The API number is all-in but carries no infrastructure risk. For workloads under roughly $8,000-$15,000 per month of API spend, self-hosting rarely wins on pure cost once engineering overhead is counted, and it becomes attractive mainly when data residency or ITAR constraints make it a compliance requirement rather than a cost optimization.
Raising utilization without buying more hardware
Before concluding you need more GPUs, check whether existing capacity is actually saturated. Consolidating multiple internal applications onto shared serving infrastructure, rather than each team running its own underutilized cluster, is consistently the highest-leverage fix. Batch or asynchronous workloads that tolerate a delay, such as nightly document summarization, should be scheduled into off-peak hours where interactive traffic has left capacity unused, converting otherwise-wasted GPU-hours into productive throughput at zero marginal hardware cost.
- Consolidate separate team deployments onto shared multi-tenant infrastructure before adding hardware.
- Schedule batch and asynchronous workloads into off-peak hours to fill capacity gaps.
- Autoscale replica count with traffic rather than running peak-sized capacity around the clock.
- Route simple tasks to a smaller, faster model so expensive flagship-model capacity stays reserved for hard queries.
How Netray optimizes on-prem inference economics
Netray builds shared inference platforms for manufacturers running multiple AI workloads, ERP copilots, document assistants, and agent workflows, on consolidated infrastructure rather than siloed clusters per team. We measure real utilization from production traffic, right-size hardware against it, and implement request routing and autoscaling that keeps cost per token honest instead of theoretical. Engagements typically start with a utilization audit of your existing deployment before any hardware decision.
Frequently Asked Questions
What utilization rate should I assume for planning?
For a single-application deployment with typical business-hours traffic, 30-50% is a realistic planning assumption before any optimization. Multi-tenant platforms serving several applications with staggered peak times commonly reach 55-75%. Anything above 80% sustained utilization usually means you are under-provisioned and should expect latency degradation during traffic spikes, not celebrate efficiency.
Why does this number differ so much from vendor benchmark cost-per-token figures?
Vendor benchmarks almost always measure at maximum sustained batch saturation, effectively 100% utilization, with an ideal traffic shape. Real production traffic is bursty, includes idle periods, and rarely sustains peak batch depth for hours at a time. This calculator deliberately uses your honest utilization estimate instead, which is why the resulting cost per token is typically 1.5 to 3 times higher than a headline benchmark figure, and why that higher number is the one you should trust.
Does owned hardware or cloud rental produce a lower hourly cost input?
It depends on utilization and time horizon, which is exactly why this calculator treats hourly cost as a single input regardless of source. Owned hardware amortized over three to four years often lands at $1.50-$3.50 per GPU-hour including power, well below cloud rental rates of $2-$6 per hour, but only if utilization stays reasonably high. At low utilization, cloud rental with aggressive autoscaling can beat owned hardware sitting mostly idle.
How does multi-tenancy improve cost per token specifically?
Multiple applications rarely peak at the same moment: an ERP copilot peaks during business hours, a document ingestion job peaks overnight, and a customer support assistant peaks around specific triggers. Serving all of them from shared infrastructure smooths the combined demand curve, raising average utilization well above what any single application achieves alone, which directly lowers the effective cost per token for every workload on that shared cluster.
Get a utilization audit and a real cost-per-token model built from your actual traffic, not a peak-throughput estimate.
Related Tools
GPU Cluster Utilization Calculator
Turn GPU capital, amortization, and operating cost into an effective cost per productive GPU hour, and find the utilization threshold where owning beats renting.
On-Prem AILLM Token Cost Calculator
Turn request volume, prompt length, and per-million token pricing into a defensible monthly and annual LLM budget, including the effect of prompt caching.
On-Prem AIOn-Prem AI ROI Calculator
Turn hours saved per employee into annual net benefit, payback months, and 3-year ROI for an on-prem AI investment.
Go Deeper
The LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
On-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.