# Why your cost per token is too high, and the biggest levers to fix it > Source: https://netray.co/troubleshoot/inference-cost-per-token-too-high > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Cost per token is dominated by GPU utilization far more than by hardware choice: an underutilized GPU serving one request at a time can cost more per token than a well-tuned smaller GPU serving at full continuous-batching concurrency. Before concluding self-hosting is not worth it, check utilization, whether the model is right-sized for the task, whether quantization and prefix/response caching are in use, and whether the comparison to an API is even apples-to-apples once amortization is accounted for. ## Error Title Cost per token for self-hosted LLM inference is higher than expected ## Error Signature cost per million tokens on our own gpus is not much cheaper than a hosted api ## Alternate Signatures - self-hosted inference is not saving money compared to an api provider - gpu costs are high relative to the actual tokens being served ## Category performance ## Stack - vllm - pytorch - cuda ## Affects Any organization running self-hosted LLM inference and comparing cost per token against a hosted API baseline ## Quick Fix ### Attack the biggest lever first **Steps:** Measure current GPU utilization under real traffic; if it is well under 60-70 percent, fixing batching and concurrency is almost always the highest-leverage first step, cheaper than any hardware change. Confirm you are not running a much larger model than the task actually requires; check whether a smaller or distilled model meets your quality bar at a fraction of the cost per token. Apply quantization (FP8, AWQ, or GPTQ) if not already in use, which increases effective throughput per GPU by reducing memory bandwidth pressure. Enable prefix caching and, where applicable, response caching for repeated or templated queries, since a cache hit costs essentially nothing compared to a fresh generation. Recompute the cost comparison against a hosted API including full amortization (hardware, power, staff time, redundancy) over the hardware's realistic lifetime, not just a monthly cloud GPU rental rate. ## Symptoms - Calculated cost per million tokens on owned or rented GPUs is close to or above API pricing - GPU utilization metrics are low even though the cost analysis assumes full utilization - The same task is being served by a much larger model than smaller alternatives that were never benchmarked - No caching is in place for prompts or contexts that repeat frequently across requests ## Causes ### Low GPU utilization inflates effective cost per token **Likelihood:** most-common **Explanation:** A GPU costs roughly the same per hour whether it is serving one request or fifty concurrently, so cost per token is essentially (GPU cost per hour) divided by (tokens generated per hour); utilization is the direct multiplier on the denominator. A GPU running at 20 percent effective utilization has roughly five times the cost per token of the same GPU at full, well-batched utilization, independent of any other factor. **Fix:** Fix continuous batching, concurrency, and max-num-seqs sizing (see the GPU utilization and batch size tuning pages) before considering any hardware or model change; this is almost always the single largest lever on cost per token. ### Model is larger than the task requires **Likelihood:** common **Explanation:** Cost scales strongly with model parameter count, both in memory bandwidth per token and in raw compute, so running a 70B model for a task a well-tuned 7B or 14B model could handle multiplies cost per token for no quality benefit actually being used. **Fix:** Benchmark smaller or distilled models against your actual task and quality bar rather than defaulting to the largest available model, and right-size deliberately. ### No quantization applied, leaving throughput per GPU below what the hardware supports **Likelihood:** common **Explanation:** Full-precision or even FP16 weights consume more memory bandwidth per token than a quantized model, directly limiting how much concurrency and throughput a given GPU can sustain; quantization (FP8, AWQ, GPTQ) shrinks that footprint and typically raises achievable throughput per GPU meaningfully with limited quality cost when applied carefully. **Fix:** Apply FP8 or a validated INT4 method (AWQ/GPTQ) and re-measure throughput per GPU, confirming quality holds using a task-specific eval, not just perplexity. ### No caching for repeated prompts, system prompts, or templated queries **Likelihood:** occasional **Explanation:** Workloads with a shared system prompt, RAG template, or frequently repeated queries pay full prefill and/or generation cost on every request even when a meaningful fraction of traffic is effectively duplicate or near-duplicate work, when prefix caching or a response cache could serve much of it far more cheaply. **Fix:** Enable prefix caching for shared prompt structure, and add an explicit response cache keyed on normalized query content for genuinely repeated queries where fresh generation is not required. ### Cost comparison against a hosted API does not account for full on-prem amortization **Likelihood:** rare **Explanation:** Comparing only a cloud GPU rental's hourly rate against API per-token pricing ignores that owned or reserved hardware amortizes differently over its lifetime, and also omits power, cooling, staff time, and redundancy costs that a fair comparison must include; conversely it can also omit that API pricing embeds a margin that scales with the same utilization dynamics. **Fix:** Build the cost comparison using a full total-cost-of-ownership model over a realistic hardware lifetime and expected utilization, not a spot-rental hourly rate compared to a per-token API price in isolation. ## Diagnostics ### Compute effective cost per token from real utilization data **Command:** nvidia-smi --query-gpu=utilization.gpu --format=csv -l 60 **Interpretation:** Multiply GPU hourly cost by hours run, divide by actual tokens generated in that window; comparing this real number against a theoretical full-utilization estimate quantifies exactly how much utilization is costing you. ### Check whether prefix or response caching is enabled **Command:** curl -s localhost:8000/v1/models | python -m json.tool **Interpretation:** If caching is off and a meaningful share of traffic shares a system prompt or repeats queries, this is low-effort, high-leverage cost savings being left on the table. ### Benchmark cost per token across model sizes for the actual task **Command:** python -c "print('compare tokens/sec and quality across model sizes on your eval set')" **Interpretation:** If a smaller model meets the task's quality bar at meaningfully lower cost per token, the largest lever may simply be right-sizing the model rather than any infrastructure change. ## Prevention - Track cost per token as a dashboard metric derived from real utilization, not a theoretical full-utilization estimate - Re-benchmark model size choices against task-specific evals periodically rather than assuming the largest model is always necessary - Enable prefix and response caching by default for any workload with repeated structure - Build TCO comparisons with full amortization whenever justifying self-hosting versus an API to stakeholders ## When To Escalate If utilization, model right-sizing, quantization, and caching are all already optimized and cost per token is still not competitive with alternatives at your actual traffic volume, the remaining lever is fleet-level architecture: consolidating workloads onto fewer, better-utilized GPUs, renegotiating hardware acquisition strategy, or reconsidering whether self-hosting is the right choice for that specific workload's volume, which is a cost-architecture decision rather than a serving optimization. ## Faq ### Is self-hosting always cheaper than an API per token? **Answer:** Only above a certain utilization and volume threshold; at low utilization or low volume, the fixed cost of owned or reserved GPU capacity can exceed API pricing, since APIs effectively pool utilization across many customers. The crossover point depends heavily on your actual utilization, not just list GPU pricing. ### What is the single biggest lever for reducing cost per token? **Answer:** In the overwhelming majority of cases, GPU utilization. Fixing continuous batching and concurrency to move from a lightly-loaded server to a well-batched one has a bigger cost-per-token impact than switching hardware or even quantization, because it directly multiplies the tokens produced per GPU-hour. ### Does quantization always reduce cost per token? **Answer:** It usually does by increasing achievable throughput per GPU, but the actual savings should be validated against a task-specific quality eval, since a quantization method that degrades output quality enough to require retries or a larger fallback model can erase the cost benefit. ## Search terms llm cost per token too high, self-hosted llm cost optimization, reduce inference cost gpu, cost per million tokens, gpu utilization cost efficiency, on-prem vs api cost llm --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json