LLM Model Routing Cost Savings Calculator
Most ERP questions are simple lookups, and a small model answers them well. Routing the easy traffic to a cheap model and escalating failures to a larger one can cut cost, but only if escalations and router overhead are counted.
Your numbers
Share of small-model answers that fail a check and are retried on the large model.
Classifier or embedding call used to decide the route.
Your results
Cost only. Routing quality is not modeled: measure answer quality per route on your own questions before shifting traffic.
Monthly cost vs share routed to the small model
Sweeps Share routed to small model from 0 to 100, holding your other inputs fixed.
Benchmark routing on your questions
Optional: we can test small and large models on a sample of your ERP questions.
Optional. The tool and every export work without this. No spam.
How to use it
Enter your monthly volume and average token counts from gateway logs. Enter each model's price per million tokens, whether from an API provider or your own cost per million tokens on owned GPUs. Set the share your router sends to the small model, and the share of those answers that fail a verification check and are retried on the large model. Include the cost of the routing step itself.
The formula
Per-request cost = (input tokens x input price + output tokens x output price) / 1,000,000. Routed cost = requests x [ small share x (small cost + escalation x large cost) + (1 - small share) x large cost ] + router cost. The escalation term is why a high small share is not automatically cheap: every failed attempt is paid for twice.
- Escalation rate depends on the check you use.
- Retries also add latency.
- Router must be far cheaper than the savings.
Reading the result
If saving is under about ten percent, routing adds complexity for little gain and a single mid-size model may be simpler to run. Savings are larger when the price gap between models is wide and your traffic is dominated by lookups. Validate with a labeled set of real questions per route. A small model that fails silently on aggregations or multi-table joins can cost far more in wrong answers than it saves in tokens.
Frequently Asked Questions
How do I set the escalation rate?
Run a labeled sample of real questions through the small model with your verification check, then count how many fail. Use that fraction, and re-measure after prompt or model changes.
Does this work for self-hosted models?
Yes. Replace the prices with your cost per million tokens, which is GPU cost per hour divided by tokens served per hour at realistic utilization.
What about latency?
Escalated requests take two model calls. Small-model answers are usually faster, so average latency often improves but the tail gets longer.
When is routing not worth it?
When the saving is small, traffic is low, or accuracy risk on the cheap path is high. A single well-chosen model is simpler to operate and audit.
Netray runs open-weight models of several sizes on customer GPUs and can help design routing with verification. Ask us if you want it benchmarked on your ERP questions.
Related Tools
LLM Response Cache Token Savings Calculator
See how much a result cache and prompt prefix caching cut your monthly LLM API bill when many ERP questions repeat.
On-Prem AILLM API vs Self-Hosted Cost Calculator
Model your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
On-Prem AIAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
On-Prem AIAI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- GPU Sizing Calculator for LLM Inference
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator
- On-Prem AI Deployment Checklist