LLM Model Routing Cost Savings Calculator
Most ERP questions are simple lookups, and a small model answers them well. Routing the easy traffic to a cheap model and escalating failures to a larger one can cut cost, but only if escalations and router overhead are counted.
Your numbers
Share of small-model answers that fail a check and are retried on the large model.
Classifier or embedding call used to decide the route.
Your results
Cost only. Routing quality is not modeled: measure answer quality per route on your own questions before shifting traffic.
Monthly cost vs share routed to the small model
Sweeps Share routed to small model from 0 to 100, holding your other inputs fixed.
Benchmark routing on your questions
Optional: we can test small and large models on a sample of your ERP questions.
Optional. The tool and every export work without this. No spam.
How to use it
Enter your monthly volume and average token counts from gateway logs. Enter each model's price per million tokens, whether from an API provider or your own cost per million tokens on owned GPUs. Set the share your router sends to the small model, and the share of those answers that fail a verification check and are retried on the large model. Include the cost of the routing step itself.
The formula
Per-request cost = (input tokens x input price + output tokens x output price) / 1,000,000. Routed cost = requests x [ small share x (small cost + escalation x large cost) + (1 - small share) x large cost ] + router cost. The escalation term is why a high small share is not automatically cheap: every failed attempt is paid for twice.
- Escalation rate depends on the check you use.
- Retries also add latency.
- Router must be far cheaper than the savings.
Reading the result
If saving is under about ten percent, routing adds complexity for little gain and a single mid-size model may be simpler to run. Savings are larger when the price gap between models is wide and your traffic is dominated by lookups. Validate with a labeled set of real questions per route. A small model that fails silently on aggregations or multi-table joins can cost far more in wrong answers than it saves in tokens.
Frequently Asked Questions
How do I set the escalation rate?
Run a labeled sample of real questions through the small model with your verification check, then count how many fail. Use that fraction, and re-measure after prompt or model changes.
Does this work for self-hosted models?
Yes. Replace the prices with your cost per million tokens, which is GPU cost per hour divided by tokens served per hour at realistic utilization.
What about latency?
Escalated requests take two model calls. Small-model answers are usually faster, so average latency often improves but the tail gets longer.
When is routing not worth it?
When the saving is small, traffic is low, or accuracy risk on the cheap path is high. A single well-chosen model is simpler to operate and audit.
Netray runs open-weight models of several sizes on customer GPUs and can help design routing with verification. Ask us if you want it benchmarked on your ERP questions.
Related Tools
LLM Response Cache Token Savings Calculator
See how much a result cache and prompt prefix caching cut your monthly LLM API bill when many ERP questions repeat.
On-Prem AILLM API vs Self-Hosted Cost Calculator
Model your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
On-Prem AIAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
On-Prem AIAI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- GPU Sizing Calculator for LLM Inference
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator
- On-Prem AI Deployment Checklist
Keep exploring
Free tools
- Model Distillation Savings CalculatorCompare the ongoing cost of calling a large teacher model API against the one-time cost of distilling and self-hosting a small student model.
- LLM API vs Self-Hosted Cost CalculatorModel your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
- LLM Quantization Memory Savings CalculatorCompare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
ERP fixes and how-tos
- How to replace paper travelers with a digital travelerHow to replace paper travelers with a digital traveler: ERP routing as source, step data capture, signoffs, revision control, offline use and audit records.
- JDE Work Order Routing: The F3112 Routing File and How It Attaches to a Work OrderHow JD Edwards work order routing works: the F3112 routing file, how the F3003 master routing copies onto a work order, setup steps and SQL to inspect it.
- Private AI for SAP Business One: Service Layer, a read replica and a local modelAdd a private AI assistant to SAP Business One: Service Layer reads, SQL Server or HANA replica, document index, local LLM, permissions and approvals.
Guides
- Small Language Models vs LLMs: Choosing the Right Size for Enterprise WorkloadsSmall language models vs LLMs for enterprise workloads: accuracy, cost, GPU footprint, and when a 7B model beats a frontier model for ERP automation.
- Model Distillation for Production: Shrinking Models Without Losing QualityModel distillation for production: teacher-student data generation, sequence vs logit distillation, and real serving cost savings by model size.
- SyteLine Routing Setup and Configuration Best PracticesConfigure SyteLine routings. Operation sequences, setup/run times, work center assignment, and routing optimization for production scheduling accuracy.
AI on your ERP
- AI for SAP Business One, Without Sending Your Books to a Public APIAdd AI to SAP Business One without sending customer data or financials to a public API. Service Layer and DI API integration, sized for an SMB budget.
- AI agents for NetSuite manufacturing operationsAI agents for NetSuite manufacturing: WIP tracking, routing exceptions, and work order status grounded in SuiteQL, with human approval on anything that writes back.
Troubleshooting
- Why a model is too large for a single GPU, and how to actually fit itA model too large for a single GPU needs quantization, tensor parallelism, or a smaller model, not a configuration tweak. Here is how to choose correctly.
- Why you get CUDA out of memory just loading a model, and how to actually fix itCUDA out of memory when loading an LLM means weights alone exceed your GPU's VRAM. Quantize to 4-bit, add tensor parallelism, or size hardware to fit.