LLM Response Cache Token Savings Calculator
ERP chat tends to repeat itself: the same open-order, stock and aging questions arrive every morning. Two caches can cut spend: a result cache that skips the model, and prefix caching that discounts the repeated instruction and schema text.
Your numbers
System prompt, schema context, retrieved rows and the question.
Share of queries answered from a stored answer with no model call.
Instructions and schema that are identical across calls.
Check your provider's current cached-input price. Self-hosted engines get this via prefix caching in the serving layer.
Your results
Prices change often and cache write fees are ignored. Cached answers can go stale: ERP data such as stock or order status needs short expiry or invalidation.
Monthly LLM cost vs result cache hit rate
Sweeps Result cache hit rate from 0 to 90, holding your other inputs fixed.
Review your LLM spend
Optional: share your gateway token logs and we will point out the biggest savings.
Optional. The tool and every export work without this. No spam.
How to use it
Pull a week of chat logs and count how many questions are repeats after normalization. That is your plausible hit rate. Enter token counts from your gateway logs, not estimates, since retrieved rows often dominate input. Set the static prefix share to the portion of the prompt that is byte-identical across calls. Then check your provider's cached-input price, or whether your self-hosted serving engine enables prefix caching.
The formula
Base cost = queries x (input tokens x input price + output tokens x output price) / 1,000,000. Result caching multiplies by (1 - hit rate). Prefix caching then discounts the static share of input on the queries that still reach the model: saving = remaining queries x input tokens x static share x discount x input price. Both effects stack, but the second applies only to misses.
- Hit rate is the biggest lever.
- Prefix caching needs identical leading text.
- Put the changing question last in the prompt.
Cautions for ERP data
A cached answer to 'what is on hand for part X' is wrong the moment stock moves. Key the cache by user role and question, set an expiry per data type, and never share cached answers across users with different row-level permissions. Dashboards of slow-changing data, policy questions and report definitions cache well; live inventory and order status do not. Measure answer correctness on cache hits, not only the hit rate.
Frequently Asked Questions
What is a realistic hit rate?
It varies widely with how repetitive your users are. Measure normalized repeats in a week of real logs before assuming a number; many deployments see a modest share, not a majority.
Is semantic caching safe for ERP?
Only with permission-aware keys and short expiry for live data. Similarity matching can return an answer to a nearly identical question with different filters, so test for that.
Does this apply to self-hosted models?
Result caching does. Prefix caching exists in engines such as vLLM, where it reduces compute rather than a bill, so use the discount as a rough proxy for GPU time saved.
Why is prefix saving only on misses?
Cache hits never call the model, so there is no prompt to discount. The prefix saving applies to the remaining calls.
Netray designs ERP chat with role-aware caching and visible queries. If you want to cut inference cost without leaking data across roles, talk to us.
Related Tools
LLM Model Routing Cost Savings Calculator
Work out blended cost per request when a router sends most traffic to a small model and escalates failures to a large one.
On-Prem AILLM API vs Self-Hosted Cost Calculator
Model your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
On-Prem AIRAG Context Window Budget Calculator
Allocate your context window across system prompt, retrieved chunks, and conversation history, then see window utilization and the real cost of every RAG query.
AI Agents & AutomationERP AI Copilot ROI Calculator
Turn user count, query volume, and time saved per question into a monthly savings, license cost offset, and payback period for an ERP AI copilot.
More AI Agents & Automation tools
See all- Business Process Automation ROI Calculator
- AI Agent Use Case Prioritizer
- Document Processing Automation Savings Calculator
- AP Invoice Automation ROI Calculator
- Order Entry Automation Savings Calculator
- Quote-to-Cash Cycle Time Calculator
- Manufacturing AI Readiness Assessment
- RPA vs AI Agent Selector
- ERP AI Maturity Assessment