KV Cache Calculator: Size Your KV Cache Memory in GB
This free KV cache calculator (also searched as a KV cache size calculator) applies the standard attention cache formula, two times layers times KV heads times head dimension times precision times context length, to tell you exactly how much GPU memory your model's attention cache consumes. It is built for platform engineers who need to know why VRAM disappears after weights load and how many concurrent users a given GPU configuration can actually support. Enter your model's architecture parameters and target context length, and the tool returns cache size per token, per sequence, and the maximum concurrent sequences your available VRAM supports.
Your numbers
Depth of the model. A 70B-class dense model commonly has 80 layers; smaller models have fewer.
Grouped-query attention shares KV heads across query heads, which is why this is often far fewer than total attention heads.
Size of each attention head, commonly 64-128 across current open-weight model families.
FP8 KV cache quantization roughly halves cache memory with a small accuracy cost.
Maximum tokens (prompt plus generated) the cache must hold for one active sequence.
How many requests are actively decoding at once and each need their own KV cache slot.
Total VRAM minus model weights and runtime overhead, the space left for KV cache blocks.
Your results
Estimates use the standard 2 x layers x kv_heads x head_dim x precision formula per token. Real deployments rarely fill every sequence to maximum context, so PagedAttention typically supports higher practical concurrency than this worst-case figure.
Get your full KV cache capacity report
We will email you a personalized cache sizing breakdown across context lengths and concurrency targets, and a Netray inference specialist will follow up with tuning recommendations.
Optional. The tool and every export work without this. No spam.
Why KV cache dominates memory at scale
Model weights are a fixed, one-time memory cost. KV cache is not: it grows linearly with context length and with the number of concurrent sequences, which means a chat application with long conversations and many simultaneous users can consume far more memory in cache than the model weights themselves ever did. Grouped-query attention, used by essentially every current open-weight model family, shrinks this cost dramatically by sharing key/value projections across multiple query heads, which is why the KV head count in this calculator is often 4-16 rather than the 32-64 total attention heads a model might have.
- The formula counts 2x for storing both keys and values separately at every layer.
- Grouped-query attention (GQA) cuts KV cache size by using far fewer KV heads than query heads, often 4-8x fewer.
- Cache grows linearly with context length, so doubling your context window doubles cache memory per sequence.
- FP8 KV cache quantization roughly halves cache memory with typically small quality impact on long-context recall.
Interpreting per-sequence and total cache size
Per-sequence cache size tells you the worst case for one fully-populated context, which matters for setting a hard concurrency ceiling. Total cache size at your target batch size tells you what to actually provision for. The gap between these two numbers and your available VRAM is your real headroom: if total cache exceeds available memory, requests will queue, get rejected, or in poorly configured deployments crash the server outright. PagedAttention, the technique vLLM introduced, allocates cache in fixed-size blocks rather than reserving worst-case memory per sequence up front, which is why real deployments often support meaningfully higher concurrency than a naive worst-case calculation implies.
Levers for reducing KV cache pressure
When max concurrent sequences comes back too low for your traffic target, you have several real options beyond just buying more VRAM. Reducing default context length for use cases that do not need it is the highest-leverage lever, since cache scales linearly with it. FP8 KV cache quantization is close to a free win on supporting hardware. Sliding-window attention variants and cache eviction policies for older tokens in long conversations trade some long-range recall for materially lower steady-state memory.
- Cap default context length per use case rather than always reserving the model's maximum window.
- Enable FP8 or INT8 KV cache quantization where your serving engine and hardware support it.
- Use PagedAttention-style block allocation instead of static per-sequence reservation to raise effective concurrency.
- Summarize or truncate long conversation history rather than replaying full context on every turn.
How Netray sizes serving capacity around KV cache
Netray designs on-prem inference platforms where getting concurrency right the first time matters, because ITAR and CMMC environments cannot simply burst to a public cloud API when capacity runs short. We model KV cache pressure against your real conversation length and concurrency patterns, not synthetic benchmarks, and tune context policies and quantization to hit your target user count on the hardware you have. Engagements typically start with a capacity model built from your actual usage logs.
Frequently Asked Questions
How do I calculate KV cache size?
Multiply 2 (keys and values) by layers, KV heads, head dimension and bytes per value to get cache bytes per token, then multiply by context length for one sequence and by concurrent sequences for the batch. Example: 80 layers, 8 KV heads, head dimension 128 and FP16 (2 bytes) gives 327,680 bytes, about 320 KiB, per token, so an 8,192 token sequence needs about 2.7 GB. The calculator above does this and also shows how many sequences fit in your free VRAM.
Why does grouped-query attention matter so much for memory?
Standard multi-head attention stores a separate key and value projection for every attention head, which scales cache size directly with head count. Grouped-query attention shares KV projections across groups of query heads, so a model with 32 query heads might only maintain 8 KV heads, cutting cache memory by 4x with minimal quality impact. Nearly every current open-weight model family uses GQA specifically because unconstrained KV cache growth made long-context serving impractical.
Does PagedAttention actually change the total memory required?
Not the theoretical maximum, but it dramatically improves practical utilization. Without it, serving engines reserve worst-case contiguous memory per sequence up front, wasting large amounts of VRAM on sequences that end up shorter than their reservation. PagedAttention allocates cache in small fixed-size blocks on demand, similar to virtual memory paging in an operating system, which lets a GPU serve substantially more real-world concurrent sequences from the same VRAM budget.
How much does long context cost in KV cache memory?
Linearly and often surprisingly. For a 70B-class model with 80 layers and 8 KV heads at FP16, one sequence at 8K context needs roughly 2.7 GB of cache; the same sequence at 128K context needs roughly 43 GB, more memory than many entire GPUs. This is why long-context deployments either require far fewer concurrent users per GPU, aggressive KV quantization, or both.
Should I quantize KV cache separately from model weights?
Yes, they are independent decisions. You can serve FP16 weights with an FP8 KV cache, or a quantized model with a full-precision cache, depending on which constraint binds harder for your workload. KV cache quantization tends to have a smaller and more predictable quality impact than weight quantization because attention scores are more tolerant of reduced precision than the weights that generate them, but validate on long-context tasks specifically since that is where degradation shows up first.
Get a KV cache and concurrency model built from your real traffic patterns, not a worst-case estimate.
Related Tools
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AIConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
On-Prem AIModel Context Window Planner
Allocate a fixed context window across system prompt, retrieved chunks, conversation history, and reserved output, then see exactly how much retrieval headroom is left.
On-Prem AIMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- GPU Sizing Calculator for LLM Inference
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- LLM API vs Self-Hosted Cost Calculator
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator
Go Deeper
KV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
On-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Keep exploring
Free tools
- On-Prem GPU Sizing Calculator for ERP AIWork out how much GPU memory and how many cards you need to serve an open-weight model to your ERP users on-prem.
- Llama 4 Hardware Requirements CalculatorEstimate VRAM, GPU count, and capital cost to run Llama 4 Scout or Maverick on your own hardware, accounting for full mixture-of-experts weight loading and KV cache growth.
- Oracle Essbase Index, Data File and Data Cache Sizing CalculatorEnter the total size of your .ind and .pag files and available memory to get recommended index cache, data file cache and data cache settings for a BSO database.
Guides
- KV Cache Optimization: Prefix Caching and Chunked Prefill in ProductionKV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
- ERP Caching Strategies for Application PerformanceImplement caching strategies for ERP application performance. Covers in-memory caching, distributed cache, cache invalidation, and ERP-specific caching patterns.
- AI Agent Architecture Design for Enterprise ERP SystemsDesign AI agent architectures for ERP systems with multi-agent orchestration, tool-use patterns, memory management, and enterprise integration strategies.
Troubleshooting
- Why vLLM says there is no memory for the KV cache, and how to fix itvLLM fails to start with no memory for the KV cache when weights already consume most of the VRAM budget. Raise gpu-memory-utilization or cut max-model-len.
- Why vLLM says the model's max seq len is larger than the KV cache, and how to fix itThis vLLM error means the KV cache pool is too small for your context length. Raise gpu_memory_utilization, lower max_model_len, or add GPUs/quantize to fix it.