# KV Cache Calculator: Size Your KV Cache Memory in GB

> Source: https://netray.co/tools/kv-cache-memory-calculator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.

## Category

on-prem-ai

## Tool

**Kind:** calculator

**Inputs:**

#### Transformer layers

**Key:** num_layers

**Type:** number

**Default:** 80

**Min:** 8

**Max:** 200

**Step:** 1

**Unit:** layers

**Help:** Depth of the model. A 70B-class dense model commonly has 80 layers; smaller models have fewer.

#### Key/value heads

**Key:** kv_heads

**Type:** number

**Default:** 8

**Min:** 1

**Max:** 128

**Step:** 1

**Unit:** heads

**Help:** Grouped-query attention shares KV heads across query heads, which is why this is often far fewer than total attention heads.

#### Head dimension

**Key:** head_dim

**Type:** number

**Default:** 128

**Min:** 32

**Max:** 256

**Step:** 8

**Unit:** dims

**Help:** Size of each attention head, commonly 64-128 across current open-weight model families.

#### KV cache precision

**Key:** precision_bytes

**Type:** select

**Default:** 2

**Unit:** bytes/value

**Help:** FP8 KV cache quantization roughly halves cache memory with a small accuracy cost.

**Options:**

##### FP16 / BF16 (2 bytes)

**Value:** 2

##### FP8 quantized KV cache (1 byte)

**Value:** 1

#### Context length per sequence

**Key:** context_length

**Type:** number

**Default:** 8192

**Min:** 512

**Max:** 200000

**Step:** 512

**Unit:** tokens

**Help:** Maximum tokens (prompt plus generated) the cache must hold for one active sequence.

#### Concurrent sequences

**Key:** batch_size

**Type:** number

**Default:** 16

**Min:** 1

**Max:** 512

**Step:** 1

**Unit:** sequences

**Help:** How many requests are actively decoding at once and each need their own KV cache slot.

#### GPU VRAM available for KV cache

**Key:** gpu_vram_available_gb

**Type:** number

**Default:** 40

**Min:** 1

**Max:** 500

**Step:** 1

**Unit:** GB

**Help:** Total VRAM minus model weights and runtime overhead, the space left for KV cache blocks.

**Outputs:**

#### KV cache per token

**Key:** bytes_per_token_kb

**Expr:** 2 * num_layers * kv_heads * head_dim * precision_bytes / 1024

**Format:** number

**Description:** Kilobytes of cache consumed by each token position across all layers and KV heads.

#### KV cache per sequence

**Key:** kv_cache_per_sequence_gb

**Expr:** 2 * num_layers * kv_heads * head_dim * precision_bytes * context_length / 1000000000

**Format:** number

**Description:** Gigabytes of cache one fully-populated sequence at your context length consumes.

#### Total KV cache for this batch

**Key:** total_kv_cache_gb

**Expr:** kv_cache_per_sequence_gb * batch_size

**Format:** number

**Description:** Combined cache footprint across all concurrent sequences at full context length.

**Primary:** true

#### Max sequences your VRAM supports

**Key:** max_concurrent_sequences

**Expr:** floor(gpu_vram_available_gb / max(kv_cache_per_sequence_gb, 0.001))

**Format:** number

**Description:** The concurrency ceiling before you run out of memory, at full context length per sequence.

**Primary:** true

**Disclaimer:** Estimates use the standard 2 x layers x kv_heads x head_dim x precision formula per token. Real deployments rarely fill every sequence to maximum context, so PagedAttention typically supports higher practical concurrency than this worst-case figure.

## Content

**Intro:** This free KV cache calculator (also searched as a KV cache size calculator) applies the standard attention cache formula, two times layers times KV heads times head dimension times precision times context length, to tell you exactly how much GPU memory your model's attention cache consumes. It is built for platform engineers who need to know why VRAM disappears after weights load and how many concurrent users a given GPU configuration can actually support. Enter your model's architecture parameters and target context length, and the tool returns cache size per token, per sequence, and the maximum concurrent sequences your available VRAM supports.

**Sections:**

#### Why KV cache dominates memory at scale

**Body:** Model weights are a fixed, one-time memory cost. KV cache is not: it grows linearly with context length and with the number of concurrent sequences, which means a chat application with long conversations and many simultaneous users can consume far more memory in cache than the model weights themselves ever did. Grouped-query attention, used by essentially every current open-weight model family, shrinks this cost dramatically by sharing key/value projections across multiple query heads, which is why the KV head count in this calculator is often 4-16 rather than the 32-64 total attention heads a model might have.

**Bullets:**

- The formula counts 2x for storing both keys and values separately at every layer.
- Grouped-query attention (GQA) cuts KV cache size by using far fewer KV heads than query heads, often 4-8x fewer.
- Cache grows linearly with context length, so doubling your context window doubles cache memory per sequence.
- FP8 KV cache quantization roughly halves cache memory with typically small quality impact on long-context recall.

#### Interpreting per-sequence and total cache size

**Body:** Per-sequence cache size tells you the worst case for one fully-populated context, which matters for setting a hard concurrency ceiling. Total cache size at your target batch size tells you what to actually provision for. The gap between these two numbers and your available VRAM is your real headroom: if total cache exceeds available memory, requests will queue, get rejected, or in poorly configured deployments crash the server outright. PagedAttention, the technique vLLM introduced, allocates cache in fixed-size blocks rather than reserving worst-case memory per sequence up front, which is why real deployments often support meaningfully higher concurrency than a naive worst-case calculation implies.

#### Levers for reducing KV cache pressure

**Body:** When max concurrent sequences comes back too low for your traffic target, you have several real options beyond just buying more VRAM. Reducing default context length for use cases that do not need it is the highest-leverage lever, since cache scales linearly with it. FP8 KV cache quantization is close to a free win on supporting hardware. Sliding-window attention variants and cache eviction policies for older tokens in long conversations trade some long-range recall for materially lower steady-state memory.

**Bullets:**

- Cap default context length per use case rather than always reserving the model's maximum window.
- Enable FP8 or INT8 KV cache quantization where your serving engine and hardware support it.
- Use PagedAttention-style block allocation instead of static per-sequence reservation to raise effective concurrency.
- Summarize or truncate long conversation history rather than replaying full context on every turn.

#### How Netray sizes serving capacity around KV cache

**Body:** Netray designs on-prem inference platforms where getting concurrency right the first time matters, because ITAR and CMMC environments cannot simply burst to a public cloud API when capacity runs short. We model KV cache pressure against your real conversation length and concurrency patterns, not synthetic benchmarks, and tune context policies and quantization to hit your target user count on the hardware you have. Engagements typically start with a capacity model built from your actual usage logs.

**Cta:** Get a KV cache and concurrency model built from your real traffic patterns, not a worst-case estimate.

**Faq:**

#### How do I calculate KV cache size?

**Answer:** Multiply 2 (keys and values) by layers, KV heads, head dimension and bytes per value to get cache bytes per token, then multiply by context length for one sequence and by concurrent sequences for the batch. Example: 80 layers, 8 KV heads, head dimension 128 and FP16 (2 bytes) gives 327,680 bytes, about 320 KiB, per token, so an 8,192 token sequence needs about 2.7 GB. The calculator above does this and also shows how many sequences fit in your free VRAM.

#### Why does grouped-query attention matter so much for memory?

**Answer:** Standard multi-head attention stores a separate key and value projection for every attention head, which scales cache size directly with head count. Grouped-query attention shares KV projections across groups of query heads, so a model with 32 query heads might only maintain 8 KV heads, cutting cache memory by 4x with minimal quality impact. Nearly every current open-weight model family uses GQA specifically because unconstrained KV cache growth made long-context serving impractical.

#### Does PagedAttention actually change the total memory required?

**Answer:** Not the theoretical maximum, but it dramatically improves practical utilization. Without it, serving engines reserve worst-case contiguous memory per sequence up front, wasting large amounts of VRAM on sequences that end up shorter than their reservation. PagedAttention allocates cache in small fixed-size blocks on demand, similar to virtual memory paging in an operating system, which lets a GPU serve substantially more real-world concurrent sequences from the same VRAM budget.

#### How much does long context cost in KV cache memory?

**Answer:** Linearly and often surprisingly. For a 70B-class model with 80 layers and 8 KV heads at FP16, one sequence at 8K context needs roughly 2.7 GB of cache; the same sequence at 128K context needs roughly 43 GB, more memory than many entire GPUs. This is why long-context deployments either require far fewer concurrent users per GPU, aggressive KV quantization, or both.

#### Should I quantize KV cache separately from model weights?

**Answer:** Yes, they are independent decisions. You can serve FP16 weights with an FP8 KV cache, or a quantized model with a full-precision cache, depending on which constraint binds harder for your workload. KV cache quantization tends to have a smaller and more predictable quality impact than weight quantization because attention scores are more tolerant of reduced precision than the weights that generate them, but validate on long-context tasks specifically since that is where degradation shows up first.

## Search terms

KV cache memory calculator, KV cache calculator, KV cache size calculator, kvcache calculator, KV cache size formula, attention cache GPU memory, LLM context length memory, grouped query attention memory, LLM serving capacity calculator, paged attention memory sizing, GPU VRAM for LLM inference

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json