# Why your vector index uses far more memory than the raw embeddings, and how to fix it

> Source: https://netray.co/troubleshoot/vector-index-memory-usage-high
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Vector index memory usage exceeds raw embedding size because HNSW stores a graph of neighbor connections on top of every vector, typically adding 1.5 to 3 times the raw vector size in overhead depending on the M parameter, while IVFFlat has lower overhead but worse recall at the same speed. Estimate memory as dimension times row count times 4 bytes for raw vectors, then add graph overhead, and consider quantization if the total does not fit in available RAM.

## Error Title

Vector index memory usage is too high (HNSW blowing up RAM)

## Error Signature

server ran out of memory while building or loading the HNSW index

## Alternate Signatures

- OOM killed while creating vector index
- vector database memory usage grows far beyond raw embedding size
- HNSW index will not fit in available RAM

## Category

rag-retrieval

## Stack

- vector-db

## Affects

Any HNSW or IVFFlat vector index once the corpus reaches millions of vectors, regardless of vector database vendor.

## Quick Fix

### Fastest path to a sized-correctly index

**Steps:**

- Calculate raw vector memory first: dimensions times number of vectors times 4 bytes for float32; this is the floor, not the total.
- Add HNSW graph overhead, which scales with the M parameter (default M=16 typically adds roughly 1.5-2x the raw vector size in graph edges and metadata).
- If the total exceeds available RAM, lower M to reduce graph size before considering more drastic changes.
- Consider quantized or lower-precision embeddings, such as int8 or binary quantization, if your vector database supports it, which can cut memory 4x or more with a modest recall tradeoff.
- If a single node cannot hold the full index in RAM, shard the collection across multiple nodes rather than letting the index spill to disk, since disk-resident HNSW graph traversal is dramatically slower.

## Symptoms

- The database process consumes several times more RAM than the raw embedding data size alone would suggest
- Building the vector index triggers an out-of-memory kill on a machine that has enough RAM for the raw data
- Query latency degrades sharply once the index no longer fits entirely in RAM and starts touching disk
- Memory usage keeps climbing as more documents are added, faster than the raw embedding size alone accounts for
- Reducing the embedding dimension or switching to a smaller embedding model measurably reduces total memory footprint

## Causes

### HNSW graph overhead is not accounted for in capacity planning

**Likelihood:** most-common

**Explanation:** HNSW builds a multi-layer navigable graph where each vector stores connections to M (commonly 16) neighbors per layer across multiple layers. This graph structure, plus per-node metadata, typically adds 1.5x to 3x the raw vector storage size on top of the vectors themselves, and teams that only budget for dimension times count times 4 bytes are routinely surprised when actual usage is far higher.

**Fix:** Budget for total memory as raw vector size multiplied by roughly 2-3x to account for graph overhead at typical M values, and re-measure actual usage on a representative subset before committing to a hardware size for the full corpus.

**Commands:**

- SELECT pg_size_pretty(pg_relation_size('items_embedding_idx'));

### The HNSW M parameter is set higher than necessary for the recall target

**Likelihood:** common

**Explanation:** M controls how many neighbor connections each node maintains; higher M improves recall and query speed but increases both index size and build time roughly linearly. Many deployments copy a high M value from a tutorial or default without checking whether their actual recall requirements justify the extra memory cost.

**Fix:** Benchmark recall at a lower M, such as 8-12 instead of 16-32, against your actual query workload; if recall is still acceptable, the memory savings can be substantial at scale.

### Embeddings are stored at full float32 precision when lower precision would suffice

**Likelihood:** occasional

**Explanation:** Most embedding models output float32 vectors by default, but the marginal precision beyond 8-bit or even binary quantization often has minimal impact on retrieval quality for many use cases, especially when a reranker downstream corrects for any precision loss in the initial candidate retrieval.

**Fix:** Evaluate int8 or binary quantization support in your vector database and measure the recall impact against your evaluation set; quantization can reduce memory 4x (int8) or up to 32x (binary) with the tradeoff typically recovered by reranking.

### The index does not fit in RAM and is spilling to disk

**Likelihood:** occasional

**Explanation:** Once total memory demand exceeds physical RAM, the operating system or database engine has to page parts of the index to disk. HNSW graph traversal involves many small, effectively random-access reads across the graph, which is dramatically slower on disk than in RAM, so this shows up as both high memory pressure and degraded query latency simultaneously.

**Fix:** Either add RAM to the instance, shard the collection across multiple nodes so each holds a subset that fits in RAM, or reduce the effective index size through quantization or a lower M value.

## Diagnostics

### Measure actual index size on disk/in memory

**Command:** SELECT pg_size_pretty(pg_relation_size('items_embedding_idx'));

**Interpretation:** Compare this against the raw vector size calculation (dimensions times rows times 4 bytes); the ratio tells you exactly how much overhead the graph structure is adding for your current M setting.

### Check system memory pressure and swap activity during query load

**Command:** free -h && vmstat 1 5

**Interpretation:** Significant swap usage or memory pressure during query serving confirms the index does not comfortably fit in RAM, which explains both memory alerts and latency degradation together.

### Benchmark recall and memory at two different M values on a subset

**Command:** CREATE INDEX ON items_subset USING hnsw (embedding vector_cosine_ops) WITH (m = 8);

**Interpretation:** Compare index size and recall@k against the same subset indexed at m = 16 or higher; this quantifies the specific memory/recall tradeoff for your own data rather than relying on generic defaults.

## Prevention

- Include HNSW graph overhead, roughly 2-3x raw vector size at typical M, in capacity planning from the start, not just raw vector size.
- Benchmark recall at the lowest M value that still meets your accuracy target before committing to a memory budget.
- Evaluate quantization options early in the project, especially for corpora expected to grow into the tens of millions of vectors.
- Monitor memory headroom continuously as the corpus grows, since index memory grows with document count, not just initial load size.
- Plan sharding or horizontal scaling before the index approaches available RAM, rather than reacting to an outage.

## When To Escalate

If a single instance genuinely cannot hold the required index in RAM even after tuning M and applying quantization, either because the corpus is very large or the required recall does not tolerate aggressive compression, this becomes a distributed vector database architecture decision rather than a configuration tweak, and is worth sizing properly against your growth projections before committing hardware.

## Faq

### How much RAM does a 10 million row, 1024-dimension HNSW index need?

**Answer:** Raw vectors alone are roughly 10,000,000 times 1024 times 4 bytes, about 40GB. With typical HNSW graph overhead at default M, expect total memory closer to 80-120GB, so plan hardware around the graph-inclusive figure, not just the raw vector calculation.

### Does IVFFlat use less memory than HNSW?

**Answer:** Yes, IVFFlat generally has lower memory overhead than HNSW at comparable recall because it does not maintain a multi-layer graph, but it typically has slower query times and requires retraining cluster centroids as data grows, which is a meaningful operational tradeoff against its memory savings.

### Is quantization safe for production RAG?

**Answer:** For most use cases, yes, especially when paired with a reranker that operates on the original text or a small candidate set. Measure recall impact on your own evaluation set before rolling out broadly, since tolerance for precision loss varies by domain and query type.

### Why does memory usage keep growing even though we deleted old documents?

**Answer:** Deleted rows in most vector databases are not immediately reclaimed from the index structure; both the underlying table and the HNSW graph may retain stale entries until a vacuum, index rebuild, or compaction runs, so periodic maintenance is necessary to actually shrink the index after large deletions.

## Search terms

HNSW memory usage, vector index out of memory, vector database RAM requirements, HNSW graph overhead, vector quantization RAG, pgvector memory, vector index sizing, HNSW M parameter memory

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json