Why your vector index uses far more memory than the raw embeddings, and how to fix it
server ran out of memory while building or loading the HNSW index
Also appears as
- OOM killed while creating vector index
- vector database memory usage grows far beyond raw embedding size
- HNSW index will not fit in available RAM
Short answer
Vector index memory usage exceeds raw embedding size because HNSW stores a graph of neighbor connections on top of every vector, typically adding 1.5 to 3 times the raw vector size in overhead depending on the M parameter, while IVFFlat has lower overhead but worse recall at the same speed. Estimate memory as dimension times row count times 4 bytes for raw vectors, then add graph overhead, and consider quantization if the total does not fit in available RAM.
Affects: Any HNSW or IVFFlat vector index once the corpus reaches millions of vectors, regardless of vector database vendor.
Fastest path to a sized-correctly index
- 1Calculate raw vector memory first: dimensions times number of vectors times 4 bytes for float32; this is the floor, not the total.
- 2Add HNSW graph overhead, which scales with the M parameter (default M=16 typically adds roughly 1.5-2x the raw vector size in graph edges and metadata).
- 3If the total exceeds available RAM, lower M to reduce graph size before considering more drastic changes.
- 4Consider quantized or lower-precision embeddings, such as int8 or binary quantization, if your vector database supports it, which can cut memory 4x or more with a modest recall tradeoff.
- 5If a single node cannot hold the full index in RAM, shard the collection across multiple nodes rather than letting the index spill to disk, since disk-resident HNSW graph traversal is dramatically slower.
How to confirm this is your problem
- The database process consumes several times more RAM than the raw embedding data size alone would suggest
- Building the vector index triggers an out-of-memory kill on a machine that has enough RAM for the raw data
- Query latency degrades sharply once the index no longer fits entirely in RAM and starts touching disk
- Memory usage keeps climbing as more documents are added, faster than the raw embedding size alone accounts for
- Reducing the embedding dimension or switching to a smaller embedding model measurably reduces total memory footprint
Root causes and fixes
HNSW graph overhead is not accounted for in capacity planning
HNSW builds a multi-layer navigable graph where each vector stores connections to M (commonly 16) neighbors per layer across multiple layers. This graph structure, plus per-node metadata, typically adds 1.5x to 3x the raw vector storage size on top of the vectors themselves, and teams that only budget for dimension times count times 4 bytes are routinely surprised when actual usage is far higher.
Fix: Budget for total memory as raw vector size multiplied by roughly 2-3x to account for graph overhead at typical M values, and re-measure actual usage on a representative subset before committing to a hardware size for the full corpus.
SELECT pg_size_pretty(pg_relation_size('items_embedding_idx'));The HNSW M parameter is set higher than necessary for the recall target
M controls how many neighbor connections each node maintains; higher M improves recall and query speed but increases both index size and build time roughly linearly. Many deployments copy a high M value from a tutorial or default without checking whether their actual recall requirements justify the extra memory cost.
Fix: Benchmark recall at a lower M, such as 8-12 instead of 16-32, against your actual query workload; if recall is still acceptable, the memory savings can be substantial at scale.
Embeddings are stored at full float32 precision when lower precision would suffice
Most embedding models output float32 vectors by default, but the marginal precision beyond 8-bit or even binary quantization often has minimal impact on retrieval quality for many use cases, especially when a reranker downstream corrects for any precision loss in the initial candidate retrieval.
Fix: Evaluate int8 or binary quantization support in your vector database and measure the recall impact against your evaluation set; quantization can reduce memory 4x (int8) or up to 32x (binary) with the tradeoff typically recovered by reranking.
The index does not fit in RAM and is spilling to disk
Once total memory demand exceeds physical RAM, the operating system or database engine has to page parts of the index to disk. HNSW graph traversal involves many small, effectively random-access reads across the graph, which is dramatically slower on disk than in RAM, so this shows up as both high memory pressure and degraded query latency simultaneously.
Fix: Either add RAM to the instance, shard the collection across multiple nodes so each holds a subset that fits in RAM, or reduce the effective index size through quantization or a lower M value.
Diagnostic commands
Measure actual index size on disk/in memory
SELECT pg_size_pretty(pg_relation_size('items_embedding_idx'));Compare this against the raw vector size calculation (dimensions times rows times 4 bytes); the ratio tells you exactly how much overhead the graph structure is adding for your current M setting.
Check system memory pressure and swap activity during query load
free -h && vmstat 1 5
Significant swap usage or memory pressure during query serving confirms the index does not comfortably fit in RAM, which explains both memory alerts and latency degradation together.
Benchmark recall and memory at two different M values on a subset
CREATE INDEX ON items_subset USING hnsw (embedding vector_cosine_ops) WITH (m = 8);
Compare index size and recall@k against the same subset indexed at m = 16 or higher; this quantifies the specific memory/recall tradeoff for your own data rather than relying on generic defaults.
Stopping it from happening again
- Include HNSW graph overhead, roughly 2-3x raw vector size at typical M, in capacity planning from the start, not just raw vector size.
- Benchmark recall at the lowest M value that still meets your accuracy target before committing to a memory budget.
- Evaluate quantization options early in the project, especially for corpora expected to grow into the tens of millions of vectors.
- Monitor memory headroom continuously as the corpus grows, since index memory grows with document count, not just initial load size.
- Plan sharding or horizontal scaling before the index approaches available RAM, rather than reacting to an outage.
When this becomes an architecture problem
If a single instance genuinely cannot hold the required index in RAM even after tuning M and applying quantization, either because the corpus is very large or the required recall does not tolerate aggressive compression, this becomes a distributed vector database architecture decision rather than a configuration tweak, and is worth sizing properly against your growth projections before committing hardware.
Frequently asked questions
How much RAM does a 10 million row, 1024-dimension HNSW index need?
Raw vectors alone are roughly 10,000,000 times 1024 times 4 bytes, about 40GB. With typical HNSW graph overhead at default M, expect total memory closer to 80-120GB, so plan hardware around the graph-inclusive figure, not just the raw vector calculation.
Does IVFFlat use less memory than HNSW?
Yes, IVFFlat generally has lower memory overhead than HNSW at comparable recall because it does not maintain a multi-layer graph, but it typically has slower query times and requires retraining cluster centroids as data grows, which is a meaningful operational tradeoff against its memory savings.
Is quantization safe for production RAG?
For most use cases, yes, especially when paired with a reranker that operates on the original text or a small candidate set. Measure recall impact on your own evaluation set before rolling out broadly, since tolerance for precision loss varies by domain and query type.
Why does memory usage keep growing even though we deleted old documents?
Deleted rows in most vector databases are not immediately reclaimed from the index structure; both the underlying table and the HNSW graph may retain stale entries until a vacuum, index rebuild, or compaction runs, so periodic maintenance is necessary to actually shrink the index after large deletions.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Vector Database Sizing Calculator
Convert vector count, embedding dimensions, and precision into a real storage footprint, including index overhead and replica factor, before you pick a vector database.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolDocument Ingestion Pipeline Estimator
Estimate total pipeline time from document count, OCR share, and embedding throughput, so ingestion timelines stop being a guess in the project plan.
Related problems
pgvector similarity queries are slow
pgvector queries are almost always slow because of an index and operator class mismatch (an index built for one distance function while queries use a different operator), a missing index entirely so Postgres falls back to a sequential scan, or search-time parameters (ef_search, probes) set too low. Confirm EXPLAIN ANALYZE shows an index scan, match the operator class to your distance function, and tune maintenance_work_mem before building large HNSW indexes.
Embedding dimension mismatch after switching models
Different embedding models produce vectors of different fixed dimensions, so swapping models without re-embedding the entire corpus produces a hard dimension mismatch error or, worse, silently meaningless similarity scores if the column is resized without re-indexing. There is no shortcut: changing the embedding model requires re-embedding every document and rebuilding the vector index from scratch.
Vector database connection errors under load
Vector database connection errors under production load are almost always pool exhaustion, either too many application processes each opening their own connections, or a pool sized for development traffic rather than real concurrent RAG query volume, not an actual network or database outage. Use a connection pooler sized for your real concurrency, add retry logic with exponential backoff for transient failures, and separate TLS/auth failures from timeout and pool errors since they need different fixes.
Guidepgvector vs Dedicated Vector Databases: An Honest Comparison
pgvector vs Milvus, Qdrant, and Weaviate for enterprise RAG: real tradeoffs on scale, latency, operational overhead, and when Postgres is genuinely enough.
GuideEnterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.