# Private RAG Corpus Sizing Calculator: Chunks, Vectors, and Storage > Source: https://netray.co/tools/private-rag-corpus-sizing-calculator > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer Estimate chunk counts, vector index storage, raw text volume, and embedding compute time before you build a private retrieval system over your document estate. ## Category on-prem-ai ## Tool **Kind:** calculator **Inputs:** #### Documents in the corpus **Key:** document_count **Type:** number **Default:** 250000 **Min:** 100 **Max:** 50000000 **Step:** 1000 **Unit:** docs **Help:** Every file you intend to index: work instructions, specs, drawings text, service reports, and ERP attachments. #### Average pages per document **Key:** avg_pages_per_doc **Type:** number **Default:** 6 **Min:** 1 **Max:** 500 **Step:** 1 **Unit:** pages **Help:** Manufacturing corpora skew short, but quality manuals and contracts pull the average up quickly. #### Average tokens per page **Key:** tokens_per_page **Type:** number **Default:** 500 **Min:** 50 **Max:** 3000 **Step:** 25 **Unit:** tokens **Help:** A dense text page runs 500-700 tokens. Drawing sheets and forms run far lower after extraction. #### Chunk size **Key:** chunk_size_tokens **Type:** number **Default:** 512 **Min:** 128 **Max:** 4096 **Step:** 64 **Unit:** tokens **Help:** Target passage length for embedding. Structure-aware chunking usually lands between 400 and 1,000. #### Chunk overlap **Key:** chunk_overlap_pct **Type:** slider **Default:** 15 **Min:** 0 **Max:** 50 **Step:** 5 **Unit:** % **Help:** Overlap preserves context across boundaries but multiplies chunk count and storage. #### Embedding dimensions **Key:** embedding_dimensions **Type:** select **Default:** 1024 **Unit:** dims **Help:** Higher dimensions can improve retrieval but scale index size and memory linearly. **Options:** ##### 384 dimensions (compact models) **Value:** 384 ##### 768 dimensions (base models) **Value:** 768 ##### 1024 dimensions (large open models) **Value:** 1024 ##### 1536 dimensions (common API embeddings) **Value:** 1536 ##### 3072 dimensions (high-capacity models) **Value:** 3072 #### Index replication factor **Key:** index_replication_factor **Type:** select **Default:** 2 **Unit:** copies **Help:** Copies of the index kept for availability and query throughput. **Options:** ##### Single copy - development or pilot **Value:** 1 ##### Two copies - standard production HA **Value:** 2 ##### Three copies - high availability across sites **Value:** 3 **Outputs:** #### Total corpus size (million tokens) **Key:** total_tokens_m **Expr:** document_count * avg_pages_per_doc * tokens_per_page / 1000000 **Format:** number **Description:** Extracted text volume across the whole corpus, in millions of tokens. #### Total chunks to embed **Key:** chunk_count **Expr:** ceil(total_tokens_m * 1000000 / max(chunk_size_tokens * (1 - chunk_overlap_pct / 100), 1)) **Format:** number **Description:** Number of vectors your index will hold after chunking with overlap. **Primary:** true #### Vector index storage **Key:** vector_storage_gb **Expr:** chunk_count * embedding_dimensions * 4 * index_replication_factor / 1073741824 **Format:** number **Description:** Gigabytes of float32 vectors including replication, before index structure overhead. **Primary:** true #### Raw text and metadata storage **Key:** raw_text_storage_gb **Expr:** total_tokens_m * 1000000 * 4 / 1073741824 **Format:** number **Description:** Gigabytes of extracted chunk text held alongside the vectors for retrieval and citation. #### Initial embedding compute **Key:** embedding_compute_hours **Expr:** total_tokens_m * 1000000 / 50000 / 3600 **Format:** hours **Description:** Single-GPU hours to embed the corpus once, assuming about 50,000 tokens per second. **Disclaimer:** Planning estimates only. Actual index size varies with vector compression, quantization, and index type (HNSW graphs add 20-40% overhead). Validate on a representative 1% sample before provisioning production storage. ## Content **Intro:** This free private RAG corpus sizing calculator estimates the chunk count, vector storage, text storage, and embedding compute required to index an enterprise document estate, and it is built for solution architects and infrastructure leads planning on-prem or air-gapped retrieval systems. Enter document count, page and token density, chunk size and overlap, embedding dimensions, and replication factor, and the tool returns corpus token volume, total chunks, index storage, raw text storage, and initial embedding hours. Teams routinely discover that indexing is far cheaper than expected while re-indexing after a chunking change is far more expensive. **Sections:** #### How the sizing math works **Body:** The calculator converts documents into tokens, tokens into chunks, and chunks into stored vectors. With the defaults, 250,000 documents at six pages and 500 tokens per page gives 750 million tokens. Chunking at 512 tokens with 15% overlap means each chunk advances only about 435 tokens, producing roughly 1.72 million vectors. At 1,024 dimensions stored as float32, each vector occupies 4 KB, so a single index copy is about 6.6 GB and a two-copy production deployment is around 13 GB. Extracted chunk text adds roughly 2.8 GB. Embedding the corpus once on a single GPU at about 50,000 tokens per second takes roughly four hours. #### Sizing benchmarks for manufacturing corpora **Body:** These reference points come from indexing engineering and quality documentation for aerospace, defense, and electronics manufacturers. Document estates in these environments are larger and messier than most teams estimate before they inventory them, largely because content is scattered across ERP attachments, network shares, PLM vaults, and a decade of email exports nobody has audited. Run a file inventory before trusting a document count, and sample page density rather than assuming it, since scanned drawings and forms extract to far fewer tokens per page than prose. The figures below hold up well across the manufacturing corpora we have indexed, but validate them on your own sample. **Bullets:** - Vector storage is rarely the constraint: a million chunks at 1,024 dimensions is under 4 GB per copy. - HNSW and other graph index structures add 20-40% on top of raw vector size, so budget headroom. - Overlap above 20% inflates chunk count sharply while adding little retrieval benefit for structured technical content. - Metadata and permission attributes often consume more operational effort than the vectors themselves, because they drive access control. #### Interpreting the numbers before you build **Body:** Storage is almost never the problem. The real planning constraints are embedding throughput during initial load, the cost of re-indexing when you change chunking or embedding models, and the query-time memory required to hold the index hot. If embedding compute returns more than about twenty-four hours, plan a parallel load across multiple GPUs or a staged rollout by document class. Also decide your re-index strategy up front: changing embedding models invalidates every existing vector, so a corpus that takes four hours to build today takes four hours again every time you upgrade. Teams that ignore this end up frozen on a model they chose in week one. #### How Netray builds private retrieval systems **Body:** Netray builds RAG systems over manufacturing content that general-purpose tools handle poorly: revision-controlled work instructions, engineering specifications, supplier certifications, non-conformance reports, and decades of ERP transaction history in SyteLine, LN, or M3. We chunk on document structure rather than fixed character counts, carry ERP and PLM entitlements into the retrieval layer so users only see what they are authorized to see, and run the entire pipeline inside your network for ITAR and CMMC-constrained customers. Engagements typically start by indexing a representative subset and measuring retrieval quality against a golden question set before scaling to the full corpus. **Cta:** Get a sized retrieval architecture with chunking strategy, index design, and a phased indexing plan for your document estate. **Faq:** #### How much does chunk overlap really cost me? **Answer:** More than most teams expect, because it multiplies chunk count directly. Moving from zero to fifteen percent overlap raises chunk count by about eighteen percent, and thirty percent overlap raises it by roughly forty-three percent. That flows straight into storage, embedding compute, and query-time memory. For structured technical content with clean section boundaries, ten to fifteen percent is usually sufficient. Reserve higher overlap for continuous narrative prose where meaning genuinely spans boundaries. #### Do I need high-dimensional embeddings for technical documents? **Answer:** Usually not. Domain-specific technical vocabulary is quite separable, and 768 or 1,024 dimension open models frequently match 1,536 or 3,072 dimension alternatives on manufacturing corpora while using a third of the memory. The bigger quality lever is fine-tuning or selecting an embedding model that understands your part numbering, standards references, and abbreviations. Test two or three candidates on a labeled retrieval set before defaulting to the largest available option. #### How should I plan for re-indexing? **Answer:** Assume you will re-index at least twice in the first year, because chunking strategy and embedding model choice both improve as you learn from real queries. Keep extracted text separate from vectors so re-embedding does not require re-parsing source documents, which is usually the slowest step. Build the pipeline to run incrementally by document class so you can re-index a segment and compare retrieval quality before committing the whole corpus. ## Search terms RAG corpus sizing calculator, vector database storage calculator, embedding storage estimator, how many chunks in a RAG index, vector index sizing on-prem, embedding compute time estimate, enterprise RAG capacity planning --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json