Enterprise RAG Architecture: A Practitioner's Blueprint for 2026
Retrieval-augmented generation looks simple in a demo: embed some documents, retrieve the closest matches, hand them to a model, and get an answer. Production RAG is a six-stage pipeline where each stage has its own failure modes, and a system that is 95 percent accurate at each of six independent stages compounds down to roughly 74 percent end to end. The stages are ingestion, chunking, embedding, retrieval, reranking, and generation, closed by an evaluation loop that measures whether the whole thing actually works. Most enterprise RAG projects that stall in production did not fail because the model was weak. They failed because chunking was an afterthought, retrieval used vector similarity alone with no lexical fallback, and nobody built a way to measure faithfulness before shipping. This blueprint walks through each stage with the tradeoffs that matter at enterprise scale, and where a packaged pipeline like Netray's DataRay handles the plumbing for teams that want it running on their own infrastructure without building it from scratch.
The Six Stages of a Production RAG Pipeline
Ingestion pulls source documents, PDFs, wiki pages, ERP exports, scanned drawings, into a normalized text or structured form, and this is where most accuracy is won or lost before a single embedding is computed. Chunking splits that normalized content into retrievable units sized for your embedding model's context window. Embedding converts each chunk into a dense vector capturing semantic meaning. Retrieval searches an index of those vectors, often alongside a lexical index, for the chunks most relevant to a query. Reranking takes the top 50 to 100 retrieved candidates and reorders them with a more expensive, more accurate model before the top 5 to 10 go to the generator. Generation is the LLM call that synthesizes an answer from retrieved context, and it is the stage every team builds first and should optimize last, because a weak retrieval stage cannot be prompted around.
- Ingestion: parsing, OCR, metadata extraction, and access-control tagging before anything is chunked
- Chunking and embedding: splitting strategy plus the vector representation of each resulting chunk
- Retrieval and rerank: broad candidate generation followed by precision reordering of the top results
- Generation and eval: answer synthesis, then automated scoring against a golden query set
Chunking and Embedding: Decisions That Are Hard to Undo
Chunk size and strategy determine what the retriever can ever find, and re-chunking an existing production corpus is expensive enough that most teams live with the first decision for a year or more. Fixed-size token chunking with 10 to 20 percent overlap is the reasonable default for homogeneous prose, while structural and semantic chunking earn their extra complexity on mixed-format corpora like ERP documentation or engineering specs. Embedding model choice matters almost as much: dimension count trades storage and query latency against recall, and newer matryoshka-style embedding models let you truncate a 1024-dimension vector down to 256 dimensions for cheap first-pass retrieval and keep full precision for reranking. Pick an embedding model your reranker and eval harness can validate, not just the one with the highest public benchmark score, since domain vocabulary like part numbers and process names rarely matches benchmark data.
Retrieval, Hybrid Search, and Reranking in Production
Pure vector retrieval misses exact-match terms constantly: part numbers, error codes, and acronyms blur together in embedding space even when the surrounding prose is semantically distinct. Hybrid retrieval runs a lexical search like BM25 alongside vector search and merges the two ranked lists with reciprocal rank fusion, which is now close to a default best practice rather than an advanced technique. Reranking then takes the fused top 50 to 100 candidates and applies a cross-encoder model that scores query and chunk together, which is slower per pair but far more accurate than the bi-encoder similarity used for initial retrieval. The combination of hybrid retrieval plus reranking is consistently the single highest-leverage architectural change teams can make to an underperforming RAG system.
- Hybrid retrieval catches exact identifiers that vector similarity alone routinely misses
- Reciprocal rank fusion combines two incomparable score scales without manual weight tuning
- Cross-encoder reranking on the top 50 to 100 candidates is the highest-leverage accuracy fix available
Closing the Loop: Evaluation as a Pipeline Stage
A RAG system without an evaluation loop is a system nobody can improve with confidence, because every change is a guess validated only by spot-checking a handful of answers. Retrieval metrics like recall@k and mean reciprocal rank tell you whether the right chunk was found at all. Generation metrics like faithfulness and answer relevance tell you whether the model actually used what was retrieved and answered the real question. Build a golden set of 150 to 300 real queries with known-correct source chunks before you tune anything, and rerun the full suite on every chunking, embedding, or prompt change so regressions surface immediately instead of showing up as a support ticket three weeks later.
Infrastructure: pgvector, Dedicated Vector Databases, and Scale
Most enterprise RAG corpora, up to a few million chunks, run comfortably on pgvector inside a Postgres instance you already operate, which keeps vector search transactionally consistent with the source records and avoids standing up a new data plane. Dedicated vector databases like Milvus, Qdrant, and Weaviate earn their operational cost at higher scale, higher query throughput, or when you need advanced payload filtering and built-in hybrid search out of the box. The decision is rarely permanent: start with pgvector, instrument query latency and index build time, and migrate only when the numbers demand it rather than provisioning for a scale you may never reach.
How Netray Builds Enterprise RAG Architectures
Netray designs RAG pipelines stage by stage against a client's real corpus rather than defaulting to a template, because the right chunking strategy for scanned engineering drawings is not the right strategy for a knowledge base of support tickets. Our DataRay product ships the full pipeline, ingestion, hybrid retrieval, reranking, and an evaluation dashboard, as an on-premises deployment for manufacturers and defense suppliers who cannot send documents to a third-party API. We build the golden evaluation set from the client's own historical queries before writing a single chunking rule, and we hand over a system the client's team can extend and re-evaluate without a permanent consulting retainer.
Frequently Asked Questions
What are the six stages of a RAG pipeline?
Ingestion normalizes source documents into text or structured data. Chunking splits that content into retrievable units. Embedding converts each chunk into a vector. Retrieval searches an index for the most relevant chunks to a query. Reranking reorders the top candidates with a more accurate model. Generation synthesizes the final answer, and an evaluation loop measures whether the whole system actually works before and after any change.
Why does RAG accuracy degrade even when the underlying model is strong?
Because accuracy compounds across stages rather than depending on any single one. If ingestion, chunking, retrieval, reranking, and generation are each 95 percent reliable independently, the end-to-end system lands closer to 74 percent. A strong generation model cannot compensate for a chunking strategy that split the answer across two chunks or a retrieval step that never surfaced the right document in the first place.
Should we build a RAG pipeline in-house or use a packaged product?
It depends on how much of the pipeline is genuinely differentiated for your use case versus commodity plumbing. Ingestion, chunking, hybrid retrieval, and evaluation tooling are largely solved problems that a packaged on-premises product like DataRay handles out of the box. In-house effort is best spent on domain-specific chunking rules, access control mapped to your existing systems, and the prompts that reflect how your team actually asks questions.
How long does it take to build a production-grade RAG system?
A well-scoped single-corpus RAG system typically takes six to ten weeks from kickoff to production, covering ingestion pipeline development, chunking and embedding tuning, hybrid retrieval and reranking setup, and building the golden evaluation set. Multi-source corpora with mixed document types, scanned archives, or strict access-control requirements usually add two to four weeks for the additional ingestion and security work.
Key Takeaways
- 1The Six Stages of a Production RAG Pipeline: Ingestion pulls source documents, PDFs, wiki pages, ERP exports, scanned drawings, into a normalized text or structured form, and this is where most accuracy is won or lost before a single embedding is computed. Chunking splits that normalized content into retrievable units sized for your embedding model's context window.
- 2Chunking and Embedding: Decisions That Are Hard to Undo: Chunk size and strategy determine what the retriever can ever find, and re-chunking an existing production corpus is expensive enough that most teams live with the first decision for a year or more. Fixed-size token chunking with 10 to 20 percent overlap is the reasonable default for homogeneous prose, while structural and semantic chunking earn their extra complexity on mixed-format corpora like ERP documentation or engineering specs.
- 3Retrieval, Hybrid Search, and Reranking in Production: Pure vector retrieval misses exact-match terms constantly: part numbers, error codes, and acronyms blur together in embedding space even when the surrounding prose is semantically distinct. Hybrid retrieval runs a lexical search like BM25 alongside vector search and merges the two ranked lists with reciprocal rank fusion, which is now close to a default best practice rather than an advanced technique.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Enterprise RAG Security Checklist
A practical control checklist covering entitlement-aware retrieval, corpus data governance, query security, model integrity, and audit for enterprise RAG systems.
Free ToolRAG Chunking Strategy Calculator
Turn corpus size, chunk length, and overlap into a concrete chunk count, embedding cost, and vector storage footprint before you build the ingestion pipeline.
Free ToolRAG Infrastructure Sizing Calculator
Estimate vector storage, node RAM, generation GPUs, and monthly infrastructure cost for a retrieval-augmented generation deployment over your document corpus.
Terms used in this article
Building a RAG system that needs to survive contact with real enterprise data? Netray designs and deploys the full pipeline, ingestion through evaluation, on your own infrastructure.
Related Resources
RAG Chunking Strategies: Fixed, Semantic, Structural, and Late
Compare RAG chunking strategies, fixed-size, semantic, structural, and late chunking, with concrete guidance on chunk size, overlap, and when each wins.
AI & Automationpgvector vs Dedicated Vector Databases: An Honest Comparison
pgvector vs Milvus, Qdrant, and Weaviate for enterprise RAG: real tradeoffs on scale, latency, operational overhead, and when Postgres is genuinely enough.
AI & AutomationHybrid Search: Combining BM25 and Vector Retrieval with RRF
Hybrid search for RAG: why pure vector retrieval misses exact matches, how BM25 fixes it, and how reciprocal rank fusion combines both reliably.