Why RAG retrieves the wrong documents, and how to fix it
why does my RAG pipeline keep returning irrelevant chunks
Also appears as
- retrieval returns documents that have nothing to do with the query
- top-k results are consistently off-topic
- vector search returns low-relevance matches
Short answer
RAG retrieves the wrong documents most often because the embedding model used to index the corpus differs from the one used at query time, or because chunks are large enough that a single embedding averages away the passage that actually answers the question. Fix embedding consistency and chunk granularity first, then add a reranker and metadata filters before touching the LLM prompt.
Affects: Any RAG pipeline built on a vector database (pgvector, Pinecone, Qdrant, Weaviate) regardless of embedding provider, most visible once the corpus exceeds a few thousand chunks.
Fastest path to relevant retrieval
- 1Confirm the exact same embedding model name and version is used for both indexing and querying; log the model id at both call sites and diff them.
- 2Drop chunk size to 200-400 tokens with 10-20% overlap so each vector represents one idea, not a whole section.
- 3Add a cross-encoder reranker over the top 20-50 candidates and keep only the top 3-5 after reranking.
- 4Add metadata filters (document type, date, department) so retrieval narrows the candidate set before vector search runs, not after.
- 5If queries are short or use different vocabulary than the source documents, rewrite the query with an LLM call or use HyDE (embed a generated hypothetical answer instead of the raw query) before searching.
How to confirm this is your problem
- Top-k results are topically unrelated to the question even though relevant documents exist in the corpus
- Retrieval quality was fine on a small test set but degrades as the corpus grows
- The correct passage is in the corpus but never appears in the top 10 results
- Domain-specific acronyms or jargon in the query return generic or off-topic matches
- Switching embedding models made retrieval worse without any changes to chunking or search code
Root causes and fixes
Embedding model mismatch between indexing and query time
If the corpus was embedded with one model (or one version of a model) and queries are embedded with a different one, the two sets of vectors do not share the same geometry. Cosine similarity between vectors from different models is close to meaningless, so the nearest neighbors returned are essentially random with respect to actual relevance.
Fix: Pin the embedding model name and revision in config, verify it at both ingestion and query time, and if they ever diverged, re-embed the entire corpus with the query-time model before trusting results again.
python -c "import json; print(open('embed_config.json').read())"
grep -r 'embedding_model' ingest/ query/No reranker, so the vector search top-k is served directly to the LLM
Bi-encoder vector search trades accuracy for speed by embedding query and document independently. It is good at getting roughly relevant candidates into the top 50, but weak at fine-grained ranking of the top 5. Without a reranker, borderline-relevant chunks that share surface vocabulary can outrank the genuinely correct passage.
Fix: Retrieve a wider candidate set (top 20-50) with the vector search, then score all candidates with a cross-encoder reranker and keep only the top 3-5 before building the prompt.
Chunks are too large and dilute the embedding
A single embedding vector represents the average meaning of everything in the chunk. If a chunk spans multiple topics or a long section with only one relevant sentence, the embedding is pulled toward the chunk's dominant theme and away from the specific detail the query needs, so it never surfaces as a close match.
Fix: Re-chunk at 200-400 tokens per chunk with meaningful boundaries (paragraph or section, not fixed character count), and use overlap so no idea is split across a chunk edge.
Missing metadata filters let irrelevant document types compete in the vector search
When every document type (contracts, marketing decks, old drafts, meeting notes) is embedded into the same index with no structured filters, vector similarity alone has to distinguish them, and superficially similar wording from the wrong document type can outscore the right one.
Fix: Tag chunks with metadata (document type, department, effective date, product line) at ingestion time and apply hard filters before or alongside the vector search, not as a post-filter on results.
The query needs rewriting because it does not resemble how the answer is phrased
Users ask short, conversational questions while source documents are written in formal procedural language. Bi-encoder retrieval is sensitive to this phrasing gap, especially for domain vocabulary the base embedding model never saw much of during pretraining.
Fix: Use query rewriting (an LLM call that expands or reformulates the question) or HyDE, generating a hypothetical answer passage and embedding that instead of the raw query, since it resembles the target documents more closely.
Diagnostic commands
Compare embedding model identity at index and query time
python -c "from your_pipeline import get_embedder; print(get_embedder().model_name)"
If this prints a different model name or revision at ingestion time vs query time, that mismatch alone explains poor retrieval; re-embedding the corpus with the query-time model is the fix.
Inspect raw vector search scores for a known-good query
SELECT id, 1 - (embedding <=> query_vector) AS similarity FROM chunks ORDER BY embedding <=> query_vector LIMIT 20;
If the correct chunk's similarity score is close to the top-ranked wrong chunk (within 0.02-0.05), the embedding space is not discriminating well and a reranker will help more than tuning vector search further.
Check chunk size distribution in the index
python -c "import statistics as s; lens=[len(c.split()) for c in chunks]; print(s.mean(lens), max(lens))"
Average chunk length above roughly 500 tokens strongly suggests dilution is contributing to poor retrieval; smaller, single-topic chunks usually improve precision immediately.
Manually inspect top-20 (not just top-5) results for a failing query
python retrieve.py --query "your failing query" --k 20 --print-scores
If the correct passage appears in positions 6-20 but not the top 5, a reranker over a wider candidate set will likely fix it; if it does not appear at all in 20, the problem is recall (embedding/chunking), not ranking.
Stopping it from happening again
- Log and version-pin the embedding model name at both ingestion and query call sites, and fail loudly on mismatch rather than silently serving stale vectors.
- Build a small labeled evaluation set of query-to-correct-chunk pairs and run retrieval recall@k on every change to chunking, embedding model, or reranker.
- Re-embed the full corpus, not just new documents, any time the embedding model or chunking strategy changes.
- Keep chunk size and overlap as explicit, tested config values rather than defaults inherited from a tutorial.
- Add metadata tagging to the ingestion pipeline from day one; retrofitting it later requires a full re-index.
When this becomes an architecture problem
If retrieval still misses obvious answers after fixing embedding consistency, chunk size, and adding a reranker, the problem is usually the retrieval architecture itself (single vector index, no hybrid search, no domain-tuned embeddings) rather than another config tweak. That is the point to bring in an architecture review rather than iterate further; Netray's DataRay platform is built specifically to get retrieval quality right for regulated, multi-source enterprise data on customer-owned hardware.
Frequently asked questions
Why does RAG retrieval get worse as the corpus grows even though nothing else changed?
Larger corpora increase the number of superficially similar chunks competing for the same top-k slots, which exposes weaknesses in bi-encoder-only retrieval that a small test corpus hides. Adding a reranker and metadata filters becomes necessary at scale even if pure vector search seemed adequate with a few hundred documents.
Should I fix chunking or add a reranker first?
Fix chunking first. A reranker can only reorder the candidates the vector search actually retrieves, so if the correct chunk never makes the top 50 because it is diluted inside an oversized chunk, no reranker will recover it. Once recall at k=20-50 looks reasonable, a reranker improves precision at the top of the list.
Does HyDE always help?
HyDE helps most when queries are short, conversational, or use different vocabulary than the source documents, since it embeds a generated hypothetical answer instead of the literal question. It adds an extra LLM call and some latency, and provides little benefit when queries already closely match document phrasing.
How do I know if my embedding model understands my domain vocabulary?
Embed a handful of domain-specific terms and their known synonyms, then check cosine similarity between them. If closely related domain terms score no higher than unrelated words, the general-purpose embedding model was not exposed to enough domain text and a domain-adapted embedding model will likely outperform it.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
RAG Chunking Strategy Calculator
Turn corpus size, chunk length, and overlap into a concrete chunk count, embedding cost, and vector storage footprint before you build the ingestion pipeline.
Free ToolRAG Accuracy Readiness Assessment
Score your retrieval-augmented generation system across eight dimensions that actually predict production accuracy, from chunking strategy to groundedness verification.
Free ToolVector Database Sizing Calculator
Convert vector count, embedding dimensions, and precision into a real storage footprint, including index overhead and replica factor, before you pick a vector database.
Related problems
RAG hallucinates even though the correct context was retrieved
RAG hallucination with good context in hand usually means the prompt never explicitly instructs the model to answer only from the provided passages, or the correct passage is buried in the middle of a long context window where attention is weakest. Add an explicit grounding instruction, place the most relevant passage first, resolve conflicting retrieved chunks before generation, and give the model an explicit abstain option.
Adding a reranker isn't improving RAG results
A reranker that shows no improvement usually means it is only reordering a candidate pool that was already too small, top-3 to top-5, to contain the correct answer, so there is nothing better to promote, or the wrong reranker model was chosen for the domain. Retrieve a wider candidate set of 20-50 before reranking, verify the reranker model actually outperforms your vector search on a labeled evaluation set, and budget the added latency deliberately rather than treating it as free.
Chunking splits tables and headers, destroying their meaning
Generic character-count or token-count chunking treats a document as an undifferentiated stream of text, so it routinely cuts a table's header row away from its data rows, leaving a retrieved chunk full of numbers with no column labels to explain what they mean. The fix is structure-aware chunking that detects table boundaries, keeps the header attached to every chunk of that table's rows, and never splits mid-row.
Embedding dimension mismatch after switching models
Different embedding models produce vectors of different fixed dimensions, so swapping models without re-embedding the entire corpus produces a hard dimension mismatch error or, worse, silently meaningless similarity scores if the column is resized without re-indexing. There is no shortcut: changing the embedding model requires re-embedding every document and rebuilding the vector index from scratch.
GuideEnterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
GuideRAG Chunking Strategies: Fixed, Semantic, Structural, and Late
Compare RAG chunking strategies, fixed-size, semantic, structural, and late chunking, with concrete guidance on chunk size, overlap, and when each wins.
GuideHybrid Search: Combining BM25 and Vector Retrieval with RRF
Hybrid search for RAG: why pure vector retrieval misses exact matches, how BM25 fixes it, and how reciprocal rank fusion combines both reliably.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.