# Why RAG retrieves the wrong documents, and how to fix it

> Source: https://netray.co/troubleshoot/rag-retrieving-wrong-documents
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

RAG retrieves the wrong documents most often because the embedding model used to index the corpus differs from the one used at query time, or because chunks are large enough that a single embedding averages away the passage that actually answers the question. Fix embedding consistency and chunk granularity first, then add a reranker and metadata filters before touching the LLM prompt.

## Error Title

RAG retrieves irrelevant or wrong documents

## Error Signature

why does my RAG pipeline keep returning irrelevant chunks

## Alternate Signatures

- retrieval returns documents that have nothing to do with the query
- top-k results are consistently off-topic
- vector search returns low-relevance matches

## Category

rag-retrieval

## Stack

- vector-db
- langchain
- huggingface

## Affects

Any RAG pipeline built on a vector database (pgvector, Pinecone, Qdrant, Weaviate) regardless of embedding provider, most visible once the corpus exceeds a few thousand chunks.

## Quick Fix

### Fastest path to relevant retrieval

**Steps:**

- Confirm the exact same embedding model name and version is used for both indexing and querying; log the model id at both call sites and diff them.
- Drop chunk size to 200-400 tokens with 10-20% overlap so each vector represents one idea, not a whole section.
- Add a cross-encoder reranker over the top 20-50 candidates and keep only the top 3-5 after reranking.
- Add metadata filters (document type, date, department) so retrieval narrows the candidate set before vector search runs, not after.
- If queries are short or use different vocabulary than the source documents, rewrite the query with an LLM call or use HyDE (embed a generated hypothetical answer instead of the raw query) before searching.

## Symptoms

- Top-k results are topically unrelated to the question even though relevant documents exist in the corpus
- Retrieval quality was fine on a small test set but degrades as the corpus grows
- The correct passage is in the corpus but never appears in the top 10 results
- Domain-specific acronyms or jargon in the query return generic or off-topic matches
- Switching embedding models made retrieval worse without any changes to chunking or search code

## Causes

### Embedding model mismatch between indexing and query time

**Likelihood:** most-common

**Explanation:** If the corpus was embedded with one model (or one version of a model) and queries are embedded with a different one, the two sets of vectors do not share the same geometry. Cosine similarity between vectors from different models is close to meaningless, so the nearest neighbors returned are essentially random with respect to actual relevance.

**Fix:** Pin the embedding model name and revision in config, verify it at both ingestion and query time, and if they ever diverged, re-embed the entire corpus with the query-time model before trusting results again.

**Commands:**

- python -c "import json; print(open('embed_config.json').read())"
- grep -r 'embedding_model' ingest/ query/

### No reranker, so the vector search top-k is served directly to the LLM

**Likelihood:** common

**Explanation:** Bi-encoder vector search trades accuracy for speed by embedding query and document independently. It is good at getting roughly relevant candidates into the top 50, but weak at fine-grained ranking of the top 5. Without a reranker, borderline-relevant chunks that share surface vocabulary can outrank the genuinely correct passage.

**Fix:** Retrieve a wider candidate set (top 20-50) with the vector search, then score all candidates with a cross-encoder reranker and keep only the top 3-5 before building the prompt.

### Chunks are too large and dilute the embedding

**Likelihood:** common

**Explanation:** A single embedding vector represents the average meaning of everything in the chunk. If a chunk spans multiple topics or a long section with only one relevant sentence, the embedding is pulled toward the chunk's dominant theme and away from the specific detail the query needs, so it never surfaces as a close match.

**Fix:** Re-chunk at 200-400 tokens per chunk with meaningful boundaries (paragraph or section, not fixed character count), and use overlap so no idea is split across a chunk edge.

### Missing metadata filters let irrelevant document types compete in the vector search

**Likelihood:** occasional

**Explanation:** When every document type (contracts, marketing decks, old drafts, meeting notes) is embedded into the same index with no structured filters, vector similarity alone has to distinguish them, and superficially similar wording from the wrong document type can outscore the right one.

**Fix:** Tag chunks with metadata (document type, department, effective date, product line) at ingestion time and apply hard filters before or alongside the vector search, not as a post-filter on results.

### The query needs rewriting because it does not resemble how the answer is phrased

**Likelihood:** occasional

**Explanation:** Users ask short, conversational questions while source documents are written in formal procedural language. Bi-encoder retrieval is sensitive to this phrasing gap, especially for domain vocabulary the base embedding model never saw much of during pretraining.

**Fix:** Use query rewriting (an LLM call that expands or reformulates the question) or HyDE, generating a hypothetical answer passage and embedding that instead of the raw query, since it resembles the target documents more closely.

## Diagnostics

### Compare embedding model identity at index and query time

**Command:** python -c "from your_pipeline import get_embedder; print(get_embedder().model_name)"

**Interpretation:** If this prints a different model name or revision at ingestion time vs query time, that mismatch alone explains poor retrieval; re-embedding the corpus with the query-time model is the fix.

### Inspect raw vector search scores for a known-good query

**Command:** SELECT id, 1 - (embedding <=> query_vector) AS similarity FROM chunks ORDER BY embedding <=> query_vector LIMIT 20;

**Interpretation:** If the correct chunk's similarity score is close to the top-ranked wrong chunk (within 0.02-0.05), the embedding space is not discriminating well and a reranker will help more than tuning vector search further.

### Check chunk size distribution in the index

**Command:** python -c "import statistics as s; lens=[len(c.split()) for c in chunks]; print(s.mean(lens), max(lens))"

**Interpretation:** Average chunk length above roughly 500 tokens strongly suggests dilution is contributing to poor retrieval; smaller, single-topic chunks usually improve precision immediately.

### Manually inspect top-20 (not just top-5) results for a failing query

**Command:** python retrieve.py --query "your failing query" --k 20 --print-scores

**Interpretation:** If the correct passage appears in positions 6-20 but not the top 5, a reranker over a wider candidate set will likely fix it; if it does not appear at all in 20, the problem is recall (embedding/chunking), not ranking.

## Prevention

- Log and version-pin the embedding model name at both ingestion and query call sites, and fail loudly on mismatch rather than silently serving stale vectors.
- Build a small labeled evaluation set of query-to-correct-chunk pairs and run retrieval recall@k on every change to chunking, embedding model, or reranker.
- Re-embed the full corpus, not just new documents, any time the embedding model or chunking strategy changes.
- Keep chunk size and overlap as explicit, tested config values rather than defaults inherited from a tutorial.
- Add metadata tagging to the ingestion pipeline from day one; retrofitting it later requires a full re-index.

## When To Escalate

If retrieval still misses obvious answers after fixing embedding consistency, chunk size, and adding a reranker, the problem is usually the retrieval architecture itself (single vector index, no hybrid search, no domain-tuned embeddings) rather than another config tweak. That is the point to bring in an architecture review rather than iterate further; Netray's DataRay platform is built specifically to get retrieval quality right for regulated, multi-source enterprise data on customer-owned hardware.

## Faq

### Why does RAG retrieval get worse as the corpus grows even though nothing else changed?

**Answer:** Larger corpora increase the number of superficially similar chunks competing for the same top-k slots, which exposes weaknesses in bi-encoder-only retrieval that a small test corpus hides. Adding a reranker and metadata filters becomes necessary at scale even if pure vector search seemed adequate with a few hundred documents.

### Should I fix chunking or add a reranker first?

**Answer:** Fix chunking first. A reranker can only reorder the candidates the vector search actually retrieves, so if the correct chunk never makes the top 50 because it is diluted inside an oversized chunk, no reranker will recover it. Once recall at k=20-50 looks reasonable, a reranker improves precision at the top of the list.

### Does HyDE always help?

**Answer:** HyDE helps most when queries are short, conversational, or use different vocabulary than the source documents, since it embeds a generated hypothetical answer instead of the literal question. It adds an extra LLM call and some latency, and provides little benefit when queries already closely match document phrasing.

### How do I know if my embedding model understands my domain vocabulary?

**Answer:** Embed a handful of domain-specific terms and their known synonyms, then check cosine similarity between them. If closely related domain terms score no higher than unrelated words, the general-purpose embedding model was not exposed to enough domain text and a domain-adapted embedding model will likely outperform it.

## Search terms

RAG retrieving wrong documents, RAG irrelevant retrieval, vector search returns wrong results, embedding model mismatch RAG, RAG chunk size, HyDE query rewriting, reranker RAG, retrieval augmented generation accuracy

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json