What Is Reranking?
Also known as: re-ranking, cross-encoder reranker, reranker
Definition
Reranking is a second retrieval stage in RAG where a more precise model re-scores the top candidate passages from the first search and reorders them, so the LLM receives only the most relevant context.
Reranking Explained
First-stage retrieval has to be fast, so it compares one query vector against millions of stored vectors. That is efficient but approximate. A reranker takes the top 20 to 100 candidates and reads each one together with the question, typically using a cross-encoder model, and produces a sharper relevance score. The best handful go to the LLM.
The cost trade-off is clear: the reranker is slower per item but only runs on a short list. The gain is largest when the corpus holds many near-duplicate passages, which is normal in manufacturing: dozens of work instructions that differ by one part number, revisions of the same procedure, or supplier specifications with identical wording.
Reranking is often combined with hybrid search, which merges keyword search (BM25) and vector search. Exact identifiers such as part numbers, drawing numbers and ECO numbers match poorly in embeddings but perfectly in keyword search, so hybrid retrieval followed by reranking is a common strong baseline.
Rerankers can run locally. Open models in the bge-reranker and similar families fit on modest GPUs, so the whole retrieval stack stays inside the customer network.
Why It Matters
- It raises answer accuracy without changing the LLM or re-indexing documents.
- It reduces the context sent to the LLM, which lowers latency and token cost.
- It helps most where documents are repetitive, as in controlled procedures and part catalogs.
- It can run on-prem, so it fits air-gapped deployments.
In Practice
A technician asks for the torque value for a hydraulic fitting on model X. Hybrid search returns 40 candidates from 9 similar work instructions. The reranker scores each against the full question and puts the correct step from the model X instruction first and a similar step from model Y fifth. Only the top 5 go to the LLM, which answers from the right document instead of blending two.
Frequently Asked Questions
Do I always need a reranker?
No. For a small, clean corpus, good chunking and hybrid search may be enough. Add a reranker when similar documents cause wrong or mixed answers.
What is the difference between a reranker and an embedding model?
An embedding model encodes query and passage separately for fast search. A cross-encoder reranker reads them together, which is more accurate but too slow to run over the whole corpus.
Does reranking help with part numbers?
Indirectly. Keyword search catches exact identifiers and the reranker orders the combined results. Treat part numbers as metadata filters where possible.
How much latency does it add?
Typically tens to a few hundred milliseconds for a short candidate list on a GPU, depending on the model and list size.
Related Terms
RAG (Retrieval-Augmented Generation)
RAG (Retrieval-Augmented Generation) is an architecture that searches your own document store for passages relevant to a question, inserts them into the LLM prompt, and instructs the model to answer only from that retrieved evidence.
Embedding
An embedding is a list of numbers, typically several hundred to a few thousand dimensions, that represents the meaning of a piece of text, image, or code so that semantically similar items produce vectors that sit close together.
Vector Database
A vector database is a system that stores high-dimensional embedding vectors and retrieves the ones most similar to a query vector, enabling semantic search where results match meaning rather than exact keywords.
Chunking
Chunking is the step in a RAG pipeline that splits long documents into smaller passages, usually a few hundred tokens each, so each can be embedded, indexed and retrieved on its own. Chunk boundaries strongly affect answer quality.
Semantic Search
Semantic search retrieves results by meaning rather than exact word match, using embeddings to find text that is conceptually close to the query. A search for "supplier delivered damaged goods" can find records that say "vendor shipment arrived broken".
Working with Reranking in a live environment? Our engineers do this every day - and our AI agents automate most of it.