RAG Readiness Assessment for ERP Documents
Retrieval-augmented generation is only as good as the documents behind it. This assessment checks the unglamorous things that decide whether an ERP document assistant gives answers people trust.
1. Where do your SOPs, work instructions and quality procedures live?
Retrieval needs a known set of sources.
2. How current are the documents?
A retrieval system will quote outdated documents as confidently as current ones.
3. What formats dominate?
4. Do documents carry metadata such as part number, customer, revision and document type?
5. Are document-level permissions defined and enforceable?
Retrieval must not show a user a document they cannot open.
6. How are ERP attachments handled?
7. Do you have a test set of real questions with known correct answers?
8. Is there a named owner for content quality?
9. Are there restricted documents that must stay in a specific environment?
Export-controlled or customer-restricted files affect where indexing can run.
10. Is there a plan to keep the index current when documents change?
How to use the assessment
Answer for the document set you actually want to search, not for the company as a whole. A quality manual library might score well while supplier correspondence scores poorly. Run the assessment once per source group, then start the pilot with the one that scores highest. Involve the person who maintains the documents, since IT often cannot judge revision status or permission rules.
What the questions measure
The ten questions cover four areas: source control, which means knowing where documents live and which revision is valid; extractability, meaning whether text can be read from the files; metadata, which lets retrieval filter by part, customer or revision; and security, covering permissions and restricted content. The last questions cover evaluation and maintenance, because a pilot with no test set and no refresh plan decays within months.
- Obsolete documents are the leading cause of confident wrong answers
- Scanned drawings need OCR or a separate approach, not plain text embedding
- Permission trimming must happen at retrieval time, not after generation
Reading your score
Below 40 percent, spend a quarter on content before any AI spend. Between 40 and 64, run a narrow pilot on a clean slice and fix gaps as you meet them. Between 65 and 84, a controlled pilot with citations and a test set is sensible. Above 85, your attention should shift to evaluation, cost and rollout. A high score does not remove the need for human review of answers that affect product conformity.
Common mistakes
Teams often index everything at once, including drafts and superseded files, and then blame the model for poor answers. Another is skipping the test set, which leaves accuracy as an opinion. Some forget that retrieval over ERP attachments inherits the ERP record permission model. Finally, avoid measuring success by demo quality: use real user questions and check the cited source for each answer.
Frequently Asked Questions
Do I need to clean all documents first?
No. Start with one well-controlled group, such as current work instructions for one product line. Expanding in stages is faster and exposes problems while the scope is still small.
Can RAG read scanned drawings?
Partly. OCR can extract title block text and notes, but geometry and dimensions are poorly handled by text retrieval. Treat drawings as a separate project and index their metadata first.
How big should the test set be?
Fifty questions with known source documents is a workable start, and a few hundred is better. Include questions where the correct answer is that the information is not in the documents.
Does this apply to ERP data as well?
Structured ERP data is usually better served by direct, permissioned queries than by embedding rows. Use RAG for documents and attachments, and queries for transactional records.
If you want a private document assistant that keeps indexing and answers on your own infrastructure, we can run a pilot on a clean slice of your documents and report accuracy against your test questions.
Related Tools
On-Prem GPU Sizing Calculator for ERP AI
Work out how much GPU memory and how many cards you need to serve an open-weight model to your ERP users on-prem.
AI Agents & AutomationAI Pilot Charter Template
A one-to-two page charter for an AI or agent pilot on your ERP: scope, success metrics, data access, risks, budget and stop rules.
AI Agents & AutomationAI Use Case Prioritization Worksheet for ERP
Rank candidate AI use cases on your ERP by value, feasibility, data readiness and risk, then export the shortlist to Excel.
On-Prem AIRAG Accuracy Readiness Assessment
Score your retrieval-augmented generation system across eight dimensions that actually predict production accuracy, from chunking strategy to groundedness verification.
More AI Agents & Automation tools
See all- Business Process Automation ROI Calculator
- AI Agent Use Case Prioritizer
- Document Processing Automation Savings Calculator
- AP Invoice Automation ROI Calculator
- Order Entry Automation Savings Calculator
- Quote-to-Cash Cycle Time Calculator
- Manufacturing AI Readiness Assessment
- RPA vs AI Agent Selector
- ERP AI Maturity Assessment