RAG Readiness Assessment for ERP Documents
Retrieval-augmented generation is only as good as the documents behind it. This assessment checks the unglamorous things that decide whether an ERP document assistant gives answers people trust.
1. Where do your SOPs, work instructions and quality procedures live?
Retrieval needs a known set of sources.
2. How current are the documents?
A retrieval system will quote outdated documents as confidently as current ones.
3. What formats dominate?
4. Do documents carry metadata such as part number, customer, revision and document type?
5. Are document-level permissions defined and enforceable?
Retrieval must not show a user a document they cannot open.
6. How are ERP attachments handled?
7. Do you have a test set of real questions with known correct answers?
8. Is there a named owner for content quality?
9. Are there restricted documents that must stay in a specific environment?
Export-controlled or customer-restricted files affect where indexing can run.
10. Is there a plan to keep the index current when documents change?
How to use the assessment
Answer for the document set you actually want to search, not for the company as a whole. A quality manual library might score well while supplier correspondence scores poorly. Run the assessment once per source group, then start the pilot with the one that scores highest. Involve the person who maintains the documents, since IT often cannot judge revision status or permission rules.
What the questions measure
The ten questions cover four areas: source control, which means knowing where documents live and which revision is valid; extractability, meaning whether text can be read from the files; metadata, which lets retrieval filter by part, customer or revision; and security, covering permissions and restricted content. The last questions cover evaluation and maintenance, because a pilot with no test set and no refresh plan decays within months.
- Obsolete documents are the leading cause of confident wrong answers
- Scanned drawings need OCR or a separate approach, not plain text embedding
- Permission trimming must happen at retrieval time, not after generation
Reading your score
Below 40 percent, spend a quarter on content before any AI spend. Between 40 and 64, run a narrow pilot on a clean slice and fix gaps as you meet them. Between 65 and 84, a controlled pilot with citations and a test set is sensible. Above 85, your attention should shift to evaluation, cost and rollout. A high score does not remove the need for human review of answers that affect product conformity.
Common mistakes
Teams often index everything at once, including drafts and superseded files, and then blame the model for poor answers. Another is skipping the test set, which leaves accuracy as an opinion. Some forget that retrieval over ERP attachments inherits the ERP record permission model. Finally, avoid measuring success by demo quality: use real user questions and check the cited source for each answer.
Frequently Asked Questions
Do I need to clean all documents first?
No. Start with one well-controlled group, such as current work instructions for one product line. Expanding in stages is faster and exposes problems while the scope is still small.
Can RAG read scanned drawings?
Partly. OCR can extract title block text and notes, but geometry and dimensions are poorly handled by text retrieval. Treat drawings as a separate project and index their metadata first.
How big should the test set be?
Fifty questions with known source documents is a workable start, and a few hundred is better. Include questions where the correct answer is that the information is not in the documents.
Does this apply to ERP data as well?
Structured ERP data is usually better served by direct, permissioned queries than by embedding rows. Use RAG for documents and attachments, and queries for transactional records.
If you want a private document assistant that keeps indexing and answers on your own infrastructure, we can run a pilot on a clean slice of your documents and report accuracy against your test questions.
Related Tools
On-Prem GPU Sizing Calculator for ERP AI
Work out how much GPU memory and how many cards you need to serve an open-weight model to your ERP users on-prem.
AI Agents & AutomationAI Pilot Charter Template
A one-to-two page charter for an AI or agent pilot on your ERP: scope, success metrics, data access, risks, budget and stop rules.
AI Agents & AutomationAI Use Case Prioritization Worksheet for ERP
Rank candidate AI use cases on your ERP by value, feasibility, data readiness and risk, then export the shortlist to Excel.
On-Prem AIRAG Accuracy Readiness Assessment
Score your retrieval-augmented generation system across eight dimensions that actually predict production accuracy, from chunking strategy to groundedness verification.
More AI Agents & Automation tools
See all- Business Process Automation ROI Calculator
- AI Agent Use Case Prioritizer
- Document Processing Automation Savings Calculator
- AP Invoice Automation ROI Calculator
- Order Entry Automation Savings Calculator
- Quote-to-Cash Cycle Time Calculator
- Manufacturing AI Readiness Assessment
- RPA vs AI Agent Selector
- ERP AI Maturity Assessment
Keep exploring
Free tools
- ERP Agent API Integration Readiness AssessmentAnswer ten questions about APIs, roles, limits and logging to see whether your ERP is ready for an AI agent or MCP integration.
- Manufacturing AI Readiness AssessmentScore your manufacturing operation's readiness for AI across data, systems, people, and governance, and get a prioritized roadmap for closing the gaps.
- WMS Readiness AssessmentTen questions to score whether your warehouse is ready for a WMS, or whether ERP inventory control and process fixes come first.
ERP fixes and how-tos
- Is Your ERP Data Ready for AI? A Readiness ChecklistMost ERP data is ready enough to pilot AI today. A practical checklist of the master-data and access-control issues that actually break AI answers.
- SyteLine DocTrak: What It Is and How Document Attachments WorkWhat DocTrak means in Infor SyteLine, how document attachments are linked to records, where files live, and how to configure and secure them.
- How to Add a Private AI Layer to Oracle EBS 12.2 On-PremReference architecture for a private AI layer on Oracle EBS 12.2: Active Data Guard replica, ISG REST, MO and responsibility security, local GPUs and audit.
Guides
- Multimodal RAG Over Engineering Drawings and Scanned DocumentsMultimodal RAG for manufacturers: retrieving from CAD drawings, scanned specs, and PDFs with embedded tables, using vision embeddings and layout-aware parsing.
- SyteLine Document Management and Automation: From Attachments to IDMAutomate SyteLine document management: attached documents, Infor Document Management (IDM) integration, revision control, and ITAR-safe file handling.
- ERP Go-Live Readiness Assessment ChecklistValidate ERP go-live readiness with a 100-point checklist covering data, integrations, training, cutover, and support. Prevent post-go-live emergencies.
AI on your ERP
- Is Your ERP Ready for AI? A Readiness Assessment ChecklistIs your ERP actually ready for AI? A practical checklist covering master data quality, access and permissions, GPU sizing, and governance before you fund a pilot.
- A Private LLM Grounded on Your ERP DataHow a private LLM answers questions on your ERP data: RAG plus text-to-SQL, role-based permissions inherited from the ERP, and where each fits.
Integrations
- How to add an AI chatbot to IFS CloudHow to add an AI chatbot to IFS Cloud: grounded answers from IFS data and documents, embedding in Aurena or a portal, role-aware access and fully on-prem models.
- How to add an AI chatbot to Microsoft Dynamics GPAdd an AI chatbot to Dynamics GP for staff or customers: grounded in SQL data and documents, role aware, hosted on-prem with open models, with honest trade-offs.
Troubleshooting
- Why RAG retrieves the wrong documents, and how to fix itRAG returns irrelevant chunks when the embedding model differs between indexing and query time, chunks are too large, or queries need rewriting. Fix it here.
- Why fixed-size chunking destroys tables in your documents, and how to chunk them correctlyFixed-size chunking cuts table rows away from their headers, so a retrieved row of numbers has no column labels left. Use structure-aware chunking to fix it.