# RAG Readiness Assessment for ERP Documents > Source: https://netray.co/tools/rag-readiness-assessment-for-erp-documents > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer Ten questions that show whether your documents and ERP attachments are ready to feed a retrieval-based AI assistant. ## Category ai-automation ## Audience - CIO - IT manager - Knowledge manager ## Tool **Kind:** assessment **Questions:** #### Where do your SOPs, work instructions and quality procedures live? **Help:** Retrieval needs a known set of sources. **Options:** ##### One controlled system with versions **Score:** 3 ##### A few shares and a document system **Score:** 2 ##### Scattered across shares, email and desktops **Score:** 0 #### How current are the documents? **Help:** A retrieval system will quote outdated documents as confidently as current ones. **Options:** ##### Revision status is tracked and obsolete files are archived **Score:** 3 ##### Mostly current but obsolete copies remain **Score:** 1 ##### Nobody knows which revision is valid **Score:** 0 #### What formats dominate? **Options:** ##### Text-based PDF, Word, HTML **Score:** 3 ##### Mix of text files and scanned PDFs **Score:** 1 ##### Mostly scans, images and drawings **Score:** 0 #### Do documents carry metadata such as part number, customer, revision and document type? **Options:** ##### Yes, consistently **Score:** 3 ##### Partly, in file names only **Score:** 1 ##### No **Score:** 0 #### Are document-level permissions defined and enforceable? **Help:** Retrieval must not show a user a document they cannot open. **Options:** ##### Yes, groups map to ERP or directory roles **Score:** 3 ##### Folder permissions exist but are inconsistent **Score:** 1 ##### Everything is open or access is unknown **Score:** 0 #### How are ERP attachments handled? **Options:** ##### Stored in a file service with a stable link from ERP records **Score:** 3 ##### Stored in the database as blobs **Score:** 2 ##### Local paths or broken links **Score:** 0 #### Do you have a test set of real questions with known correct answers? **Options:** ##### Yes, 50 or more with source documents **Score:** 3 ##### A handful of examples **Score:** 1 ##### No **Score:** 0 #### Is there a named owner for content quality? **Options:** ##### Yes, with time allocated **Score:** 3 ##### Informal owner **Score:** 1 ##### No owner **Score:** 0 #### Are there restricted documents that must stay in a specific environment? **Help:** Export-controlled or customer-restricted files affect where indexing can run. **Options:** ##### Identified and tagged, hosting decided **Score:** 3 ##### Known but not tagged **Score:** 1 ##### Unknown **Score:** 0 #### Is there a plan to keep the index current when documents change? **Options:** ##### Automated incremental updates with deletion handling **Score:** 3 ##### Manual periodic re-index **Score:** 1 ##### No plan **Score:** 0 **Bands:** #### Not ready: fix the content first **Min:** 0 **Max:** 39 **Verdict:** Retrieval would surface outdated, unpermissioned or unreadable material. The model is not the main problem; the document estate is. **Recommendations:** - Choose one controlled source for SOPs and quality documents - Archive obsolete revisions before indexing anything - Define who may see what and write it down - Pilot on a small, clean set such as one product line #### Partly ready: pilot on a clean slice **Min:** 40 **Max:** 64 **Verdict:** A pilot is reasonable if it is scoped to documents that are current, readable and permissioned. **Recommendations:** - Add part number, revision and document type metadata - OCR scanned PDFs and check the text output by hand - Build a test set of 50 real questions with source documents - Decide where indexing and the model will run for restricted content #### Ready for a controlled pilot **Min:** 65 **Max:** 84 **Verdict:** Your content and permissions support a useful pilot. Measure answer accuracy and citation quality against a test set. **Recommendations:** - Require citations for every answer - Test permission trimming with low-privilege users - Set a refresh schedule and deletion handling - Track unanswered questions to find content gaps #### Well prepared: focus on evaluation and scale **Min:** 85 **Max:** 100 **Verdict:** Your documents are in good shape. The risks now are evaluation discipline, cost and change management. **Recommendations:** - Automate regression tests against the question set - Monitor retrieval hit rate and user feedback - Extend to ERP data and structured records - Plan capacity before adding more sources ## Content **Intro:** Retrieval-augmented generation is only as good as the documents behind it. This assessment checks the unglamorous things that decide whether an ERP document assistant gives answers people trust. **Sections:** #### How to use the assessment **Body:** Answer for the document set you actually want to search, not for the company as a whole. A quality manual library might score well while supplier correspondence scores poorly. Run the assessment once per source group, then start the pilot with the one that scores highest. Involve the person who maintains the documents, since IT often cannot judge revision status or permission rules. #### What the questions measure **Body:** The ten questions cover four areas: source control, which means knowing where documents live and which revision is valid; extractability, meaning whether text can be read from the files; metadata, which lets retrieval filter by part, customer or revision; and security, covering permissions and restricted content. The last questions cover evaluation and maintenance, because a pilot with no test set and no refresh plan decays within months. **Bullets:** - Obsolete documents are the leading cause of confident wrong answers - Scanned drawings need OCR or a separate approach, not plain text embedding - Permission trimming must happen at retrieval time, not after generation #### Reading your score **Body:** Below 40 percent, spend a quarter on content before any AI spend. Between 40 and 64, run a narrow pilot on a clean slice and fix gaps as you meet them. Between 65 and 84, a controlled pilot with citations and a test set is sensible. Above 85, your attention should shift to evaluation, cost and rollout. A high score does not remove the need for human review of answers that affect product conformity. #### Common mistakes **Body:** Teams often index everything at once, including drafts and superseded files, and then blame the model for poor answers. Another is skipping the test set, which leaves accuracy as an opinion. Some forget that retrieval over ERP attachments inherits the ERP record permission model. Finally, avoid measuring success by demo quality: use real user questions and check the cited source for each answer. **Cta:** If you want a private document assistant that keeps indexing and answers on your own infrastructure, we can run a pilot on a clean slice of your documents and report accuracy against your test questions. **Faq:** #### Do I need to clean all documents first? **Answer:** No. Start with one well-controlled group, such as current work instructions for one product line. Expanding in stages is faster and exposes problems while the scope is still small. #### Can RAG read scanned drawings? **Answer:** Partly. OCR can extract title block text and notes, but geometry and dimensions are poorly handled by text retrieval. Treat drawings as a separate project and index their metadata first. #### How big should the test set be? **Answer:** Fifty questions with known source documents is a workable start, and a few hundred is better. Include questions where the correct answer is that the information is not in the documents. #### Does this apply to ERP data as well? **Answer:** Structured ERP data is usually better served by direct, permissioned queries than by embedding rows. Use RAG for documents and attachments, and queries for transactional records. ## Search terms rag readiness assessment, is my data ready for rag, rag document preparation checklist, erp document ai readiness, retrieval augmented generation readiness, enterprise search ai readiness, document quality for llm, rag assessment manufacturing --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json