# Document AI and OCR Pilot Readiness Checklist for AP and Quality Teams

> Source: https://netray.co/tools/idp-pilot-readiness-checklist
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

A practical checklist covering sample selection, ground truth labeling, success metrics, security and ERP posting before you start an OCR or document AI pilot.

## Category

ai-automation

## Audience

- AP manager
- Quality manager
- IT project lead

## Tool

**Kind:** checklist

**Groups:**

#### Scope and success criteria

**Items:**

**Text:** Document types in scope are listed (supplier invoice, credit note, certificate of conformance, packing slip)

**Critical:** true

Out-of-scope documents are named so they are not counted as failures

**Text:** The business owner who signs off on the pilot result is named

**Critical:** true

**Text:** Baseline is measured: current handling time per document, error rate and rework rate

**Critical:** true

**Text:** Success thresholds are written down before the pilot starts (field accuracy, straight-through rate, cost per document)

**Critical:** true

Pilot duration and the decision date are fixed

Fallback plan exists if the pilot is stopped

#### Sample set and ground truth

**Items:**

**Text:** Sample set is drawn from real production documents, not clean examples

**Critical:** true

Sample covers all major suppliers and also a long tail of small suppliers

**Text:** Poor scans, photos, faxes and rotated pages are included on purpose

**Critical:** true

At least a few hundred documents per type are available for testing

**Text:** Each document has a verified ground truth for every field to be extracted

**Critical:** true

Two people label a subset so that disagreement rate is known

**Text:** Test set is kept separate from any documents used to tune rules or prompts

**Critical:** true

Languages and character sets are listed (accents, non-Latin scripts)

#### Metrics and review process

**Items:**

**Text:** Field-level accuracy is measured per field, not as one blended number

**Critical:** true

Precision, recall and F1 are computed where fields may be missing

**Text:** Straight-through rate is measured as documents posted without a human touch

**Critical:** true

Confidence threshold is chosen from measured data, not by guess

**Text:** Human review step and queue owner are defined

**Critical:** true

Time per reviewed document is recorded

Errors are categorized (image quality, layout, engine, validation rule)

#### Validation and ERP integration

**Items:**

**Text:** Vendor master match is part of the flow

**Critical:** true

**Text:** Duplicate invoice detection is defined (supplier plus number plus amount plus date)

**Critical:** true

Three way match tolerances are agreed (price, quantity, tax)

Document type routing is defined for credit notes, proformas and corrected invoices

**Text:** Posting method is chosen (park document, workflow, direct post)

**Critical:** true

**Text:** Test environment of the ERP is available with representative master data

**Critical:** true

Audit trail keeps the source image, the extracted values and who changed them

Attachment storage in the ERP or archive is decided

#### Security, privacy and operations

**Items:**

**Text:** Data classification of the documents is confirmed (export-controlled, customer confidential, personal data)

**Critical:** true

**Text:** Where documents are processed and stored is decided and approved

**Critical:** true

Retention and deletion rules apply to the pilot copies

Access to the pilot environment is limited and logged

**Text:** Vendor contract states what happens to submitted documents and whether they are used for training

**Critical:** true

Email inbox or scan source feeding the pilot is defined

Support contact and incident process are in place for the pilot period

**Scoring Note:** Do not start the pilot until every critical item is checked. Unchecked non-critical items should have an owner and a date.

## Content

**Intro:** Most failed document AI pilots fail on preparation: an easy sample, no ground truth, and no agreed definition of success. Use this checklist to fix those before any vendor or model is switched on.

**Sections:**

#### How to use the checklist

**Body:** Work through the groups with the AP lead, quality lead and IT. Tick items as they are done and use the export to share the status. Critical items are those that make the result untrustworthy if missing, such as a ground-truth set or a pre-agreed threshold. Treat the checklist as a gate: when all critical items are complete, you can compare tools on equal terms.

#### Why ground truth comes first

**Body:** You cannot measure accuracy without a verified answer for every field. Building ground truth is slow, but it is the asset that lets you compare vendors, tune confidence thresholds and detect regression later. Use a double-labeled subset to find out how often humans disagree, because that rate is the ceiling for any automated result.

**Bullets:**

- Label real documents, including the bad ones
- Keep a separate untouched test set
- Measure per field and per supplier
- Record the document source and scan quality

#### Metrics that matter

**Body:** A single accuracy number hides the problem. Report field-level accuracy, the straight-through rate and the cost of review. A system that reads 98 percent of characters correctly can still post wrong amounts if the errors concentrate in the total field. Pick the confidence threshold by plotting accuracy against the share of documents auto-accepted, and choose the point your business can accept.

#### Common mistakes

**Body:** Teams run the pilot on clean PDFs from the biggest five suppliers and then meet scanned, skewed and photographed documents in production. Others judge the tool by demo runs rather than by a held-out test set. A third mistake is ignoring the review process: if reviewers need three screens and two minutes per document, savings vanish.

**Cta:** If you want help preparing a pilot and running it privately on your own infrastructure, Netray can support the setup and measurement.

**Faq:**

#### How many documents do I need for a document AI pilot?

**Answer:** A few hundred per document type is a practical minimum for field-level metrics, with more for rarer fields. Include enough supplier variety and poor-quality scans, since representative variety matters more than raw volume.

#### What is ground truth in OCR testing?

**Answer:** It is the verified correct value of every field for each test document, usually entered or checked by a person. Extraction output is scored against it to measure accuracy, precision and recall.

#### What should the straight-through rate target be?

**Answer:** It depends on document mix and risk tolerance, so set it from your baseline and the measured accuracy curve rather than from a vendor claim. Many teams start with a conservative threshold and raise it as evidence accumulates.

#### Can I pilot without sending documents outside my network?

**Answer:** Yes. Open-source OCR and open-weight models can run on your own servers. Decide this early, because it affects tool choice, hardware needs and the approval path for documents with export-controlled or confidential content.

## Search terms

document ai pilot checklist, ocr pilot plan, idp proof of concept checklist, invoice ocr pilot success criteria, intelligent document processing implementation checklist, ap automation pilot checklist, ocr proof of concept requirements

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json