# ERP AI KPIs: which to track in the first 90 days and which to hold for later > Source: https://netray.co/erp-help/cio-erp-ai-kpis-leading-and-lagging > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: ERP Help ## Short answer Track three layers: adoption and quality as leading indicators (repeat use, answer accuracy on a fixed test set, time to answer), operational outcomes as lagging ones (report hours, expedite count, close days), and cost per active user. Fix baselines before launch, or you will not be able to prove anything. ## Kind cio ## System cio ## Erp ERP AI (any ERP) ## Module Performance measurement ## Query ERP AI KPIs to track ## Alternate Queries - how to measure success of an AI assistant on ERP - AI KPIs for CIO manufacturing - leading and lagging indicators AI pilot ## Applies To Pilot and early production of a read-only ERP assistant or agents. Complements long-run ROI measurement after go-live. ## Steps ### Set up a KPI scorecard before the pilot starts **Items:** - Pick two or three use cases and write down the manual baseline for each: who does it, how often, how long it takes, and the error or rework rate. Measure for two to four weeks before launch. - Define leading indicators: weekly active users by role, repeat use rate, share of questions answered without escalation, and median time to answer. - Build a fixed test set of 50 to 150 real questions with known correct answers, and score accuracy on it before launch and after every model, retrieval or ERP change. - Define lagging indicators tied to the use case: planner hours on shortage reporting, number of manual expedite emails, quote turnaround, days to close, audit request turnaround. - Track trust signals: thumbs down rate, manual corrections, and the share of answers where users open the source or query. - Track cost: total run cost divided by active users, and cost per resolved question for ranges of load. - Set targets as ranges after two weeks of data, not before, and agree stop and scale thresholds with the sponsor. - Report monthly on one page: leading, lagging, cost, and decisions requested. ## Sections ### Leading indicators: adoption and quality **Body:** Leading indicators move first and tell you whether outcomes are likely. The most useful are repeat use by the intended role, accuracy on your own test set, and time to answer versus the manual route. Total logins and prompt counts are weak, because curiosity inflates them early and they say nothing about value. Accuracy needs definition. For ERP questions it usually means the number or list returned matches what a careful analyst would get from the system for the same filters. Score by comparing to queries a human writes, and record separately: wrong, partly right, right but unclear. Keep the test set stable so scores are comparable over time, and refresh a portion each quarter with newly observed questions. Escalation rate (questions the assistant declines or users abandon) is healthy at first. A tool that never declines is probably guessing. Look for a falling rate of wrong answers, not a zero rate of refusals. ### Lagging indicators: outcomes you can defend **Body:** Pick outcomes the use case can plausibly influence and that you already measure. Planner time on shortage and exception reports, number of manually built reports requested from IT, days to answer an audit sample request, quote turnaround time, and days to close are typical examples. If you do not measure it today, you cannot show improvement later. Be careful about attribution. A drop in expedite orders may follow better supplier performance or lower demand, not your assistant. Use before and after comparisons for the same teams, hold other changes constant where you can, and say plainly when evidence is suggestive rather than proof. Treat soft benefits honestly. Fewer interruptions to the ERP super user or the IT report writer is real, but record it as a count of requests avoided rather than inflating it into a monetary figure. **Code:** One-page monthly scorecard Leading: weekly active users by role | repeat-use % | test-set accuracy | median time to answer | escalation % Trust: thumbs-down % | source-opened % | corrections Lagging: use-case outcome 1 | outcome 2 | outcome 3 (vs baseline) Cost: run cost / active user | cost / question Decisions: scale | fix | stop (with reason) ### KPIs that look good and mislead **Body:** Avoid vanity measures such as 'hours saved' computed by multiplying a guessed minutes-per-query by query count. They rarely survive a finance review. Prefer measured elapsed time on a sample of tasks, before and after, with a short note on method. Avoid targets set before baselines exist and avoid a single composite score that hides which part is failing. A scorecard where adoption is high but accuracy is poor needs a different action than the reverse, and a blended number conceals that. Finally, tie KPIs to decisions. Each metric should have a threshold that triggers scale, fix or stop. If no one would change course based on a number, drop it from the scorecard. ## Pitfalls - No pre-launch baseline, so improvement can never be shown. - Counting prompts or logins as success. - Changing the accuracy test set between reviews, which breaks comparability. - Claiming savings from guessed time per query. - Attributing an operational improvement to the assistant without controlling for other changes. - Reporting a blended score that hides a failing component. ## Ai Angle A grounded assistant such as ERPray can help compute some baselines directly from the ERP: past-due purchase order counts by buyer, expedite activity, days from receipt to invoice posting, or backlog by planner. Because answers show the query, your scorecard figures are reproducible each month by the same filters. The assistant supports measurement, but the metric definitions and targets remain your decision. ## Faq ### What is the most important KPI in the first month? **Answer:** Repeat weekly use by the intended role, paired with accuracy on a fixed test set. Together they show whether people find the tool useful and whether it deserves trust. Outcome metrics take longer to move and are harder to attribute, so treat them as month two and three evidence. ### How do we measure accuracy for natural language ERP questions? **Answer:** Build a test set of real questions with answers produced by a careful analyst from the ERP. Score each assistant response as right, partly right or wrong, and record why. Rerun after every model, retrieval or ERP change, and add new real questions regularly so the set reflects actual use. ### How is this different from measuring ROI after go-live? **Answer:** KPIs here are operational and short-cycle: adoption, quality and early outcomes that guide pilot decisions. ROI measurement comes later and combines cost, benefits and risk over longer periods for finance. A good KPI scorecard is the evidence base the ROI review will draw on. ### Should we set numeric targets before launch? **Answer:** Set baselines before launch and commit to measuring. Set numeric targets after one or two weeks of real data, then agree scale, fix and stop thresholds with your sponsor. Targets invented before any data exist are usually either trivial or unreachable, and they damage credibility either way. ## Search terms ERP AI KPIs, AI pilot metrics manufacturing, measure AI assistant accuracy, leading lagging indicators AI, AI scorecard CIO, ERP AI adoption metrics, AI test set accuracy, cost per active user AI --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json