# AI Pilot Success Metrics Scorecard Worksheet > Source: https://netray.co/tools/ai-pilot-success-metrics-scorecard-worksheet > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer A weighted scorecard that shows how far a pilot has moved each metric from baseline toward target, with an overall progress figure. ## Category ai-automation ## Audience - CIO - IT manager - Project sponsor - Business owner ## Tool **Kind:** worksheet **Mode:** editable **Columns:** #### Metric **Key:** metric **Type:** text #### Unit **Key:** unit **Type:** select **Options:** - minutes - hours - percent - count - currency - score #### Baseline **Key:** baseline **Type:** number **Format:** decimal #### Target **Key:** target **Type:** number **Format:** decimal #### Actual **Key:** actual **Type:** number **Format:** decimal #### Weight **Key:** weight **Type:** number **Format:** integer **Help:** Relative importance, any positive number #### Progress to target % **Key:** progress **Type:** number **Expr:** clamp(if(eq(target, baseline), 0, (actual - baseline) / (target - baseline) * 100), 0, 120) **Format:** percent **Help:** 0 = at baseline, 100 = at target, capped at 120. Works when lower is better. #### Weighted progress **Key:** weighted **Type:** number **Expr:** weight * clamp(if(eq(target, baseline), 0, (actual - baseline) / (target - baseline) * 100), 0, 120) **Format:** decimal #### Evidence source **Key:** source **Type:** text **Rows:** **Metric:** Time to answer a supplier delivery question **Unit:** minutes **Baseline:** 18 **Target:** 4 **Actual:** 6 **Weight:** 30 **Source:** Timed sample of 40 buyer questions **Metric:** Answer accuracy on test set **Unit:** percent **Baseline:** 0 **Target:** 90 **Actual:** 84 **Weight:** 30 **Source:** 120-question test set graded by buyers **Metric:** Weekly active users in pilot group **Unit:** count **Baseline:** 0 **Target:** 25 **Actual:** 19 **Weight:** 15 **Source:** Application log **Metric:** User satisfaction **Unit:** score **Baseline:** 2.5 **Target:** 4.2 **Actual:** 4 **Weight:** 15 **Source:** End-of-pilot survey, 1-5 scale **Metric:** Data exposure incidents **Unit:** count **Baseline:** 0 **Target:** 0 **Actual:** 0 **Weight:** 10 **Source:** Security log review **Totals:** #### Total weight **Key:** weight **Agg:** sum **Format:** integer #### Total weighted progress **Key:** weighted **Agg:** sum **Format:** decimal #### Metrics tracked **Key:** metric **Agg:** count **Format:** integer **Summary:** #### Overall progress to target **Key:** overall **Expr:** sum_weighted / sum_weight **Format:** percent **Primary:** true **Description:** Weighted average of capped progress per metric **Notes:** Agree baselines, targets and weights with the business owner before the pilot starts. Safety style metrics, where target equals baseline, show 0 progress by design: track them as pass or fail in your go/no-go decision, not through this score. ## Content **Intro:** AI pilots drift when success was never defined. This scorecard asks for a baseline, a target and an actual for each metric, converts them to progress toward target, and weights them into one number for the go or no-go review. **Sections:** #### How to use the worksheet **Body:** List five to eight metrics before the pilot begins. Measure the baseline first, from real work, not estimates. Set a target the business owner signs off, give each metric a weight, and name the evidence source. At the end of the pilot, enter the actuals. Progress is calculated as (actual minus baseline) divided by (target minus baseline), so it works whether lower or higher is better. Values are capped between 0 and 120 percent so one outlier cannot hide the rest. #### Choosing good metrics **Body:** Mix speed, quality, adoption and risk. A pilot that saves time but returns wrong answers has not succeeded, and one that is accurate but unused has not either. **Bullets:** - Time to complete a defined task, measured on a sample - Accuracy on a fixed test set graded by domain experts - Weekly active users in the pilot group - User satisfaction from a short survey - Incidents, such as data exposure or wrong ERP actions - Cost per interaction when running on your own GPUs #### Reading the overall score **Body:** Treat the overall progress as a conversation starter, not a verdict. A common approach is to agree thresholds in advance: for example, 80 percent or more supports moving to production planning, 50 to 80 percent supports a second iteration with a narrower scope, and below 50 percent supports stopping or rethinking the use case. Always check individual rows too. A strong overall number can hide a failure on the one metric the business cares about most, which is what weights are meant to prevent. #### Common mistakes **Body:** Frequent problems are baselines taken from memory, targets set after seeing results, and counting vanity metrics such as number of prompts. Keep the test set stable, so results are comparable if you retest after a model change. Record risk and incident metrics as pass or fail gates alongside the score. **Cta:** If you want a pilot designed around read-only ERP questions with a measurable test set, Netray can help scope the metrics and run the evaluation with your team. **Faq:** #### How do I handle metrics where lower is better? **Answer:** Enter the baseline higher than the target. The progress formula divides the change from baseline by the gap between target and baseline, so improvement shows as positive progress in either direction. #### What weights should I use? **Answer:** Use any positive numbers that reflect relative importance, such as 30, 30, 15, 15, 10. The worksheet divides by total weight, so they do not need to add up to 100. Agree the weights with the sponsor before the pilot. #### Why cap progress at 120 percent? **Answer:** Capping stops a single metric that wildly overshoots its target from masking weak performance elsewhere. You can still see the real actual value in its column when you review the details. #### How long should a pilot run? **Answer:** Long enough to cover a representative range of real questions or transactions, often four to eight weeks for a focused ERP use case. Define the end date and review date in your charter before starting. ## Search terms ai pilot scorecard template, pilot success criteria template, ai pilot kpi worksheet, ai proof of concept evaluation excel, pilot go no go scorecard, ai project success metrics, baseline target actual worksheet --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json