AI Agent Evaluation Test Case Worksheet for ERP Questions
You cannot tell whether an ERP agent is reliable by trying it a few times. You need a fixed question set with known answers, run repeatedly, tracked over releases. This worksheet is that set, with the pass rate computed for you.
| Test ID | User question | Expected tool | Expected answer or value | Test type | Runs | Passes | Pass rate | Last tested | |
|---|---|---|---|---|---|---|---|---|---|
100% | |||||||||
80% | |||||||||
100% | |||||||||
60% |
Run each question several times because model output varies. Keep the expected answer verifiable against an ERP screen or report, and include questions the agent should refuse.
Get a starter question set
Optionally request a starter evaluation set tailored to your ERP and use case.
Optional. The tool and every export work without this. No spam.
How to use the worksheet
Write real questions from the people who will use the agent, not invented ones. For each, record the tool you expect the agent to call and an expected answer you can verify against an ERP screen or report. Run each question several times, record how many runs there were and how many passed, and the sheet computes the pass rate per row and overall. Re-run the whole set after every change to the prompt, the model, the tool descriptions or the schema.
What to include in the set
A useful set mixes types. Lookups confirm basic retrieval. Aggregations catch arithmetic and filter errors. Multi-step questions test tool chaining. Should-refuse cases confirm the agent will not perform unapproved writes. Ambiguous questions check that it asks for clarification rather than guessing. Weighted toward your highest-risk tools, thirty to a hundred cases is a common starting range, growing as production questions fail.
- Add every production failure as a new test case
- Include misspelled part numbers and vendor names
- Include questions the user is not permitted to ask
- Record the model and prompt version next to each run
Reading the results
A single pass rate hides detail, so sort by type. In the example, lookups pass reliably, the aggregation passes four of five and the ambiguous question passes three of five. That pattern points to a tool description problem on the aggregation and a missing clarification instruction on the ambiguous case. Decide a release threshold per type, for example near 100 percent for refusals and lower for open-ended questions, and write it down before testing.
Common mistakes
Testing only with the developer's own phrasing, using expected answers that no one verified and counting a correct answer reached through the wrong tool as a pass are all common. Check the tool call, not just the final text. Also avoid fixing the prompt to pass one case while silently breaking others: always re-run the full set.
Frequently Asked Questions
How many times should I run each question?
Several, commonly three to ten, because model output varies between runs. Fewer runs are acceptable for deterministic settings, but you still want repeats to catch flaky behavior before users do.
What pass rate is good enough?
It depends on risk. Refusals and write-protection should be at or near 100 percent. Read-only lookups often need to be high 90s before rollout. Set thresholds per test type before you start testing.
Should the expected answer be exact text?
Prefer verifiable facts such as a number, a list of record IDs or a decision to refuse, rather than exact wording. Wording varies legitimately between runs.
Can I automate these tests?
Yes. Export to CSV, feed the questions to your agent through its API and compare tool calls and values programmatically. Start by hand to learn which checks matter.
Netray builds agents that show the generated query and call so answers can be checked against the ERP. If you want a starter question set for your system, we can help.
Related Tools
MCP Tool Catalog and Risk Worksheet
Register each MCP tool your agent exposes, the ERP object it touches and a computed risk score, then export to Excel.
Compliance & ContractsAgent Action Audit Log Worksheet
Record agent actions, the tool called, the ERP record affected and the approver, with a computed latency and exception count.
Developer ToolsERP API Field Mapping Worksheet
Map each ERP API field to the parameter an AI tool exposes, with types, required flags and transform rules, then export to Excel.
Compliance & ContractsAI Agent Incident Response Runbook Template
A fill-in incident report for AI agent mistakes in the ERP: timeline, containment, root cause, data affected and corrective actions.
More AI Agents & Automation tools
See all- Business Process Automation ROI Calculator
- AI Agent Use Case Prioritizer
- Document Processing Automation Savings Calculator
- AP Invoice Automation ROI Calculator
- Order Entry Automation Savings Calculator
- Quote-to-Cash Cycle Time Calculator
- Manufacturing AI Readiness Assessment
- RPA vs AI Agent Selector
- ERP AI Maturity Assessment