From AI Pilot to Production: A Manufacturing Playbook
Moving an AI pilot to production means hardening three things the pilot deliberately skipped: the integration layer, the operational controls, and the ownership model. Most manufacturing AI pilots succeed technically and then die anyway, because a pilot runs on a developer laptop with a personal ERP login, no monitoring, no on-call, and no named owner after the consultant leaves. Production means a service account with least-privilege scope, promotion through dev, test, and production environments under change control, monitored error and escalation rates, a tested rollback, and a person whose job description includes the agent. Plan for six to ten weeks of hardening after the pilot proves out.
Why AI Pilots Stall Before Production
The common failure is not accuracy, it is that nobody scoped the operational work. A pilot proves the agent can classify a nonconformance report. Production requires answering who gets paged at 2 a.m. when the classification service returns 500s, how the agent behaves during your quarter-end close when the ERP is under load, what happens when IT rotates the service account credential, and which change control board approves a prompt update. Pilots also frequently use data extracts rather than live integration, so the first real production week surfaces record variants nobody saw. Budget the hardening work explicitly at pilot approval time, or the pilot becomes a permanent demo.
Hardening the ERP Integration Layer
Replace whatever expedient access the pilot used. In SyteLine or CloudSuite Industrial that means a dedicated IDO service account with permissions scoped to named IDOs and methods, called through the IDO Request Service with proper session handling rather than screen scraping. In Infor LN or M3, route through ION APIs or the API gateway with token refresh handled correctly and rate limits respected. Add retry with exponential backoff, idempotency keys on any write, circuit breaking when the ERP is degraded, and structured logging of every request and response. Test the agent against a refreshed copy of production data, not a two-year-old sandbox, because record shapes drift.
- Dedicated least-privilege service account per environment, credentials in a secrets manager not config files
- Idempotency keys on every write so retries cannot create duplicate ERP transactions
- Exponential backoff plus circuit breaker so a degraded ERP does not amplify into a request storm
- Integration tests run against a recent production data refresh, refreshed at least quarterly
Environments, Promotion, and Change Control
Treat agent artifacts as code. Prompts, tool definitions, evaluation sets, and model version pins all live in Git and promote through the same dev, test, and production path as your ERP customizations. A prompt change is a code change, and it needs a diff, a reviewer, and an evaluation run before it reaches users. Pin model versions explicitly rather than tracking a floating latest alias, because a silent provider-side model update can shift behavior overnight on a system your planners depend on. Keep a rollback that is one deployment away, and rehearse it once before go-live so the first execution is not during an incident.
- Prompts, tool schemas, and evaluation sets versioned in Git with pull request review
- Explicit model version pinning, never a floating latest alias in production
- Automated evaluation suite gating promotion: no regression on the golden set, no deploy
- A rehearsed rollback path executable within 15 minutes by someone who did not build the agent
Monitoring, Rollback, and Day One Ownership
Instrument four signals from the first production hour: task volume, escalation rate, tool error rate, and end-to-end latency at the 95th percentile. Alert on relative change, not absolute values, because a jump in escalation rate from 8 to 25 percent is your earliest warning that something upstream shifted. Add sampled human review of 5 to 10 percent of accepted outputs indefinitely, not just during the first month, since that is how you catch slow drift. Name an owner with a documented runbook, an on-call path, and the authority to disable the agent unilaterally. If nobody can turn it off without a meeting, it is not production-ready.
How Netray Runs Pilot-to-Production Cutovers
Netray treats production readiness as a defined gate with a written checklist covering integration hardening, promotion pipeline, monitoring, rollback rehearsal, and named ownership, and we do not schedule a go-live date until every item passes. We hand over the evaluation harness, the runbook, and the dashboards so your team operates the agent without a retainer dependency. For on-premises deployments in aerospace and defense environments, we deliver the same pipeline entirely inside your boundary using self-hosted models and your existing change control. Typical hardening runs six to ten weeks after a successful pilot, and the checklist is what makes that estimate hold.
Frequently Asked Questions
Why do most AI pilots never reach production?
Because the operational work was never scoped. Pilots typically run with personal credentials, data extracts instead of live integration, no monitoring, no rollback, and no named owner after the project ends. The gap is not model accuracy, it is service accounts, change control, alerting, on-call, and ownership. Budget six to ten weeks of hardening at pilot approval time and the conversion rate changes dramatically.
How long does it take to move an AI agent from pilot to production?
Plan six to ten weeks after the pilot proves out. That covers replacing pilot-grade ERP access with least-privilege service accounts, adding idempotency and retry logic, building the promotion pipeline through dev, test, and production, standing up monitoring and alerting, rehearsing rollback, and transferring ownership with a runbook. Skipping any of these usually costs more time later during an incident.
Should you pin the model version for a production AI agent?
Yes. Track an explicit model version rather than a floating latest alias. Provider-side model updates can shift agent behavior overnight, and if your planners depend on that agent you will find out through a support ticket. Pin the version, rerun your evaluation suite when you choose to upgrade, and treat the model version as a controlled dependency like any other production library.
Key Takeaways
- 1Why AI Pilots Stall Before Production: The common failure is not accuracy, it is that nobody scoped the operational work. A pilot proves the agent can classify a nonconformance report.
- 2Hardening the ERP Integration Layer: Replace whatever expedient access the pilot used. In SyteLine or CloudSuite Industrial that means a dedicated IDO service account with permissions scoped to named IDOs and methods, called through the IDO Request Service with proper session handling rather than screen scraping.
- 3Environments, Promotion, and Change Control: Treat agent artifacts as code. Prompts, tool definitions, evaluation sets, and model version pins all live in Git and promote through the same dev, test, and production path as your ERP customizations.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
AI Pilot-to-Production Readiness Assessment
Score your AI pilot against the ten gates that decide whether it reaches production, and get a prioritized list of the gaps blocking deployment.
Free ToolAI Agent Security Review Checklist
A 30-point security review for AI agents that can call tools and write to business systems, covering identity, permissions, prompt injection, data handling, and audit.
Free ToolOn-Prem AI Security Hardening Checklist
A practical control checklist for securing self-hosted language models, covering model provenance, network isolation, data governance, host hardening, and audit readiness.
Terms used in this article
Have an AI pilot that works but has not shipped? Netray will run a production readiness assessment against your SyteLine, LN, or M3 environment and give you a dated cutover plan.
Related Resources
Your First AI Agent: A Manufacturing Playbook
Your first AI agent in manufacturing: pick a bounded ERP use case, scope data access, set guardrails, and ship a working pilot in 6 to 8 weeks.
AI & AutomationMeasuring AI Agent ROI in Manufacturing
Measure AI agent ROI in manufacturing with baseline cycle times, touch-time savings, error-rate deltas, and a payback model your CFO will actually approve.
AI & AutomationAI Agent Failure Modes and How to Catch Them
AI agent failure modes in manufacturing: silent data drift, tool-call errors, hallucinated part numbers, and the monitoring that catches them before users do.