Security Review Checklist for Enterprise AI Agents
A security review for an enterprise AI agent covers four surfaces: where the data goes, what the agent is permitted to do, what untrusted input can make it do, and what evidence exists afterward. Traditional application security review misses the third surface entirely, because prompt injection has no equivalent in conventional software. For manufacturers handling export-controlled technical data under ITAR or EAR, or Controlled Unclassified Information under DFARS 252.204-7012 and CMMC 2.0, the data residency question usually determines architecture before any other requirement is discussed.
Scope the Review: Model, Data Path, and Tool Permissions
Start by drawing the full data path on one page: what leaves the user, what reaches the model, what the model can call, and what is retained where. Include retrieval sources, since an agent with access to an engineering document repository effectively inherits that repository's classification. Enumerate every tool the agent can invoke and the permissions behind each one. A common finding is an agent granted a broad ERP integration account because it needed one write, giving it read access to payroll or supplier pricing as a side effect. Review the retention and training policy of every third-party endpoint in the path, including embedding and reranking services people forget are separate vendors.
Data Residency and On-Premises Deployment Questions
For defense suppliers this is usually the binding constraint. Export-controlled technical data under ITAR cannot be processed by a service where non-US persons may access it, which rules out most general commercial APIs regardless of contractual assurance. CUI under DFARS 252.204-7012 requires a compliant cloud offering at FedRAMP Moderate equivalence or better, or on-premises processing inside your assessed boundary. On-premises open-weight models running on your own GPUs remove the question entirely and are now practical for most extraction, classification, summarization, and ERP-assistant workloads. Establish the classification of the data before selecting the model, not after the proof of concept is already built on a commercial API.
- Classify the data first: ITAR or EAR technical data, CUI, PII, or general business information
- Confirm FedRAMP Moderate equivalence and a documented CUI enclave before any cloud path for DFARS data
- Verify no-training and zero-retention terms in writing, including embedding and reranking subprocessors
- For ITAR workloads, plan on-premises open-weight models inside your assessed boundary as the default
Prompt Injection and Tool Abuse Controls
Any text the agent reads is potential instruction. A supplier email, a PDF drawing note, or a free-text field in a customer record can carry an instruction that a naive agent will follow. The mitigation is architectural, not prompt-based: treat retrieved and user-supplied content as data with clear delimiters, keep tool permissions minimal enough that a successful injection cannot cause serious harm, and require human approval for any consequential action. Add allowlists on outbound calls so an agent cannot be induced to exfiltrate data to an arbitrary URL. Test injection explicitly during evaluation with a corpus of attack strings rather than assuming your system prompt holds.
- Treat all retrieved and user-supplied text as untrusted data, never as instruction
- Least-privilege tool scoping so a successful injection cannot reach payroll, pricing, or supplier bank data
- Outbound allowlist on network calls to block exfiltration to attacker-controlled endpoints
- Injection test corpus run as part of every evaluation cycle, with results gating promotion
Identity, Logging, and Vendor Risk
Give the agent its own identity in your directory, not a borrowed human account, so its actions are attributable and its permissions are reviewable in an access recertification. Rotate credentials from a secrets manager and confirm the agent handles rotation without a manual restart. Log every prompt, tool call, and response with a correlation ID, and retain to the same standard as the underlying business record. On vendor risk, ask for a SOC 2 Type II report, subprocessor list, breach notification terms, and explicit contractual language on training use, then confirm what happens to your data if the vendor is acquired or shuts the product down.
How Netray Passes Security Review in Regulated Plants
Netray defaults to on-premises deployment for aerospace and defense clients, running open-weight models on customer-owned GPUs inside the assessed boundary so no technical data, drawing, or CUI leaves the network and no third-party endpoint appears in the data path. We deliver a documented data flow diagram, tool permission matrix, injection test results, and logging schema as project artifacts, which is what security and compliance teams actually ask for. We run the prompt injection test corpus as part of the standard evaluation cycle and publish the results alongside accuracy numbers, so the security team sees evidence rather than assurances. That package is why our agents typically clear internal security review in weeks instead of stalling for a quarter while architecture questions get answered retroactively.
Frequently Asked Questions
Can defense contractors use commercial AI APIs with ITAR data?
Generally no. ITAR-controlled technical data cannot be processed where non-US persons may access it, and most general commercial API offerings cannot contractually guarantee that. CUI under DFARS 252.204-7012 requires FedRAMP Moderate equivalence or on-premises processing inside your assessed boundary. The practical answer for most defense suppliers is running open-weight models on their own GPUs, which removes the data residency question entirely.
What is prompt injection and how do you defend against it?
Prompt injection is when untrusted text the agent reads, such as a supplier email or a PDF note, contains instructions the agent follows. Defense is architectural rather than prompt-based: treat all retrieved content as data with clear delimiters, scope tool permissions so a successful injection cannot cause serious harm, allowlist outbound network calls to prevent exfiltration, and require human approval for consequential actions. Test with an attack corpus every evaluation cycle.
What should an AI agent security review checklist include?
Five areas: a one-page data flow diagram showing everything that leaves your boundary, data classification with matching residency controls, a tool permission matrix using a dedicated directory identity rather than a borrowed account, prompt injection testing results with an outbound allowlist, and logging of every prompt, tool call, and response with a correlation ID retained to the same standard as the business record.
Key Takeaways
- 1Scope the Review: Model, Data Path, and Tool Permissions: Start by drawing the full data path on one page: what leaves the user, what reaches the model, what the model can call, and what is retained where. Include retrieval sources, since an agent with access to an engineering document repository effectively inherits that repository's classification.
- 2Data Residency and On-Premises Deployment Questions: For defense suppliers this is usually the binding constraint. Export-controlled technical data under ITAR cannot be processed by a service where non-US persons may access it, which rules out most general commercial APIs regardless of contractual assurance.
- 3Prompt Injection and Tool Abuse Controls: Any text the agent reads is potential instruction. A supplier email, a PDF drawing note, or a free-text field in a customer record can carry an instruction that a naive agent will follow.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
AI Agent Security Review Checklist
A 30-point security review for AI agents that can call tools and write to business systems, covering identity, permissions, prompt injection, data handling, and audit.
Free ToolSovereign AI Readiness Assessment
Score your organization across eleven dimensions of sovereign AI readiness, from data residency and model provenance to cleared personnel and air-gapped operations.
Free ToolOn-Prem AI Security Hardening Checklist
A practical control checklist for securing self-hosted language models, covering model provenance, network isolation, data governance, host hardening, and audit readiness.
Terms used in this article
Need an AI agent that clears security review in a CMMC or ITAR environment? Netray designs on-premises agents with the data flow and control documentation your assessors expect.
Related Resources
Human-in-the-Loop Design for ERP AI
Human-in-the-loop design for ERP AI: where to place approvals, confidence thresholds, and audit trails so agents act safely inside SyteLine, LN, and M3.
AI & AutomationBuilding an AI Center of Excellence in Manufacturing
Build an AI center of excellence in manufacturing: staffing, intake and prioritization, reusable agent patterns, and funding models that survive budget cycles.
AI & AutomationHow to Evaluate an AI Agent Before You Trust It
An AI agent evaluation framework for manufacturers: golden test sets, accuracy thresholds, latency budgets, and escalation rules before production.