Most enterprise AI governance frameworks were designed with a specific threat model in mind: data leaving the organization through cloud-based AI services. The concern was that employees would paste sensitive information into ChatGPT or Claude, and that data would be used to train future models or exposed through a breach. That concern is legitimate, and most organizations have addressed it, through acceptable use policies, data classification frameworks, and in some cases technical controls that block access to consumer AI services on corporate networks.
But there is a different threat model that most of those frameworks were not designed to handle, and it is growing fast. Open-weight models, models whose weights are publicly available and can be downloaded and run locally, are now capable enough to be genuinely useful for a wide range of enterprise tasks. Llama, Mistral, Phi, and their derivatives can run on a laptop with a modern GPU. They can be fine-tuned on proprietary data. They can be integrated into local workflows without touching any external API. And they leave no trace in any cloud service log.
This is the edge AI governance problem. And most organizations are not ready for it.
The traditional governance controls do not apply. You cannot block access to a model that runs locally. You cannot audit API calls that never leave the device. You cannot enforce data residency requirements on a process that has no network dependency. The entire control architecture that enterprise security teams built for cloud AI is simply irrelevant to a model running on an employee's workstation.
I want to be clear that I am not arguing against open-weight models. They have significant advantages, privacy, cost, latency, customizability, that make them genuinely valuable for certain enterprise use cases. The problem is not the technology. The problem is that most organizations are deploying it, or allowing it to be deployed, without the governance infrastructure to manage it responsibly.
What does responsible edge AI governance look like? It starts with visibility. You cannot govern what you cannot see. Organizations need to understand which open-weight models are being used, by whom, for what purposes, and on what data. That requires a combination of policy (employees must disclose AI tool usage), process (teams must document AI-assisted work), and in some cases technical controls (device management policies that can detect and log AI model installations).
It also requires a different kind of risk assessment. The risks of edge AI are not primarily about data exfiltration, the data stays on the device. The risks are about output quality, bias, and accountability. When an employee uses a locally-run model to draft a document, analyze data, or make a recommendation, who is responsible for the quality of that output? How do you audit it? How do you ensure that the model being used has not been fine-tuned in ways that introduce bias or error?
These are not hypothetical questions. They are questions that governance frameworks need to answer before edge AI becomes as ubiquitous as cloud AI already is. The organizations that get ahead of this will have a significant advantage, not just in risk management, but in their ability to deploy edge AI responsibly and capture its benefits. The ones that wait will be managing incidents instead of building capabilities.