Over the past 18 months, the approach for CISOs regarding generative AI has been straightforward: manage browser activity.
Security teams have reinforced cloud access security broker (CASB) policies, restricted or monitored traffic to prominent AI endpoints, and ensured usage passed through authorized gateways. The strategy was to observe, log, and halt any sensitive data leaving the network via external API calls. However, this strategy is beginning to fail.
There’s a subtle shift in hardware that is moving large language model (LLM) usage from the network to the endpoint, ushering in what’s known as Shadow AI 2.0, or the “bring your own model” (BYOM) era. Employees are now running powerful models directly on their laptops, offline, without API calls or noticeable network signatures. While discussions around governance still focus on “data exfiltration to the cloud,” the immediate risk for enterprises is increasingly about “unvetted inference inside the device.”
When inference is conducted locally, traditional data loss prevention (DLP) systems can’t detect the interaction. If security teams can’t see it, they can’t manage it.
Why local inference is suddenly practical
Running a functional LLM on a work laptop was a rare feat just two years ago. Now, it’s commonplace for technical teams.
Three factors have converged:
-
Consumer-grade accelerators have advanced: A MacBook Pro with 64GB of unified memory can now run quantized 70B-class models at practical speeds, though with some limitations on context length. Tasks that once required multi-GPU servers can now be executed on a high-end laptop.
-
Quantization has become mainstream: Compressing models into smaller, faster formats that fit within laptop memory is now easy, with quality tradeoffs that are often acceptable for numerous tasks.
-
Distribution is seamless: Open-weight models are available with a single command, and the tooling ecosystem makes the process of “download → run → chat” straightforward.
Outcome: An engineer can download a multi-GB model artifact, disconnect from Wi-Fi, and execute sensitive workflows locally, such as source code reviews, document summarizations, drafting customer communications, and exploratory analysis over regulated datasets. This activity leaves no outbound packets, proxy logs, or cloud audit trails.
From a network security viewpoint, such activities might appear as if “nothing happened.”
The risk isn’t only data leaving the company anymore
Why should a CISO be concerned if data isn’t leaving the laptop?
The focus shifts from data exfiltration to integrity, provenance, and compliance risks. Local inference introduces three classes of blind spots that most businesses have yet to address.
1. Code and decision contamination (integrity risk)
Local models are often chosen for their speed, privacy, and because they require no approval. However, they are frequently unvetted for enterprise environments.
Typical scenario: A senior developer downloads a community-tuned coding model due to its impressive benchmarks. They input internal authentication logic, payment flows, or infrastructure scripts to “optimize” them. The model outputs results that seem competent, compile, and pass unit tests but subtly weaken security (e.g., weak input validation, unsafe defaults, brittle concurrency changes, and disallowed dependency choices). The developer implements these changes.
If this interaction occurred offline, there might be no record of AI influencing the code path. During incident response, the symptom (a vulnerability) would be investigated without visibility into the root cause (uncontrolled model usage).
2. Licensing and IP exposure (compliance risk)
Many high-performance models come with licenses that include restrictions on commercial use, attribution requirements, field-of-use limitations, or obligations that conflict with proprietary product development. When employees run models locally, this usage can bypass the organization’s typical procurement and legal review processes.
If a team utilizes a non-commercial model to produce code, documentation, or product behavior, the company could inherit risks that emerge later during M&A diligence, customer security reviews, or litigation. The main issue is not just the license terms but also the lack of inventory and traceability. Without a governed model hub or usage record, proving what was used where might be impossible.
3. Model supply chain exposure (provenance risk)
Local inference also changes the software supply chain dilemma. Endpoints begin accumulating large model artifacts and the associated toolchains: downloaders, converters, runtimes, plugins, UI shells, and Python packages.
A significant technical nuance is the file format. Newer formats like Safetensors are designed to prevent arbitrary code execution, while older Pickle-based PyTorch files can execute malicious payloads when loaded. If developers download unvetted checkpoints from Hugging Face or other repositories, they might be downloading not just data but also an exploit.
Security teams have long treated unknown executables as hostile. BYOM extends this mindset to model artifacts and the associated runtime stack. The biggest organizational gap today is the absence of a software bill of materials for models, including provenance, hashes, allowed sources, scanning, and lifecycle management.
Mitigating BYOM: treat model weights like software artifacts
Local inference challenges can’t be solved by simply blocking URLs. Endpoint-aware controls and a developer experience that facilitates safe paths are necessary.
Here are three practical measures:
1. Move governance to the endpoint
While network DLP and CASB remain crucial for cloud usage, they don’t suffice for BYOM. Treat local model usage as an endpoint governance issue by tracking specific signals:
-
Inventory and detection: Look for indicators like .gguf files over 2GB, processes like llama.cpp or Ollama, and local listeners on ports such as 11434.
-
Process and runtime awareness: Monitor repeated high GPU/NPU (neural processing unit) usage from unauthorized runtimes or unknown local inference servers.
-
Device policy: Implement mobile device management (MDM) and endpoint detection and response (EDR) policies to control the installation of unauthorized runtimes and enforce baseline hardening on engineering devices. The goal isn’t to stifle experimentation but to regain oversight.
2. Provide a paved road: An internal, curated model hub
Shadow AI often results from friction. Approved tools might be too restrictive, generic, or slow to approve. Offer a curated internal catalog that includes:
-
Approved models for common tasks (coding, summarization, classification)
-
Verified licenses and usage guidance
-
Pinned versions with hashes (prioritizing safer formats like Safetensors)
-
Clear documentation for safe local usage, specifying where sensitive data can and cannot be used. Providing a superior alternative to scavenging can steer developers away from risky practices.
3. Update policy language: “Cloud services” isn’t enough anymore
Most acceptable use policies focus on SaaS and cloud tools. BYOM necessitates policy language that explicitly addresses:
-
Downloading and running model artifacts on corporate endpoints
-
Acceptable sources
-
License compliance requirements
-
Rules for using models with sensitive data
-
Retention and logging expectations for local inference tools. The policy doesn’t need to be overly restrictive, but it should be clear and precise.
The perimeter is shifting back to the device
For years, security controls were moved “up” into the cloud. Now, local inference is drawing a significant portion of AI activity back “down” to the endpoint.
Here are five indications that shadow AI has transitioned to endpoints:
-
Large model artifacts: Unexplained storage use by .gguf or .pt files.
-
Local inference servers: Processes listening on ports like 11434 (Ollama).
-
GPU utilization patterns: Spikes in GPU usage while offline or disconnected from a VPN.
-
Lack of model inventory: Inability to trace code outputs back to specific model versions.
-
License ambiguity: Presence of “non-commercial” model weights in production builds.
Shadow AI 2.0 isn’t a future possibility but a foreseeable result of advanced hardware, effortless distribution, and developer demand. CISOs who concentrate solely on network controls risk overlooking the activities occurring on the devices right in front of employees.
The next stage of AI governance involves less emphasis on blocking websites and more focus on managing artifacts, provenance, and policy at the endpoint, all while maintaining productivity.
Jayachander Reddy Kandakatla is a senior MLOps engineer.
