When Surveillance Becomes Intelligence and Risk Becomes Invisible

At a wedding event in Bali, Artificial Intelligence (AI) researcher Nicholas Carlini opened his laptop to test Anthropic’s newly released Mythos AI model. What began as an adversarial evaluation, testing whether the model could identify vulnerabilities and generate exploit pathways, quickly escalated into something more ominous. Within hours, the system was not just identifying vulnerabilities; it was exploiting them. It mapped pathways through networks, bypassed controls, and executed coordinated intrusion sequences. This was not AI-assisted analysis; it was human-independent behavior.

For decades, the intelligence behind security systems lived outside the technology. Cameras captured, sensors triggered, and access control systems enforced rules, but interpretation belonged to people. A trained operator, guided by experience and context, determined whether an event mattered and how to respond. That legacy model is shifting. AI and machine learning (ML) have moved elements of that analysis into digital systems, both at the core and the edge. Video analytics identify behaviors. Access control systems flag anomalies. SOCs correlate physical and cyber data. In many environments, AI now detects, prioritizes and sometimes initiates a response.

The objective is clear: better awareness, faster decisions and execution, and the elimination of operator overload. But moving these functions into AI platforms introduces a new class of risk.

AI is becoming embedded across the security ecosystem: static and drone-based cameras interpreting scenes in real time; access control systems evaluating behavior; SIEM/SOAR platforms correlating events; security robots conducting patrols; and agentic systems orchestrating multi-step actions. These systems are no longer passive tools. They are proactive elements, and they’re becoming targets for attack, bypass, or rendering ineffective.

The shift is subtle and consequential. While AI strengthens the security stack, it’s also part of the attack surface. Models can be manipulated, hostile inputs engineered, and outcomes influenced. In some cases, the security system can become an unwitting participant in the threat it was designed to detect. What once required a skilled attacker can now be executed and scaled at machine speed by a less trained person. This isn’t theoretical: researchers, including teams at MIT and Google, have shown that small image manipulations can lead to high-confidence misclassification.

Why It Matters Now: Speed, Scale, and Response Time Shortening

Security leaders have long relied on time as a buffer. A vulnerability is discovered, a patch is issued, and controls are updated. But that buffer is rapidly shrinking. Industry groups, including the Cloud Security Alliance, SANS Institute, and OWASP, warn that enterprises are entering a phase where attackers can identify and exploit vulnerabilities faster than defenders can respond. Put simply, capability is rising while the barrier to entry falls.

Simultaneously, the timeline from vulnerability discovery to exploitation is nearing zero. A flaw in a video platform, access control integration, or intrusion detection system can be identified and weaponized almost immediately. In converged environments, cross-platform exploits compound risk while threat behaviors continuously change.

Rather than isolated attacks, organizations face persistent probing [1]. AI-enabled systems can continuously test environments, adjusting tactics based on what succeeds or fails and how target systems respond. This creates an ongoing cycle of discovery, refinement, and exploitation. For physical security, this is a pointed concern. Vulnerabilities in cameras, APIs, and system integrations are no longer difficult to uncover; they can be discovered and exploited at machine speed.

The more significant shift is not simply response speed, but how risk must be understood and managed. AI-enabled systems operate across interconnected platforms, where behavior, data, and decision-making become tightly coupled. This warrants enterprise-level risk management. AI capabilities can no longer be treated as isolated features; they should be managed within an Enterprise Security Risk Management (ESRM) framework that includes lifecycle oversight, vendor accountability, and continuous validation.

The challenge is understanding how intelligent systems, both defensive and adversarial, interact in real time and shape operational risk.

When AI Learns the Wrong Lesson

Adversarial attacks are old news, and they still rely on manipulating inputs and exploiting weaknesses. What has changed is where new weaknesses exist: within the model itself. With AI-based systems, decision-making is no longer based on binary logic. It’s learned behavior, shaped by training and operational data. That introduces new exploitation points. Adversarial inputs are one example. Manipulation of visual and sensor data can cause a system to misclassify its observations. In physical security, that could mean a camera missing a threat or misreading benign activity.

The system operates as designed, but its ability to interpret correctly has been corrupted. Data poisoning operates over time. By manipulating training data, attackers can shape how a model learns, leading to gradual degradation. A system may appear functional but become less reliable in distinguishing normal from abnormal conditions.

Model exploitation techniques go further, allowing attackers to replicate system logic through observation of outputs and, in some cases, to expose elements of the underlying training data, thereby gaining insight into how decisions are made and how to bypass them.

Modern AI systems can identify vulnerabilities, test attack paths, and execute multi-step strategies with minimal human input, including reconnaissance, exploitation, and persistence. For example, an AI system could identify a weak integration between a video platform and an access control system, test input variations, and determine which combinations produce the desired effect. Once identified, the pathway can be exploited repeatedly and at scale.

These systems are not simply making mistakes; they are being manipulated to achieve attacker objectives. Because they operate within trusted systems, they can influence operational decisions. In this context, “learning the wrong lesson” is not a system error; it’s manipulation in which learned behavior is shaped to enable an attacker’s objectives rather than operational expectations.

When AI Risk Becomes Operational Risk

In converged security environments, AI serves as the intelligence layer, correlating data from video systems, access control, identity platforms, network activity, and facility operations. This integration enhances situational awareness but also expands the attack surface and introduces new points of failure.

In video analytics, real-world conditions and intentional manipulation can distort how the system interprets what it sees. A genuine threat may go undetected, while a benign situation may trigger unnecessary escalation. In both cases, the system operates as designed, but its interpretation of the scene has been influenced. The risk is amplified by confidence complacency. When a system presents a high-confidence assessment, operators are less likely to question it, even when it is wrong. Access control systems face a different risk. AI behavioral analysis can be manipulated to accept abnormal patterns or flag legitimate activity. In both cases, trust degrades, often without immediate visibility.

AI robotics extends this risk into the physical world. Autonomous security patrol robots rely on AI to navigate, interpret, and report. When those models fail, the consequences are physical. A hazard may be missed, a condition misreported, or an incorrect action triggered. These are failures in decision logic, whether caused by model limitations or manipulation.

Autonomous exploitation also expands impact. AI systems can scan environments, identify vulnerabilities across cyber and physical interfaces, and exploit them rapidly. In converged systems, those exploits can propagate, creating cascading effects difficult to contain.

Where Legacy Security Thinking Falls Short

Many traditional security assumptions do not fully apply to AI-enabled systems. Threats have historically been treated as relatively stable, but AI introduces adaptive behavior, making static threat models less effective. Perimeter-based thinking also breaks down. Risk is no longer confined to external actors; it can originate within system behavior, data dependencies, and integrations.

There is also a flawed assumption that systems are “secure” once incorporated into a lifecycle management program. With AI-enabled platforms, that assumption is risky because model behavior, training data, and update cycles influence outcomes, often without full transparency. A critical gap exists at the model level; traditional controls protect infrastructure, but not decision logic. Most organizations lack standardized processes to validate how models behave under stress or degrade over time. Standards are emerging but remain immature, while vendors face pressure to deliver capabilities quickly, creating tension between safety and speed.

Organizational structure can compound the issue, as responsibility for AI-enabled systems is often distributed across IT, cybersecurity, physical security, and emerging data functions, resulting in fragmented ownership and accountability that does not align with exposure.

What This Looks Like in Practice

Addressing these risks requires a structured, programmatic approach in which manufacturers and end users collaborate on model risk management. Model validation should become standard practice, with systems tested under real-world conditions, including edge cases and AI-adversarial scenarios. The goal is to understand failure modes, not just confirm functionality. Red teaming is essential, with AI-enabled systems continuously tested against realistic attack scenarios across cyber and physical domains.

Architecturally, AI systems should be managed as critical infrastructure, with segmentation, controlled interfaces, and monitored data flows to limit exposure. Operational monitoring must also evolve as models change over time and introduce drift; organizations need mechanisms to detect degradation in performance and decision quality.

Human oversight remains mandatory and intentional, with operators equipped to interpret system behavior, challenge outputs, and intervene when necessary. Vendor accountability must also increase, with organizations demanding transparency around training data, testing practices, and updating cycles, supported by contractual terms.

These practices should align with broader ESRM frameworks. AI is not a standalone capability; it’s part of the enterprise risk landscape.

What This Means for the CSO

For security leaders, the implications of AI-enabled risk are operational and immediate. The first step is recognizing that AI models are not merely features within systems; they are critical assets that require the same oversight as infrastructure, applications, and networks.

This requires expanding the definition of security risk. AI behavior, model integrity, and data dependencies must be incorporated into enterprise risk frameworks. In practical terms, this means aligning AI-enabled systems with existing ESRM programs and governing them throughout their lifecycle.

Equally important is breaking down traditional silos. AI-enabled systems sit at the intersection of cybersecurity, IT, and physical security. Gaps in visibility and accountability are inevitable without coordinated ownership. CSOs are uniquely positioned to drive shared responsibility across functions and ensure AI risk is managed at the appropriate organizational level.

Performance measurement must also shift. Historically, AI systems have been evaluated on accuracy and efficiency. While important, these metrics are insufficient on their own. Security leaders must also consider reliability, integrity, and explainability: how consistently a system performs, how resistant it is to manipulation, and whether its decisions are understandable and trustworthy.

Intelligence, Trust, and Control

Artificial intelligence is reshaping security systems in ways difficult to imagine even a few years ago. It elevates visibility, accelerates response times, and enables situational awareness not attainable in non-AI systems.

At the same time, it introduces subtle, often invisible risks embedded within the systems designed to provide protection. The convergence of cyber, physical, and AI-enabled security means these risks cannot be managed in isolation. They must be understood as part of an integrated environment where system behavior, data, and decision-making are interconnected.

The central challenge is not whether to deploy AI, but how to govern it. Machine learning amplifies capability but does not replace accountability. Systems may learn, adapt, and act, but organizations remain responsible for how those systems are designed, deployed, and managed.

The future of security will not be defined by how intelligent systems become, but by how well that intelligence is understood, controlled, and trusted.