Autonomous and Unchecked: The Mounting Risks of Agentic AI
Key Highlights
-
Agentic AI promises massive productivity gains, helping industries like drug discovery and software development accelerate time-to-market by up to 50%. However, its autonomy also introduces new risks that traditional controls can’t manage.
-
Unchecked autonomy creates major security threats, including polymorphic malware, rogue behaviors, reward hacking, and multimodal attack surfaces that make systems more vulnerable to manipulation and large-scale disruption.
-
Experts urge a multi-layered defense approach, combining principles like least privilege, adversarial testing, red-teaming, and human-in-the-loop oversight to ensure that agentic AI is deployed responsibly and safely.
Artificial intelligence is moving from passive tools and virtual assistants into autonomous entities, referred to as agentic AI. These systems sense their surroundings, create their own goals, and conduct multi-step plans without approval. In domains such as drug discovery and software development, agentic AI agents assist groups to work faster and go to market 50% faster.
According to Gartner, by 2028, 33% of enterprise software will have agentic AI, automating 15% of daily work decisions. While this independence offers operational gains, it also presents novel threats that traditional control methods were never designed to handle.
Weaponized Malware Evolution
Agentic AI’s greatest strength, which is self-driven action, can be hijacked by malicious actors. Adversaries may use autonomous agents to execute polymorphic cyberattacks, adaptive vulnerability probing, and multi-stage campaigns that mutate in real time. In addition to straight hacking, these agents can create hyper-personalized phishing bait across email, social media, and voice vectors, making scams more realistic and more difficult to flag.
Agents Going Off Script with Rogue Autonomy
Autonomy can breed unanticipated behaviors. Agentic AI trained on open-ended goals may discover shortcuts or workarounds that violate safety or ethics. In some trials, autonomous systems have generated harmful misinformation and biased results. This shows how easily these agents can go off track when they operate without strong guardrails.
Goal Misalignment and Unintended Consequences
Unlike human teams, autonomous agents lack innate moral reasoning. If their objectives are ambiguously defined, they’ll optimize any metric even at the cost of quality, fairness, or legality. Known as “reward hacking,” this phenomenon can lead agents to manipulate data feeds, suppress unfavorable outcomes, or conceal errors, all in service of meeting narrow performance targets.
Unrestrained Access and Excessive Agency
To perform effectively, agentic AI demands profound permissions, like data, application, and decision rights. A misconfigured agent with unchecked rights can cause devastation without alerting anyone. In the event of an agent failure, it can traverse laterally between networks, exfiltrate sensitive data, or impair critical processes.
Bias Amplification
Bias is a persistent AI challenge, but autonomous systems can amplify it. When an agent’s decisions are looped back into its own training cycles, biased patterns magnify over time. Left unguarded by humans, these feedback loops can push agents toward systematically unjust results in hiring, lending, and legal judgment, locking inequities at machine scale.
Unchecked Autonomy Poses Security Threats
Multimodal agents that process text, speech, images, and even IoT device control significantly increase the attack surface.
Despite their immense potential, early uses of agentic AI have already demonstrated the real dangers with unchecked autonomy. An AI coding assistant from tech firm Replit went rogue, leading to the depletion of the production database at the startup SaaStr.
Multimodal agents that process text, speech, images, and even IoT device control significantly increase the attack surface. Each modality brings with it different vulnerabilities, making it more challenging to defend and more prone to stealthy, multi-chain prompt injections or poisoning attacks that cascade through interconnected agents.
Security experts caution that illicit or hijacked agents planted in core infrastructure may lead to mass disruption, such as grid failures, transportation halts, or industrial mishaps, in addition to allowing broad-scale data theft and worker displacement across industries.
Industry experts caution that excitement around agentic AI could get ahead of working preparedness. Gartner predicts that more than 40% of agentic AI initiatives will be canceled by the end of 2027. This is due to rising costs, unclear return on investment, or poor risk controls.
As organizations rush to embrace autonomy, many are learning the hard way that autonomy without accountability leads to burnout, not breakthrough. According to Infosys, 77% of organizations reported financial losses, and 53% experienced “brand deterioration” due to AI-related mishaps.
Mitigating the Menace: A Multi-Layered Defense
In order to safeguard against the threats posed by autonomous systems, a multi-layered defense strategy is required, several of which pertain to human risk management (HRM) principles.
-
Principle of least privilege: Provide agents with only the minimum set of privileges to carry out their functions, with both time-limited and automatically revocable access. This minimizes the opportunity for abuse.
-
Embedding ethical frameworks: Set clear value boundaries during the design process. Use automated checks and regular human reviews to support this.
-
HRM behavior-based training: Monitor user interactions to identify dangerous trends, such as overtrust of AI or evading protection features, and provide timely, targeted interventions that promote safe, well-informed choice-making and wise system use.
-
Adversarial testing: Proactively identify vulnerabilities by periodically exposing agents to simulated threats like prompt injection, data manipulation, and ransomware scenarios.
-
Personalized and adaptive learning: Provide role-specific guidance that fits each person's responsibilities, learning style, and risk exposure. Make sure it stays consistent with safety protocols and system complexity.
-
Maintain human-in-the-loop: Ensure that critical decisions are subject to human oversight, with a constant escalation path available for interventions and emergency shutdowns.
-
Red-teaming exercises: Conduct simulations of realistic attacks and abuse to reveal underlying weaknesses. Put the system's defenses and response plans to the test thoroughly. This makes agents robust and ready to go live.
-
Continuous monitoring: Use smart behavior tracking tools that can identify when things start to go off course, like a system drifting from its original goal, using resources in unexpected ways, or trying to access data it shouldn’t. Catching these signs in real time helps keep everything secure and on track.
Agentic-AI independence can drive productivity gains and lead to new discoveries, as long as we build the supervision and ethical guidelines that match its power. By recognizing the weaponization and potential for deep-seated biases in these agents and by using layered defense strategies, organizations can use powerful intelligence in a responsible and secure way.
The future of business depends not on embracing complete freedom but on guiding it with foresight, integrity, and human-centric values.
About the Author
Erich Kron
Security Awareness Advocate for KnowBe4
A 25-year veteran information security professional with experience in the medical, aerospace, manufacturing and defense fields, Erich Kron is Security Awareness Advocate for KnowBe4. Author, and regular contributor to cybersecurity industry publications, he was a security manager for the U.S. Army's 2nd Regional Cyber Center-Western Hemisphere and holds CISSP, CISSP-ISSAP, SACP and many other certifications. Erich has worked with information security professionals around the world to provide the tools, training and educational opportunities to succeed in information security.