AI-Generated Code, Untrusted by Design: The New Rules of Software Security

As AI accelerates software development, insecure code, supply-chain vulnerabilities and unchecked automation threaten enterprise security. Human oversight, Zero Trust and immutable data protection are essential to preserving trust and recovery.

Key Highlights

  • Embedding AI into core application functionality is crucial for gaining a true competitive advantage, rather than treating it as an afterthought.
  • AI models often lack the security context needed for safe code generation, increasing the risk of vulnerabilities and malicious code introduction.
  • Human oversight remains essential; superficial review of AI-generated code can lead to security breaches and operational failures.
  • Applying Zero Trust principles helps safeguard AI workflows, data pipelines, and proprietary models from misuse and attack.
  • Building resilient data management strategies, including immutable backups, ensures organizations can recover quickly from AI-related failures or breaches.

AI Vibe Coding has recently become a shiny new object for consumers and enterprises alike, promising to save time, increase productivity and creativity, and make previously complex projects more accessible to all team members. However, the seductive nature of vibe coding comes at a price.

Are You Adopting AI or Just Bolting It on at the Cost of Security?

Today, vendors increasingly take the shortcut of bolting AI onto their platforms to look modern and gain a competitive advantage. AI that isn’t fully implemented into the foundation of applications is often attached just to stay relevant. Simply adding AI to legacy software can result in insufficient systems. True competitive advantage comes from embedding AI into core functionality, not treating it as an afterthought. But this goes deeper than the vendor; AI is fundamentally reshaping enterprise workflows.

CISOs must be vigilant about how much access AI models have, avoid chasing AI trends, and avoid relying on experimental AI. However, a few AI trends are shifting enterprise functions and workflows for the better. The shift to agentic AI means moving away from generative AI and chatbots to AI agents that can perform multi-step, goal-oriented, and task-driven projects. Successful companies are also using AI to enhance their data for better AI outputs, creating a self-reinforcing feedback loop that becomes a sustainable competitive advantage. However, these gains come with a tradeoff, especially when AI moves into code creation.

Giving AI too much autonomy to create and fix code can cause major vulnerabilities, blind spots, and challenges that may outweigh the positives. These AI models do not traditionally follow secure coding standards because they lack the contextual and security knowledge needed to build correct, foundational code. They often suggest open-source packages that introduce vulnerable or malicious code into the software supply chain. Developers can then become the weakest link in that supply chain if they copy code from forums such as Slack and GitHub. It also introduces prompt-injection threats by blurring the line between developers’ instructions and external data. Malicious prompts hidden in code can trick AI into generating insecure code and leaking credentials. This risk is not theoretical; attackers are already exploiting the same dynamics.

When AI Writes the Code, Who Owns the Risk?

Attackers can now use the same tools to build working malware. Bad actors with no real coding experience can create multiple ransomware variants. Anthropic recently reported that a cybercriminal sold malware packages built with Claude, each featuring advanced evasion capabilities, encryption, and anti-recovery mechanisms, for $400 to $1200. Similarly, the founder of a prominent SaaS community experimented with AI-generated code. At first, the code handled complex logic and built a sophisticated application. But then the AI agent decided the database needed “cleaning up” and deleted everything. If both defenders and attackers use AI-generated code, the real question becomes who is accountable.

As discussed in my previous SIW article, vibe coding can have detrimental effects if human oversight and regular testing aren't a core part of the coding process. Even when that oversight exists, it must be effective and cover every layer of the code. “Rubber-stamping” AI-generated code is equal to having no human in the loop at all. A human reviewer's critical evaluation is necessary to keep control and accountability intact. 

Attackers can now use the same tools to build working malware. Bad actors with no real coding experience can create multiple ransomware variants. Anthropic recently reported that a cybercriminal sold malware packages built with Claude, each featuring advanced evasion capabilities, encryption, and anti-recovery mechanisms, for $400 to $1200.

The illusion of control often stems from high-volume workflows and shortcuts. At the same time, these projects are typically undocumented, making it hard to track progress, maintain and update projects, and understand how the code is built, proving impossible for accurate testing and oversight. When oversight is weak, even foundational security models break down.

No security team should make critical decisions outside Zero Trust principles, and using AI to code actively goes against those standards. These principles include least-privilege access, explicit code verification, continuous validation, and processes and systems built to assume breaches. Applying these principles protects prompts, secures data pipelines, and safeguards proprietary AI models. The impact isn’t just technical; it’s reshaping the people behind the code. 

On the one hand, non-technical coders could introduce functional but insecure applications into an IT ecosystem that lacks rigorous IT governance. On the flip side, reliance on AI may erode the developer and coding skills programmers build throughout their careers. It may also break down accountability and ownership. Who is to blame when the code fails: the coder? The reviewer? The AI? How can organizations hold employees accountable and enforce security risk cultures and governance if AI models do the brunt of the work?

What Happens to Your Data When AI Gets It Wrong?

Not all hope is lost. Strong data foundations, along with data management, protection tools, and procedures, can offset the pitfalls of faulty AI code. Because detection is inherently reactive, organizations should adopt data protection strategies that assume breaches are inevitable. This means building recovery and resilience into the foundation of cybersecurity. By treating data security as a core pillar rather than an add-on, companies can better withstand and recover from AI manipulation. Resilience should be automatic, not aspirational. When your last line of defense is immutable, you can restore data with confidence.

Immutable backup storage appliances ensure every backup is permanently protected, preserving a secure, unchangeable copy you can always recover. So even if AI-driven code fails, your data remains isolated and recoverable. These applications are only truly immutable if they adhere to three core principles: S3 object storage, zero time to immutability, and target storage appliance. Understanding the core concept of Absolute Immutability helps separate secure backup systems from empty vendor claims.

Vibe coding isn’t just a faster way to build software; it’s a fundamental shift in how code is created, understood, and trusted. When speed and intuition replace rigor and verification, even well-established security models like Zero Trust begin to erode. Human oversight alone is not enough if it becomes superficial or overwhelmed by volume, making a strong data foundation essential to maintain integrity across systems.

Organizations must treat AI-generated code as inherently untrusted and invest in immutable backups and storage to keep data protected and recoverable when failures occur. Those that balance innovation with control and resilience at the data layer will be far better positioned than those chasing velocity without accountability.

About the Author

Geoff Burke

Geoff Burke

Community Manager at Object First

Geoff Burke is the Community Manager at Object First and has over 20 years of experience in IT, with 12 years of experience in Data Protection. He is VMCE and VMCA certified, as well as Kubernetes CKA and CKAD. Burke is active in IT Communities as a Veeam Vanguard, Veeam Legend, Tanzu Vanguard, and Calico Big Cat Ambassador. Geoff’s enthusiasm for technology is contagious, and his energy and passion for the subject inspire others to expand their knowledge and interests.

Sign up for our eNewsletters
Get the latest news and updates