Red Team AI now to build safer, smarter models tomorrow

Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more

Editor’s note: Louis will lead an editorial roundtable on this topic at VB Transform this month. Register today.

AI models are under siege. With 77% of enterprises already hit by adversarial model attacks and 41% of those attacks exploiting prompt injections and data poisoning, attackers’ tradecraft is outpacing existing cyber defenses.

To reverse this trend, it’s critical to rethink how security is integrated into the models being built today. DevOps teams need to shift from taking a reactive defense to continuous adversarial testing at every step.

Red Teaming needs to be the core

Protecting large language models (LLMs) across DevOps cycles requires red teaming as a core component of the model-creation process. Rather than treating security as a final hurdle, which is typical in web app pipelines, continuous adversarial testing needs to be integrated into every phase of the Software Development Life Cycle (SDLC).

Adopting a more integrative approach to DevSecOps fundamentals is becoming necessary to mitigate the growing risks of prompt injections, data poisoning and the exposure of sensitive data. Severe attacks like these are becoming more prevalent, occurring from model design through deployment, making ongoing monitoring essential.

Microsoft’s recent guidance on planning red teaming for large language models (LLMs) and their applications provides a valuable methodology for starting. NIST’s AI Risk Management Framework reinforces this, emphasizing the need for a more proactive, lifecycle-long approach to adversarial testing and risk mitigation. Microsoft’s recent red teaming of over 100 generative AI products underscores the need to integrate automated threat detection with expert oversight throughout model development.

As regulatory frameworks, such as the EU’s AI Act, mandate rigorous adversarial testing, integrating continuous red teaming ensures compliance and enhanced security.

OpenAI’s approach to red teaming integrates external red teaming from early design through deployment, confirming that consistent, preemptive security testing is crucial to the success of LLM development.

Why traditional cyber defenses fail against AI

Traditional, longstanding cybersecurity approaches fall short against AI-driven threats because they are fundamentally different from conventional attacks. As adversaries’ tradecraft surpasses traditional approaches, new techniques for red teaming are necessary. Here’s a sample of the many types of tradecraft specifically built to attack AI models throughout the DevOps cycles and once in the wild:

• Data Poisoning: Adversaries inject corrupted data into training sets, causing models to learn incorrectly and creating persistent inaccuracies and operational errors until they are discovered. This often undermines trust in AI-driven decisions.
• Model Evasion: Adversaries introduce carefully crafted, subtle input changes, enabling malicious data to slip past detection systems by exploiting the inherent limitations of static rules and pattern-based security controls.
• Model Inversion: Systematic queries against AI models enable adversaries to extract confidential information, potentially exposing sensitive or proprietary training data and creating ongoing privacy risks.
• Prompt Injection: Adversaries craft inputs specifically designed to trick generative AI into bypassing safeguards, producing harmful or unauthorized results.
• Dual-Use Frontier Risks: In the recent paper, Benchmark Early and Red Team Often: A Framework for Assessing and Managing Dual-Use Hazards of AI Foundation Models, researchers from The Center for Long-Term Cybersecurity at the University of California, Berkeley emphasize that advanced AI models significantly lower barriers, enabling non-experts to carry out sophisticated cyberattacks, chemical threats, or other complex exploits, fundamentally reshaping the global threat landscape and intensifying risk exposure.

Integrated Machine Learning Operations (MLOps) further compound these risks, threats, and vulnerabilities. The interconnected nature of LLM and broader AI development pipelines magnifies these attack surfaces, requiring improvements in red teaming.

Cybersecurity leaders are increasingly adopting continuous adversarial testing to counter these emerging AI threats. Structured red-team exercises are now essential, realistically simulating AI-focused attacks to uncover hidden vulnerabilities and close security gaps before attackers can exploit them.

How AI leaders stay ahead of attackers with red teaming

Adversaries continue to accelerate their use of AI to create entirely new forms of tradecraft that defy existing, traditional cyber defenses. Their goal is to exploit as many emerging vulnerabilities as possible.

Industry leaders, including the major AI companies, have responded by embedding systematic and sophisticated red-teaming strategies at the core of their AI security. Rather than treating red teaming as an occasional check, they deploy continuous adversarial testing by combining expert human insights, disciplined automation, and iterative human-in-the-middle evaluations to uncover and reduce threats before attackers can exploit them proactively.

Their rigorous methodologies allow them to identify weaknesses and systematically harden their models against evolving real-world adversarial scenarios.

• Anthropic relies on rigorous human insight as part of its ongoing red-teaming methodology. By tightly integrating human-in-the-loop evaluations with automated adversarial attacks, the company proactively identifies vulnerabilities and continually refines the reliability, accuracy and interpretability of its models.

• Meta scales AI model security through automation-first adversarial testing. Its Multi-round Automatic Red-Teaming (MART) systematically generates iterative adversarial prompts, rapidly uncovering hidden vulnerabilities and efficiently narrowing attack vectors across expansive AI deployments.

• Microsoft harnesses interdisciplinary collaboration as the core of its red-teaming strength. Using its Python Risk Identification Toolkit (PyRIT), Microsoft bridges cybersecurity expertise and advanced analytics with disciplined human-in-the-middle validation, accelerating vulnerability detection and providing detailed, actionable intelligence to fortify model resilience.

• OpenAI taps global security expertise to fortify AI defenses at scale. Combining external security specialists’ insights with automated adversarial evaluations and rigorous human validation cycles, OpenAI proactively addresses sophisticated threats, specifically targeting misinformation and prompt-injection vulnerabilities to maintain robust model performance.

In short, AI leaders know that staying ahead of attackers demands continuous and proactive vigilance. By embedding structured human oversight, disciplined automation, and iterative refinement into their red teaming strategies, these industry leaders set the standard and define the playbook for resilient and trustworthy AI at scale.

As attacks on LLMs and AI models continue to evolve rapidly, DevOps and DevSecOps teams must coordinate their efforts to address the challenge of enhancing AI security. VentureBeat is finding the following five high-impact strategies security leaders can implement right away:

• Integrate security early (Anthropic, OpenAI)
  Build adversarial testing directly into the initial model design and throughout the entire lifecycle. Catching vulnerabilities early reduces risks, disruptions and future costs.

• Deploy adaptive, real-time monitoring (Microsoft)
  Static defenses can’t protect AI systems from advanced threats. Leverage continuous AI-driven tools like CyberAlly to detect and respond to subtle anomalies quickly, minimizing the exploitation window.

• Balance automation with human judgment (Meta, Microsoft)
  Pure automation misses nuance; manual testing alone won’t scale. Combine automated adversarial testing and vulnerability scans with expert human analysis to ensure precise, actionable insights.

• Regularly engage external red teams (OpenAI)
  Internal teams develop blind spots. Periodic external evaluations reveal hidden vulnerabilities, independently validate your defenses and drive continuous improvement.

• Maintain dynamic threat intelligence (Meta, Microsoft, OpenAI)
  Attackers constantly evolve tactics. Continuously integrate real-time threat intelligence, automated analysis and expert insights to update and strengthen your defensive posture proactively.

Taken together, these strategies ensure DevOps workflows remain resilient and secure while staying ahead of evolving adversarial threats.

Red teaming is no longer optional; it’s essential

AI threats have grown too sophisticated and frequent to rely solely on traditional, reactive cybersecurity approaches. To stay ahead, organizations must continuously and proactively embed adversarial testing into every stage of model development. By balancing automation with human expertise and dynamically adapting their defenses, leading AI providers prove that robust security and innovation can coexist.

Ultimately, red teaming isn’t just about defending AI models. It’s about ensuring trust, resilience, and confidence in a future increasingly shaped by AI.

Join me at Transform 2025

I’ll be hosting two cybersecurity-focused roundtables at VentureBeat’s Transform 2025, which will be held June 24–25 at Fort Mason in San Francisco. Register to join the conversation.

My session will include one on red teaming, AI Red Teaming and Adversarial Testing, diving into strategies for testing and strengthening AI-driven cybersecurity solutions against sophisticated adversarial threats.