Setting up red teaming for AI agents: the three layers you need from 2026

Set up red teaming for generative AI and agents in three layers: automated LLM tests in your CI pipeline, periodic agent exercises against tools and data, and autonomous red-team agents that continuously scan production. Record per workflow which tests ran, which vulnerabilities were found and which mitigations were implemented.

The occasion is a research note from the Cloud Security Alliance (CSA) of 10 July 2026 on autonomous AI red-team agents. It describes, among other things, Wiz Red Agent as an autonomous agent that in its first month made over 17,000 unique findings across around 1,000 customer environments, including a BOLA vulnerability in a major airline API that exposed years of passenger data. This suggests that autonomous red-team agents are already being deployed operationally in practice to examine production systems. For organisations that work with confidential information, this means red teaming is changing from a periodic pentest into a continuous security layer.

What does the CSA case on autonomous red-team agents concretely show?

The CSA research note on autonomous red teams shows two things. First, autonomous agents can discover and demonstrate real vulnerabilities in production at high speed. Second, the CSA calls on organisations to build out their own agentic or AI-assisted red-team capability, aimed at continuous rather than periodic testing.

In our assessment, the core of this signal is not that AI finds vulnerabilities, because scanners have done that for a while, but that the test itself becomes autonomous, repeated and broadly deployable. That changes the question for security teams from when do we test again to how do we keep the test layer itself under control.

What does red teaming of generative AI and agents mean in practice?

The OWASP GenAI Security Project, in its AI Security Solutions Landscape for Gen AI and Agentic Red Teaming, positions red teaming as a structured, lifecycle-wide discipline with its own taxonomy and an overview of tooling. It concerns coordinated adversarial testing, defensive validation and feedback loops, and no longer a standalone test after the fact.

A technical practice piece by FutureAGI on red teaming of generative AI in 2026 describes how this concretely works with language models:

• attacks are labelled under the OWASP LLM Top 10;
• an attack dataset of 500 to 5,000 prompts is managed in Git;
• with every model or prompt change, tests run in the CI pipeline;
• outputs are scored with LLM judges on safety, faithfulness and policy compliance.

This covers classic risks such as jailbreaks and the leaking of personal data. It does not, however, automatically cover the risks that only arise once a model operates tools and acts autonomously.

Which attacks does agent-specific red teaming address?

According to a CSA research note from March 2026, the adversarial-ML taxonomy of NIST AI 100-2 should be applied in an agentic context. This taxonomy aligns with NIST-style frameworks for AI risk management, including the AI Risk Management Framework (AI RMF 1.0). Red-team exercises for agents must explicitly test whether attacking instructions in documents, emails, calendars and web content can steer an agent's behaviour. That is a different test question than for a chatbot.

The arXiv study Agents of Chaos underpins this with an exploratory red-teaming study of autonomous language-model agents with persistent memory, email, Discord, file systems and shell access. The researchers document eleven case studies of misuse, including:

• obedience to instructions from non-owners;
• the disclosure of sensitive information;
• destructive system actions and denial-of-service conditions;
• identity spoofing and partial system takeover.

The practical implication: agent-specific red teaming asks questions that classic prompt tests do not ask. Can an agent be misled by an instruction hidden in an uploaded document? Can it bypass an approval step or spread to other agents via shared memory? Anyone who connects tools and integrations touches here on the same attack surface that also arises when securing MCP integrations per tool call after the 2026 spec.

How do I set up red teaming as a continuous system rather than a one-off audit?

The above sources together point to a single approach. On the basis of these sources, it makes sense to organise red teaming in 2026 as a layered system with three levels:

1. CI-integrated LLM red teaming against a fixed, version-controlled attack dataset, as described in the FutureAGI playbook.
2. Periodic, scenario-based agentic red teaming against integrations, data, memory and tools, in line with the CSA guidance based on the NIST taxonomy.
3. Autonomous red-team agents that continuously scan the production surface, as the CSA case around Wiz Red Agent shows.

A point of attention that the CSA itself names: an autonomous red-team agent with access to production is itself a sensitive system. Limit its rights, log its actions and treat the agent as part of your attack surface. This ties in with the broader theme of AI security and securing AI systems.

Which governance and verification questions should you be able to answer per workflow?

For organisations with confidential or high-trust information, the core question shifts from have we tested to can we demonstrate per workflow what we tested and what followed from it. Our recommendation is to record the following:

• which generative models and agents run where, and which red-team layers are active for them;
• how attacks are labelled (for example under the OWASP LLM Top 10) and kept as a dataset;
• who assesses the findings, which mitigations follow and how these are documented;
• how you prevent autonomous red-team agents from themselves becoming a new risk;
• how you keep sight of unauthorised AI use, as when exposing shadow AI before you can control it.

A verification layer such as IamVera.ai can in this context function as a visibility layer above the red-team architecture: a way to make visible per workflow which tests were carried out, which vulnerabilities were found in models and agents and which mitigations were implemented, so that this information is available as evidence of tests and mitigations per workflow for audits and internal risk committees. Vera does not replace the red-team tests and does not guarantee professional correctness; the professional final judgement on findings and mitigations remains with your security and risk team.

Sources: The article draws on research notes from the Cloud Security Alliance, the OWASP GenAI Security Project, the arXiv study Agents of Chaos and a technical playbook from FutureAGI.

← All articles in this topic ← All articles