We're Still Testing the Wrong Thing: AI Red Teaming in 2026

Nine months ago I argued that we were red teaming the wrong layer of the AI stack. I did not expect the argument to age this quickly.

Last July I wrote about a paper out of Stanford and Georgetown that made a claim most of the industry was not ready to hear: that AI red teaming, as practiced, was pointed at the wrong target. The field had organized itself around finding flaws in individual models, when the failures that actually hurt organizations tend to emerge somewhere else. The authors drew a line between two kinds of testing. Micro-level red teaming probes the model itself, the prompts, the guardrails, the outputs. Macro-level red teaming examines the whole system as it lives in the world, including the people and institutions that build on top of it. At the time, that framing felt a step ahead of the conversation. Most teams were still learning to handle prompt injection, and "sociotechnical risk" sounded like something you discussed at a conference rather than something that showed up in an incident report.

Nine months is not a long time. It has been long enough. The paper no longer reads like a forward-looking proposal. It reads like a description of problems we are actively failing to solve.

The ground shifted under the argument

The clearest change is that agentic AI stopped being a demo and became infrastructure. Gartner projects that forty percent of enterprise applications will embed task-specific AI agents by 2026, up from under five percent a year earlier. That is not a gentle curve. And these are not chatbots that answer a question and forget it. They call APIs, reach into data stores, execute workflows, and make decisions across long chains of steps that no human reviews in the moment. Once a system can act, the interesting attack surface is no longer the model that writes a response. It is the entire environment that model is allowed to touch.

The security community noticed. Late in 2025, OWASP published a Top 10 for Agentic Applications, assembled with input from more than a hundred practitioners, and it reads like a catalog of exactly the risks the original paper was gesturing at. Agent goal hijacking and tool misuse sit at the top of the list. Neither is a prompt-level bug. They describe what happens when an agent's mission gets quietly redirected across a sequence of actions, or when the access you granted for a legitimate purpose is turned against you. You do not find these by throwing adversarial strings at a model in isolation. You find them by testing a system that does things.

Meanwhile, the one force that should have pushed organizations toward that kind of testing moved the other way. Executive Order 14110 had required red teaming for high-risk models and put NIST in charge of the guidelines. In January 2025, Executive Order 14148 rescinded it, trading a safety-first posture for one built around deregulation and speed. NIST has kept working, continuing its Risk Management Framework and standing up an AI Agent Standards Initiative earlier this year, but the federal mandate that gave many red teaming programs their budget line simply evaporated. Teams that justified the work by pointing at a regulation now have to justify it on the merits.

What the first piece got right, and what it was too polite to say

The core claim has only hardened. Most red teaming still hunts for model-level defects while the consequential failures are systemic, and agentic AI has made the point faster than the authors likely expected. Their recommendation to build multidisciplinary teams, mixing ML engineers with social scientists, domain experts, and security practitioners, no longer looks like good practice. It looks like a prerequisite. You cannot assess a system by asking whether its model can be tricked. You have to understand the business process it sits inside, the data it can reach, the authority it has been handed, and the failure modes that appear when it chains a series of individually reasonable steps into an outcome no one intended.

Reading it back, I was too conservative in a few places.

I framed macro-level red teaming as something organizations should prepare for. Many of them shipped agents into production with no red teaming program at all, macro or otherwise. For a large part of the market, the gap is not that we test models but not systems. The gap is that we do not test.

I also gave the supply chain a paragraph when it deserved a chapter. Every tool, plugin, and external data source an agent consumes is a trust boundary, and attackers have already found them. Indirect prompt injection, where the malicious instruction arrives inside untrusted content rather than from the user, appeared in seventy-three percent of production AI deployments in 2025. The same adaptive, machine-generated phishing I described in AI-Driven Social Engineering is one of the delivery vehicles. In one ACL study, multi-agent denial-of-service attacks succeeded more than eighty percent of the time. These are not exotic edge cases. They are the default condition of any system that lets a model read from the open world.

And I underplayed the most uncomfortable part, which is organizational rather than technical. Internal red teams operate inside the same incentives that produced the system they are testing. They are not always free to probe the scenarios that would embarrass leadership or call a shipped product into question. That is not a criticism of the people doing the work. It is a structural reason why external testing and independent disclosure are not niceties. They are how you surface the risks your own team is quietly paid not to look for.

What to do about it

The through line is easy to state and hard to live by: test the system, not the model. If your agent can read email, query a database, and post to Slack, your red team's job is to find out what happens when a poisoned email convinces it to move the contents of that database into a Slack channel. A jailbreak test will never surface that, because the vulnerability does not live in the model. It lives in the wiring between the model and everything you let it reach.

From there the rest follows. Adopt the OWASP agentic framework, because it did not exist when I wrote the first piece and it now gives you a shared vocabulary for goal hijacking, tool misuse, delegated-trust failures, and memory poisoning. Stop waiting for regulation to tell you this matters, because the federal signal is weaker than it was a year ago and it is not returning soon. Wire adversarial testing into your release process, so that a changed prompt or a newly granted permission triggers a fresh round of attacks, because the systems that break in production are the ones that changed since anyone last looked. And bring in outsiders, because your internal team shares the blind spots of the people who built the thing.

The part that keeps me up

The Stanford and Georgetown authors were right about the direction of travel. The field needed to move from testing models to evaluating whole systems for resilience, and the need is more urgent now than when they wrote it. What has not happened is the closing of the distance between where most organizations are and where they need to be. If anything, agents arrived faster than the discipline to secure them, and the gap widened.

Red teaming in 2026 is not a hunt for clever jailbreaks. It is the practice of understanding how autonomous systems fail when they meet a messy, adversarial, real-world environment, and then building the organizational muscle to test for that failure continuously rather than once, at launch, as a box to check. The paper handed us the map. Whether we use it is still, frustratingly, an open question.

---

Reference: Sharkey, L., Pasquinelli, M., Cheng, B., Dobbe, R., et al. "Operationalizing Red Teaming for AI Systems." arXiv:2507.05538, July 2025.