ROGUE-AI: Most Agent Disasters Had No Attacker
The industry spent two years building defenses against prompt injection. A new paper says that is mostly the wrong fight. ROGUE-AI, submitted to arXiv on November 19 and listed in Friday's batch, rebuilt 108 publicly reported agent incidents offline, the kind of story where a coding agent wipes a production database, and replayed them on five current GPT, Claude, Gemini and DeepSeek models. Of the 72 incidents that caused real-world harm, 62 had no attacker at all. Paper: https://arxiv.org/abs/2611.26553
The method is what makes the number worth trusting. Each incident gets a twin: an identical environment that lacks only the cause the original report blamed. Then agents run on both. The framework reproduced the reported harm in 89% of its replications. With the blamed cause present, agents made the same mistake in 18% to 33% of runs. Without it, at most 1%. So the stories in the news were not flukes, and the causes people named were mostly the real ones.
The uncomfortable part is what those causes look like. In the attacker-free cases the agent did exactly what its task asked and walked past an ordinary hazard, for example a database that turned out to be shared with the live site. These replications caused more harm than the prompt-injection replications for all five models. Injection filters look for planted instructions, and these cases contain none. There is nothing to catch. The agent is not tricked. It is just careless in a way that looks like diligence.
The fix the authors tested is almost embarrassingly simple. A warning after each tool output that names the hazard and its consequence lowered harm for all five models. A note of the same length that did not mention the hazard left most of the harm in place. In other words, generic safety boilerplate does nothing, while specific context about what can break does a lot. That puts the burden on whoever builds the environment, not on the model vendor.
My read: this is the strongest evidence yet that agent safety is an environment problem first and an adversary problem second. The replications are released as a reusable audit set vendors can rerun before each release, which is the right shape for it. If your agent touches anything shared with production, the question is not whether someone will attack it. It is whether anyone told it what that thing is.
← Back to all articles
The method is what makes the number worth trusting. Each incident gets a twin: an identical environment that lacks only the cause the original report blamed. Then agents run on both. The framework reproduced the reported harm in 89% of its replications. With the blamed cause present, agents made the same mistake in 18% to 33% of runs. Without it, at most 1%. So the stories in the news were not flukes, and the causes people named were mostly the real ones.
The uncomfortable part is what those causes look like. In the attacker-free cases the agent did exactly what its task asked and walked past an ordinary hazard, for example a database that turned out to be shared with the live site. These replications caused more harm than the prompt-injection replications for all five models. Injection filters look for planted instructions, and these cases contain none. There is nothing to catch. The agent is not tricked. It is just careless in a way that looks like diligence.
The fix the authors tested is almost embarrassingly simple. A warning after each tool output that names the hazard and its consequence lowered harm for all five models. A note of the same length that did not mention the hazard left most of the harm in place. In other words, generic safety boilerplate does nothing, while specific context about what can break does a lot. That puts the burden on whoever builds the environment, not on the model vendor.
My read: this is the strongest evidence yet that agent safety is an environment problem first and an adversary problem second. The replications are released as a reusable audit set vendors can rerun before each release, which is the right shape for it. If your agent touches anything shared with production, the question is not whether someone will attack it. It is whether anyone told it what that thing is.
Comments