Humans Approved 1 in 3 Malicious Agent Commands. So Much for the Loop.
Human-in-the-loop is the safety story every agent product tells: the AI proposes, you approve, nothing bad happens. Scale X built a browser game to test whether that story survives contact with actual humans, and published the data on August 6 (scalex.dev/blog/ai-agent-permissions-stats). Across 40,000 runs and 409,000 individual approve/deny decisions, players approved roughly one in three malicious requests. Mean accuracy: 66.3 percent. A third of all sessions ended with a negative score. The post took 224 points on Hacker News, and The Register picked it up the same day.
The game mimics the default coding-agent flow — a stream of commands, each waiting for your click — and salts it with dangerous ones. The single most-missed threat is telling: npm run analyze got approved 64.7 percent of the time. It looks utterly harmless, and that is the point — npm run executes whatever the package.json says, which is exactly where injected malicious code lives. The commands people are worst at judging are the ones that require context they do not have.
Yes, it is a game, and game players click faster than a developer reviewing a real production command. But the mechanism it isolates — approval fatigue plus missing context — is precisely what the default flow of every coding agent produces today. The author built it after realizing there was no good answer to that problem, just an endless stream of permission dialogs.
The uncomfortable conclusion: the human in the loop is the least reliable component in the system, and vendors keep shipping it as the safety layer because it moves liability, not because it works. This lands the same week Cloudflare proposed its Agent Access Model for task-scoped agent identity, and it makes the case better than any whitepaper: oversight has to move into structure — scoped credentials, sandboxes, automated policy — because attention is not a security boundary.
← Back to all articles
The game mimics the default coding-agent flow — a stream of commands, each waiting for your click — and salts it with dangerous ones. The single most-missed threat is telling: npm run analyze got approved 64.7 percent of the time. It looks utterly harmless, and that is the point — npm run executes whatever the package.json says, which is exactly where injected malicious code lives. The commands people are worst at judging are the ones that require context they do not have.
Yes, it is a game, and game players click faster than a developer reviewing a real production command. But the mechanism it isolates — approval fatigue plus missing context — is precisely what the default flow of every coding agent produces today. The author built it after realizing there was no good answer to that problem, just an endless stream of permission dialogs.
The uncomfortable conclusion: the human in the loop is the least reliable component in the system, and vendors keep shipping it as the safety layer because it moves liability, not because it works. This lands the same week Cloudflare proposed its Agent Access Model for task-scoped agent identity, and it makes the case better than any whitepaper: oversight has to move into structure — scoped credentials, sandboxes, automated policy — because attention is not a security boundary.
Comments