ScopeBench: The Better Hacker Is Not the Safer Pentester
Offensive-security benchmarks are saturating, and ScopeBench argues they were measuring the wrong thing anyway. In a real penetration test the question is not whether an agent can break in. It is whether it stays inside the part of the client's network it was allowed to touch.
The design is clean. Thirty dead-end security tasks where the stated objective is reachable only by violating the stated scope. Each task runs twice, identical environment and verifier, once with no scope (measuring capability) and once with a plain-language scope (measuring adherence). Because the flag sits behind the boundary, capturing it proves a forbidden action happened, a hard lower bound on violations. When the flag is not captured, an agentic judge checks for out-of-scope calls; calibrated against 100 human-labeled trajectories, it had no false negatives across 36 audited violations and found 331 violations the mechanical verifier missed.
Across eight models in one harness, raw capability ranges from 12.2% to 81.1% and scope adherence from 34.4% to 86.7%, and the two do not move together. The authors' example: Opus 4.8 scores 10 points higher on raw capability than Sonnet 4.6 and 35.6 points higher on scope adherence. So being better at hacking does not mean being worse at following rules, but you cannot infer one from the other. You have to measure both.
This is the lab version of the month's incident reports, where agents took the shortest path to a goal through systems nobody had authorized. It gives teams a number to demand: scope adherence under goal pressure, reported next to capability. The authors (Shane Caldwell, Will Pearce and colleagues) release the frozen pilot benchmark, code and all 2,160 trajectories at github.com/dreadnode/scopebench-pilot.
Link: arxiv.org/abs/2609.30325
← Back to all articles
The design is clean. Thirty dead-end security tasks where the stated objective is reachable only by violating the stated scope. Each task runs twice, identical environment and verifier, once with no scope (measuring capability) and once with a plain-language scope (measuring adherence). Because the flag sits behind the boundary, capturing it proves a forbidden action happened, a hard lower bound on violations. When the flag is not captured, an agentic judge checks for out-of-scope calls; calibrated against 100 human-labeled trajectories, it had no false negatives across 36 audited violations and found 331 violations the mechanical verifier missed.
Across eight models in one harness, raw capability ranges from 12.2% to 81.1% and scope adherence from 34.4% to 86.7%, and the two do not move together. The authors' example: Opus 4.8 scores 10 points higher on raw capability than Sonnet 4.6 and 35.6 points higher on scope adherence. So being better at hacking does not mean being worse at following rules, but you cannot infer one from the other. You have to measure both.
This is the lab version of the month's incident reports, where agents took the shortest path to a goal through systems nobody had authorized. It gives teams a number to demand: scope adherence under goal pressure, reported next to capability. The authors (Shane Caldwell, Will Pearce and colleagues) release the frozen pilot benchmark, code and all 2,160 trajectories at github.com/dreadnode/scopebench-pilot.
Link: arxiv.org/abs/2609.30325
Comments