RecreationWorld: GPT-6 Astra scores 58 percent, and passes everything 2.8 percent of the time
RecreationWorld, arXiv 2609.22000, submitted September 18 and on HuggingFace's board today, sets a task that is brutally simple to state. Here is a running application. Figure out what it does and build a faithful implementation. No prescribed workflow, no spec handed to you, no hints about where to start.
That framing does something clever. The running reference is the oracle, so hidden behavioral tests can be generated from it, which gives you execution-grounded rewards on a task that would otherwise be impossible to grade. It spans five platforms, Ubuntu, macOS, Windows, Android and Web, and the benchmark, RecreationBench, is 250 tasks across them. It is also explicitly hybrid: the agent has to use the GUI to discover behavior and then write software to reproduce it, which is most of an actual engineering job and almost none of an actual benchmark.
Now the numbers. GPT-6 Astra scores 58.1 percent overall. Only 2.8 percent of tasks pass all programmatic tests. Sit with that gap for a second. Partial credit says the frontier is doing well. Full credit says that on 97 out of 100 tasks, the result is something that looks right and is wrong somewhere you would have to find yourself.
The breakdown says exactly where it goes wrong: agents replicate static interface structure far more reliably than interactions and computed outputs. They get the boxes in the right places. They do not get what happens when you click, and they do not get the number that comes out. Which is another way of saying they recreate what a screenshot shows and miss what only shows up if you actually use the thing.
The authors release the benchmark, environments and test suites, and that is the right call because the 2.8 percent is the useful number and it only stays useful if other people can reproduce it. The broader point is one this field keeps having to relearn: aggregate scores on agent benchmarks flatter the models, and all-or-nothing scoring is closer to what shipping feels like. https://arxiv.org/abs/2609.22000
← Back to all articles
That framing does something clever. The running reference is the oracle, so hidden behavioral tests can be generated from it, which gives you execution-grounded rewards on a task that would otherwise be impossible to grade. It spans five platforms, Ubuntu, macOS, Windows, Android and Web, and the benchmark, RecreationBench, is 250 tasks across them. It is also explicitly hybrid: the agent has to use the GUI to discover behavior and then write software to reproduce it, which is most of an actual engineering job and almost none of an actual benchmark.
Now the numbers. GPT-6 Astra scores 58.1 percent overall. Only 2.8 percent of tasks pass all programmatic tests. Sit with that gap for a second. Partial credit says the frontier is doing well. Full credit says that on 97 out of 100 tasks, the result is something that looks right and is wrong somewhere you would have to find yourself.
The breakdown says exactly where it goes wrong: agents replicate static interface structure far more reliably than interactions and computed outputs. They get the boxes in the right places. They do not get what happens when you click, and they do not get the number that comes out. Which is another way of saying they recreate what a screenshot shows and miss what only shows up if you actually use the thing.
The authors release the benchmark, environments and test suites, and that is the right call because the 2.8 percent is the useful number and it only stays useful if other people can reproduce it. The broader point is one this field keeps having to relearn: aggregate scores on agent benchmarks flatter the models, and all-or-nothing scoring is closer to what shipping feels like. https://arxiv.org/abs/2609.22000
Comments