ExplorationBench Builds Alien Worlds So Agents Can't Cheat With Memory
Every science-agent benchmark has the same hole: when the agent gets the right answer, did it discover it or remember it? ExplorationBench (arXiv 2609.30199) closes the hole by making the knowledge impossible to remember. It builds alien worlds whose rules are executable, so every answer checks exactly, and deliberately contradict familiar knowledge, so recall actively hurts.
Two sandboxes. AlienCode has 31 discovery targets and 70 tasks, AlienLogic has 24 targets and 70 tasks. Each gives the agent a flawed manual, task-specific environment feedback and a dedicated tool-call schema. The agent explores, frames hypotheses, runs experiments, then has to solve held-out tasks with whatever it figured out. Ten AI systems were evaluated.
The strongest systems can learn unfamiliar rules and apply them, which is the encouraging part. The warning is in the second half of the result: performance swings a lot across trajectories, and continued exploration can stall or even reverse gains the agent already made. More steps did not reliably mean more knowledge. Sometimes the agent explored its way out of a correct understanding.
That second finding is the one worth carrying around. Most autonomous research loops assume that running longer is at worst wasted compute. This says running longer can actively corrupt what was learned, which pairs uncomfortably with the self-correction and memory papers of the week. The fix people will reach for is checkpointing beliefs and testing against them before overwriting. The flawed-manual design also deserves copying: real documentation is wrong, and an agent that trusts it blindly should score badly.
← Back to all articles
Two sandboxes. AlienCode has 31 discovery targets and 70 tasks, AlienLogic has 24 targets and 70 tasks. Each gives the agent a flawed manual, task-specific environment feedback and a dedicated tool-call schema. The agent explores, frames hypotheses, runs experiments, then has to solve held-out tasks with whatever it figured out. Ten AI systems were evaluated.
The strongest systems can learn unfamiliar rules and apply them, which is the encouraging part. The warning is in the second half of the result: performance swings a lot across trajectories, and continued exploration can stall or even reverse gains the agent already made. More steps did not reliably mean more knowledge. Sometimes the agent explored its way out of a correct understanding.
That second finding is the one worth carrying around. Most autonomous research loops assume that running longer is at worst wasted compute. This says running longer can actively corrupt what was learned, which pairs uncomfortably with the self-correction and memory papers of the week. The fix people will reach for is checkpointing beliefs and testing against them before overwriting. The flawed-manual design also deserves copying: real documentation is wrong, and an agent that trusts it blindly should score badly.
Comments