RECLAIM: The Best Agent Reproduces 15% of ML Papers When It Has to Write the Code
41 percent when the authors released code, data and weights. 27 percent when the agent has to retrain the model. 15 percent when it has to write the code itself. Those are the best-agent reproduction rates in RECLAIM, a benchmark of 100 NeurIPS 2025 papers from Mithil Salunkhe, Volodymyr Kindratenko and colleagues, posted September 23.
The design is careful in a way most agent benchmarks are not. For every paper the target result, the definition of success and a GPU-hour budget are fixed in advance. Difficulty is set by what the original authors actually released, not by a human rating. And the grading is done by a separate language model reading logs and outputs, not by trusting the agent's own report of what it achieved. The benchmark can also be rebuilt every year from a new conference, which is the only real defense against contamination in this category.
The most useful number is about how agents fail. Failed attempts used only 29 percent of their budget on average, so they did not run out of compute. They gave up, or declared victory, with time left. And the single most common error, in 63 of 400 runs, was implementing the method without checking any part of it against the paper's reported numbers. That is the research-agent version of shipping without running the tests.
Put that next to the autonomous-research claims going around and you get the honest state of play. Agents can execute a recipe when every ingredient is on the table. When they have to reconstruct something from prose, they mostly do not verify, and they stop early. The fix is not more GPU hours. It is a harness that forces intermediate checks against known numbers. Paper: arxiv.org/abs/2609.28850.
← Back to all articles
The design is careful in a way most agent benchmarks are not. For every paper the target result, the definition of success and a GPU-hour budget are fixed in advance. Difficulty is set by what the original authors actually released, not by a human rating. And the grading is done by a separate language model reading logs and outputs, not by trusting the agent's own report of what it achieved. The benchmark can also be rebuilt every year from a new conference, which is the only real defense against contamination in this category.
The most useful number is about how agents fail. Failed attempts used only 29 percent of their budget on average, so they did not run out of compute. They gave up, or declared victory, with time left. And the single most common error, in 63 of 400 runs, was implementing the method without checking any part of it against the paper's reported numbers. That is the research-agent version of shipping without running the tests.
Put that next to the autonomous-research claims going around and you get the honest state of play. Agents can execute a recipe when every ingredient is on the table. When they have to reconstruct something from prose, they mostly do not verify, and they stop early. The fix is not more GPU hours. It is a harness that forces intermediate checks against known numbers. Paper: arxiv.org/abs/2609.28850.
Comments