Frontier Agents Are Optimizers, Not Researchers
On Hugging Face's daily papers board at 40 upvotes: "Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI R&D" (arXiv 2608.13417, submitted August 13, thirteen authors including Fei Sun, Xunliang Cai and Jingang Wang). The setup: seven frontier models on 36 long-horizon AI research and development tasks, evaluated not on the final leaderboard number but with rule-based metrics that characterize what happens inside a run, across three dimensions, Solution Framing, Execution, and Feedback Control.
The verdict is the title of this article: current frontier agents operate like engineering optimizers, not autonomous researchers. They can frame practical solutions, but with high variance across runs. Their solutions adapt existing techniques rather than introduce anything novel. And outcomes hinge on process bottlenecks, the quality of experience reuse, and harness design.
Two things make this more than another benchmark paper. First, it is direct evidence for what final-score evaluation hides: a good process and a lucky one produce the same number, and the difference is exactly what you need to know before trusting an agent with a week-long task. The within-run metrics are the contribution, whatever you think of the specific tasks. Second, harness design showing up as a determinant of research outcomes is now a finding arriving from every direction, the same conclusion DarwinX and AutoDesign reached by optimizing the harness, reached here by measuring it as a confound. If you evaluate agents on outcomes only, you are measuring your harness as much as your model, and you cannot tell which one is failing.
arxiv.org/abs/2608.13417
← Back to all articles
The verdict is the title of this article: current frontier agents operate like engineering optimizers, not autonomous researchers. They can frame practical solutions, but with high variance across runs. Their solutions adapt existing techniques rather than introduce anything novel. And outcomes hinge on process bottlenecks, the quality of experience reuse, and harness design.
Two things make this more than another benchmark paper. First, it is direct evidence for what final-score evaluation hides: a good process and a lucky one produce the same number, and the difference is exactly what you need to know before trusting an agent with a week-long task. The within-run metrics are the contribution, whatever you think of the specific tasks. Second, harness design showing up as a determinant of research outcomes is now a finding arriving from every direction, the same conclusion DarwinX and AutoDesign reached by optimizing the harness, reached here by measuring it as a confound. If you evaluate agents on outcomes only, you are measuring your harness as much as your model, and you cannot tell which one is failing.
arxiv.org/abs/2608.13417
Comments