SWE-bench Science: Coding Agents Meet Real Research Code
The Fudan OpenMOSS group (Xipeng Qiu's lab) just built the benchmark that punctures the "coding is solved" mood. SWE-bench Science is 119 repository-level tasks pulled from 98 real GitHub repos across 20 scientific domains — the messy computational code that actual research runs on. The headline: the best agent tested, Claude Code running Opus 5 on max settings, lands under 50% pass@1. The same generation of agents casually clears 80% on regular SWE-bench Verified.
The task design is what makes it bite. Beyond standard issue-driven fixes, there are expert-exploratory tasks and engineering-integration tasks — the kinds of problems where you need to understand the science to know what the code should do. The failure analysis found four recurring patterns: scientific knowledge deficits, shallow exploration, incomplete repairs, and poor generalization. In other words, the agents fail like a competent programmer who skipped the domain class.
The most interesting finding is about guidance. Feeding the agent scientific context helps when it's well-aligned with the task — better performance, fewer steps. But misaligned guidance produces anchoring: the agent latches onto the hint, stops exploring, and doesn't even gain success rate for it. Anyone building retrieval into a coding agent should sit with that — bad context isn't neutral, it's actively worse than nothing.
There's a 30-point gap between engineering code and science code, and that gap is a map of what's left. Yesterday we covered agents that can't quit; today it's agents that can't read a physics repo. The benchmark era is getting specific about where the frontier actually is.
Paper: https://arxiv.org/abs/2608.19799
← Back to all articles
The task design is what makes it bite. Beyond standard issue-driven fixes, there are expert-exploratory tasks and engineering-integration tasks — the kinds of problems where you need to understand the science to know what the code should do. The failure analysis found four recurring patterns: scientific knowledge deficits, shallow exploration, incomplete repairs, and poor generalization. In other words, the agents fail like a competent programmer who skipped the domain class.
The most interesting finding is about guidance. Feeding the agent scientific context helps when it's well-aligned with the task — better performance, fewer steps. But misaligned guidance produces anchoring: the agent latches onto the hint, stops exploring, and doesn't even gain success rate for it. Anyone building retrieval into a coding agent should sit with that — bad context isn't neutral, it's actively worse than nothing.
There's a 30-point gap between engineering code and science code, and that gap is a map of what's left. Yesterday we covered agents that can't quit; today it's agents that can't read a physics repo. The benchmark era is getting specific about where the frontier actually is.
Paper: https://arxiv.org/abs/2608.19799
Comments