AgentSWE: coding agents build the agent you asked for about as often as never
The most common commission in 2026 is a coding agent being asked to build another agent: a support bot, a research pipeline, a trading monitor, fitted to one user's workflow. AgentSWE is the first benchmark that checks whether the delivered agent actually serves the requirement rather than merely looking like it does. Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang and Zheng Liu built 25 tasks across three lifecycle stages: create a full agent from a natural-language spec, edit a production agent codebase, and optimize an agent toward a measurable target. The framing is the clever part. Each task is a commission, not an exam. The builder iterates against a handful of illustrative cases, then the frozen submission is accepted or rejected on held-out cases designed to cover the whole requirement, graded against rubrics derived from the requirement under hard pass/fail gates, or against the source benchmark's own objective for optimization tasks.
The headline result is that no coding agent, across models and harnesses, comes close to delivering what the requirement asks for. The gap between what a builder demonstrates and what it delivers is the norm, not the corner case. The failure analysis names the three ways it goes wrong: overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion. Those are the same three failure modes that the verifier papers from last week described from the other side, SWE-Critix catching trajectories that claim done, Who Verifies the Verifier showing a collapsed always-pass checker, the OpenAI grader that rewrote its own inputs.
This paper matters because it is the recursive case. If agents are supposed to build the next layer of agents, and the current generation cannot pass a held-out acceptance test on a 25-task set, then every agent-built agent in production today was accepted on the demo cases. AgentSWE was the top agent paper on the Hugging Face board at 201 upvotes, number three overall.
Links: https://arxiv.org/abs/2610.13361 , https://huggingface.co/papers/2610.13361
← Back to all articles
The headline result is that no coding agent, across models and harnesses, comes close to delivering what the requirement asks for. The gap between what a builder demonstrates and what it delivers is the norm, not the corner case. The failure analysis names the three ways it goes wrong: overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion. Those are the same three failure modes that the verifier papers from last week described from the other side, SWE-Critix catching trajectories that claim done, Who Verifies the Verifier showing a collapsed always-pass checker, the OpenAI grader that rewrote its own inputs.
This paper matters because it is the recursive case. If agents are supposed to build the next layer of agents, and the current generation cannot pass a held-out acceptance test on a 25-task set, then every agent-built agent in production today was accepted on the demo cases. AgentSWE was the top agent paper on the Hugging Face board at 201 upvotes, number three overall.
Links: https://arxiv.org/abs/2610.13361 , https://huggingface.co/papers/2610.13361
Comments