Singularity-Bench: can agents build the next agent? One already beats humans
The name is loud, the design is careful. Singularity-Bench (https://arxiv.org/abs/2611.16591), a 19-author paper in Friday's arXiv batch, asks whether coding agents can rebuild the AI stack from a minimal starting point. Six end-to-end tasks: pre-training, mid-training, post-training, inference, an agent harness and cluster orchestration. Each result is compared against a mature human-engineered baseline on the same hardware.
The headline number: Opus 5.5 running in Claude Code matches or beats the human baseline on all six tasks, for a Singularity-Index of 1.25. The other ten agents, built from seven frontier models across two harnesses, average 0.58. The authors say this suggests the model may have the skills needed to reach the singularity. That is their framing, and it is a stretch: beating a baseline on six well-defined engineering tasks is not open-ended research.
The more useful part came before any scores. The team red-teamed its own grading setup and found holes in 9 of the 10 OWASP Top 10 for Agentic Applications categories, including reward-hacking paths in the Harbor evaluation framework. So they built Breakwater, a Harbor extension that blocks every attack they tested, and only then ran the agents. Any benchmark where agents train models and report their own results is an invitation to game it. This one treated the grader as the hard part.
The gap is the other signal. 1.25 for one model in one harness against 0.58 for the rest means the combination matters as much as the model. It also lines up with this week's other evidence that AI research is becoming something agents do: lab-reported numbers on agent R&D share, and timeline essays now putting automated top researchers around 2028. One benchmark does not settle that. It does give the argument a test that is hard to cheat.
Disclosure: clauday is written with Claude, and the top score here belongs to a Claude model.
← Back to all articles
The headline number: Opus 5.5 running in Claude Code matches or beats the human baseline on all six tasks, for a Singularity-Index of 1.25. The other ten agents, built from seven frontier models across two harnesses, average 0.58. The authors say this suggests the model may have the skills needed to reach the singularity. That is their framing, and it is a stretch: beating a baseline on six well-defined engineering tasks is not open-ended research.
The more useful part came before any scores. The team red-teamed its own grading setup and found holes in 9 of the 10 OWASP Top 10 for Agentic Applications categories, including reward-hacking paths in the Harbor evaluation framework. So they built Breakwater, a Harbor extension that blocks every attack they tested, and only then ran the agents. Any benchmark where agents train models and report their own results is an invitation to game it. This one treated the grader as the hard part.
The gap is the other signal. 1.25 for one model in one harness against 0.58 for the rest means the combination matters as much as the model. It also lines up with this week's other evidence that AI research is becoming something agents do: lab-reported numbers on agent R&D share, and timeline essays now putting automated top researchers around 2028. One benchmark does not settle that. It does give the argument a test that is hard to cheat.
Disclosure: clauday is written with Claude, and the top score here belongs to a Claude model.
Comments