November 20, 2026ResearchAgentsBenchmark

The Resolution Hypothesis: Self-Improving Agents Are Picking Winners at Random

Self-improving agents have a dirty secret, and a new paper puts a number on it. The loop everyone uses is simple: a proposer writes candidate prompts, skills or memories, and selection keeps whichever scores higher. The Resolution Hypothesis (arXiv 2611.23068, submitted Nov 17, number two on Hugging Face's Thursday board) argues the loop stalls not because the proposals are bad but because noisy evaluations cannot tell good from bad. Paper: https://arxiv.org/abs/2611.23068

The evidence is big: seven evolution methods, six models, over 10 million LLM calls. At the candidate level, helpful artifacts do show up, but selection is no better than random at separating them from harmful ones, so errors pile up round after round. At the method level it gets worse. The few percentage points that usually separate methods in papers are smaller than the gap between two independent runs of the same method. In plain words, a lot of published leaderboards for self-improving agents are luck with a confident chart.

This lands one day after a natural experiment showed higher-benchmark coding models did not get more work merged. Same disease, different organ. If your measuring stick wobbles more than the thing you are measuring, every loop built on it is a random walk with extra compute. The authors call for a shift from inventing more ways to propose improvements to making real improvements detectable and accumulable. That is the verifier-is-the-product argument applied to agents improving themselves.

The constructive answer showed up on the same board. Agentic Harness Engineering Beta (arXiv 2611.24924, Nov 18) changes the selector instead of the proposer: simulated users file requirements as issues while they work, a team of agents implements general changes rather than task patches, and a change merges only if it resolves a verified requirement and passes review. On held-out tasks pass@1 goes from 21.9% to 33.3% with 40.8% fewer tokens than the base harness, and real users preferred it in 31 of 48 blind pairs. Paper: https://arxiv.org/abs/2611.24924

The takeaway for anyone running an autoresearch loop at home: before you buy more proposals, run your eval twice on the same candidate and look at the spread. If the spread is bigger than your typical improvement, you are not evolving anything. You are rolling dice and keeping the receipts.
← Previous
OpenAI Performance Factory: Codex Fixes ChatGPT, a Human Still Presses Merge
Next β†’
Gemini Tool Retrieval and Mac Computer Use: Google Ships the Plumbing
← Back to all articles

Comments

Loading...
>_