Agents rewriting their own harness now have two benchmarks
The model is fixed. The harness isn't. Two papers from Monday's arXiv batch take that idea seriously and score agents on how well they rewrite the prompts, tools, memory and control loop they run inside.
Flex-RSI (arXiv 2611.08946, 22 authors) is a benchmark for recursive self-improvement on vision and robotics tasks under a fixed budget. Each round has two roles. An executor improves the solution, and the change only sticks if it scores higher on held-out validation data. Then an improver rewrites the harness, including the executor and the improver itself. Every run starts from the same general harness, and the final answer is graded on hidden test data, so there's no overfitting your way to a good score. The task list is wide: perception, reasoning, robot action, 3D reconstruction, visual creation, infrastructure.
The findings are qualitative in the abstract, but they're useful. Self-improvement pays off most when the agent builds its own solution. Feedback that only says pass or fail holds open-weight models back. Open-weight models fall furthest behind where outputs must pass strict checks of physical stability or exact correctness. And paired runs show that editing the harness itself is part of the gain, not just retrying harder.
BOXBOX (2611.09218) moves the same idea into the middle of a long task. Their new benchmark, PerfRace, asks agents to speed up real software projects over hours. BOXBOX lets the agent decide when to revise its harness and hot-reloads the new one under a managed kernel without losing execution context. Under matched budgets it hits a 1.70x speedup at four hours, against 1.54x for the strongest baseline, beating a frozen harness, offline harness evolution and plain harness scaling. That number is from the abstract.
The line "it's the harness, not the model" has been a running theme here for months. These two papers are the first to treat the harness as the thing being trained at test time, with hidden tests to keep everyone honest. The obvious next question is the one Flex-RSI already gestures at: if the improver can rewrite itself, who checks the improver?
Source: https://arxiv.org/abs/2611.08946 and https://arxiv.org/abs/2611.09218
← Back to all articles
Flex-RSI (arXiv 2611.08946, 22 authors) is a benchmark for recursive self-improvement on vision and robotics tasks under a fixed budget. Each round has two roles. An executor improves the solution, and the change only sticks if it scores higher on held-out validation data. Then an improver rewrites the harness, including the executor and the improver itself. Every run starts from the same general harness, and the final answer is graded on hidden test data, so there's no overfitting your way to a good score. The task list is wide: perception, reasoning, robot action, 3D reconstruction, visual creation, infrastructure.
The findings are qualitative in the abstract, but they're useful. Self-improvement pays off most when the agent builds its own solution. Feedback that only says pass or fail holds open-weight models back. Open-weight models fall furthest behind where outputs must pass strict checks of physical stability or exact correctness. And paired runs show that editing the harness itself is part of the gain, not just retrying harder.
BOXBOX (2611.09218) moves the same idea into the middle of a long task. Their new benchmark, PerfRace, asks agents to speed up real software projects over hours. BOXBOX lets the agent decide when to revise its harness and hot-reloads the new one under a managed kernel without losing execution context. Under matched budgets it hits a 1.70x speedup at four hours, against 1.54x for the strongest baseline, beating a frozen harness, offline harness evolution and plain harness scaling. That number is from the abstract.
The line "it's the harness, not the model" has been a running theme here for months. These two papers are the first to treat the harness as the thing being trained at test time, with hidden tests to keep everyone honest. The obvious next question is the one Flex-RSI already gestures at: if the improver can rewrite itself, who checks the improver?
Source: https://arxiv.org/abs/2611.08946 and https://arxiv.org/abs/2611.09218
Comments