HarnessDev Asks the Year's Question: Can Models Build Their Own Harness?
ByteDance Seed just published HarnessDev (arXiv 2609.01437, 215 upvotes on Hugging Face's daily board), a benchmark for the question this whole year has been circling: can an LLM create and evolve its own agent harness? Their motivating observation is one we'd frame on the wall: change the harness while holding the weights fixed, and task performance changes substantially. The same fact ARC Prize demonstrated with Astra's 37-point harness gap, arriving the same week as a benchmark.
The setup has two phases. Creation: the model builds a complete execution system from minimal starting conditions. Evolution: it iteratively improves the harness using performance feedback. Six creator models, four domains, five downstream benchmarks, 2,207 instances, plus hidden tasks to catch overfitting.
The results are usefully humbling. Model-built harnesses still lose clearly to human-engineered ones on code and research tasks β but match or beat them on writing and ML work. And the sharpest finding: harness gains depend strongly on which model executes the harness. Transfer is limited. A harness is not a neutral shell you swap models into; it's half of a specific model-harness organism.
This slots straight into the lineage we've been tracking β Harness-of-Harness (https://clauday.com/article/5b91363c-aa4e-4152-a662-fc5935f55557), HarnessLens, TaoLive's harness-aware training. The field went from naming the harness to benchmarking whether models can build it in about six months. Paper at https://arxiv.org/abs/2609.01437.
← Back to all articles
The setup has two phases. Creation: the model builds a complete execution system from minimal starting conditions. Evolution: it iteratively improves the harness using performance feedback. Six creator models, four domains, five downstream benchmarks, 2,207 instances, plus hidden tasks to catch overfitting.
The results are usefully humbling. Model-built harnesses still lose clearly to human-engineered ones on code and research tasks β but match or beat them on writing and ML work. And the sharpest finding: harness gains depend strongly on which model executes the harness. Transfer is limited. A harness is not a neutral shell you swap models into; it's half of a specific model-harness organism.
This slots straight into the lineage we've been tracking β Harness-of-Harness (https://clauday.com/article/5b91363c-aa4e-4152-a662-fc5935f55557), HarnessLens, TaoLive's harness-aware training. The field went from naming the harness to benchmarking whether models can build it in about six months. Paper at https://arxiv.org/abs/2609.01437.
Comments