Long-Horizon-Terminal-Bench: Best Agent Scores 15 Percent
The top agent paper on HuggingFace right now, with 45 upvotes: Long-Horizon-Terminal-Bench (arXiv 2607.08964, submitted July 9). The headline number is brutal — across 46 long-horizon terminal tasks spanning nine domains, the strongest model manages a 15.2 percent success rate at the high-performance threshold, burning an average of 9.9 million tokens per task.
The design criticism behind the benchmark is the interesting part. Existing terminal benchmarks mostly test problems that finish within minutes and grade only the final outcome — pass or fail, nothing in between. That made sense when agents could barely chain three commands. It is useless for measuring what everyone actually cares about in 2026: can an agent work for hours on something real. So this benchmark grades with dense rewards and partial credit — intermediate signals along the way, so an agent that got 70 percent through a gnarly infrastructure task scores differently from one that faceplanted at step two.
Put this next to the leaderboards everyone brags about. SWE-Bench is basically saturated, labs trade decimal points on coding evals, and the marketing says agents are ready for real work. Then a benchmark makes the tasks long, the grading honest, and the best frontier model lands at 15 percent. Both things are true: agents genuinely got good at bounded tasks, and the long-horizon frontier is still wide open. Benchmarks that measure the gap honestly — this, UniClawBench with its hidden graders, dense-reward designs — are worth more right now than another saturated leaderboard.
Paper: https://arxiv.org/abs/2607.08964
← Back to all articles
The design criticism behind the benchmark is the interesting part. Existing terminal benchmarks mostly test problems that finish within minutes and grade only the final outcome — pass or fail, nothing in between. That made sense when agents could barely chain three commands. It is useless for measuring what everyone actually cares about in 2026: can an agent work for hours on something real. So this benchmark grades with dense rewards and partial credit — intermediate signals along the way, so an agent that got 70 percent through a gnarly infrastructure task scores differently from one that faceplanted at step two.
Put this next to the leaderboards everyone brags about. SWE-Bench is basically saturated, labs trade decimal points on coding evals, and the marketing says agents are ready for real work. Then a benchmark makes the tasks long, the grading honest, and the best frontier model lands at 15 percent. Both things are true: agents genuinely got good at bounded tasks, and the long-horizon frontier is still wide open. Benchmarks that measure the gap honestly — this, UniClawBench with its hidden graders, dense-reward designs — are worth more right now than another saturated leaderboard.
Paper: https://arxiv.org/abs/2607.08964
Comments