Frontier models score 59.7% on taste, and thinking longer doesn't help
There's a word everyone uses about agents and nobody measures: taste. The sense that at a fork in a long task, this direction is better than that one. Microsoft's Wenbo Pan and eight collaborators finally put a number on it, and the number is 59.7%.
Taste-Bench mines decision forks out of agent trajectories, automatically, with no human annotation. Two ways: comparing parallel attempts at the same task where the runs diverged, and finding detours inside a single trajectory where the agent went somewhere and came back. Each fork becomes a question with multiple directions, only one of which leads somewhere better. Since outcomes are known after the fact, the ground truth comes free.
Frontier models get 59.7%. On multiple choice about which way to go next. That is the headline and it is worse than it sounds, because these are forks that real agents actually reached in real tasks, not adversarial constructions.
Two findings underneath matter more than the score. First, decisions whose supporting evidence only shows up later in the trajectory are much harder for every model tested β the agent has to commit before the information that would justify the commitment exists, which is the defining condition of long-horizon work and apparently the condition models are worst at. Second, and this is the one to sit with: more reasoning budget does not improve performance. You cannot think your way to taste. Extra tokens at the fork buy nothing.
The constructive half is that taste is trainable by distillation. A teacher model that gets to observe outcomes judges the forks; a student trained on those judgments improves on unseen tasks and, crucially, improves end-to-end success on held-out SWE-bench Pro. So the signal transfers out of the benchmark, which is the test most benchmark papers quietly fail.
The frame worth taking away: a lot of what gets blamed on model capability in long agent runs is actually a decision-quality failure at three or four specific moments, and scaling inference-time compute is the intervention that doesn't fix it. 33 pages, submitted September 22, code and dataset released.
https://arxiv.org/abs/2609.25804
← Back to all articles
Taste-Bench mines decision forks out of agent trajectories, automatically, with no human annotation. Two ways: comparing parallel attempts at the same task where the runs diverged, and finding detours inside a single trajectory where the agent went somewhere and came back. Each fork becomes a question with multiple directions, only one of which leads somewhere better. Since outcomes are known after the fact, the ground truth comes free.
Frontier models get 59.7%. On multiple choice about which way to go next. That is the headline and it is worse than it sounds, because these are forks that real agents actually reached in real tasks, not adversarial constructions.
Two findings underneath matter more than the score. First, decisions whose supporting evidence only shows up later in the trajectory are much harder for every model tested β the agent has to commit before the information that would justify the commitment exists, which is the defining condition of long-horizon work and apparently the condition models are worst at. Second, and this is the one to sit with: more reasoning budget does not improve performance. You cannot think your way to taste. Extra tokens at the fork buy nothing.
The constructive half is that taste is trainable by distillation. A teacher model that gets to observe outcomes judges the forks; a student trained on those judgments improves on unseen tasks and, crucially, improves end-to-end success on held-out SWE-bench Pro. So the signal transfers out of the benchmark, which is the test most benchmark papers quietly fail.
The frame worth taking away: a lot of what gets blamed on model capability in long agent runs is actually a decision-quality failure at three or four specific moments, and scaling inference-time compute is the intervention that doesn't fix it. 33 pages, submitted September 22, code and dataset released.
https://arxiv.org/abs/2609.25804
Comments