Jev Makes Every Decision Cheaper. Across 4,500 Runs, Cost per Successful Task Went Up for One Agent.
Decision models were the fastest-moving thread on this site last week: OpenAI's Decisions API, Microsoft-Decision-1, a build-your-own tutorial at the top of Hacker News, and TypedBench saying the probabilities are not calibrated. Monday's arXiv adds the measurement everyone skipped: what happens to the task, not the decision, when you drop a fast decision model into an agent loop. The paper is When Faster Decisions Do Not Make Better Agents, an empirical study of Jev across five agents in two settings, embodied tasks in ALFWorld and mobile app use in AndroidWorld, 4,500 task executions in total.
The per-decision result is exactly what the vendors promise. Jev integration substantially reduces decision cost and latency across all five agents. The task-level result is where it gets uncomfortable. On ALFWorld, success stays close to baseline. On two of the mobile agents it declines substantially. The savings get absorbed: agents spend the freed budget on auxiliary reasoning and additional interactions, and the failure modes that eat delivery are failed recovery and premature completion, the agent declaring done before it is. In one agent, mean cost per successful outcome rose even though spending per attempt fell. Cheaper decisions, more attempts, fewer finishes.
The authors' recommendation reads like a checklist for anyone wiring a decision model in this month: preserve progress, support recovery, verify completion, and judge the value by successful deliveries and resources per delivered outcome, not by the per-call price. That is the cost-per-task number this site has been asking vendors for since the September price cuts, measured here by a third party and coming out the wrong way for some workflows.
This is also a direct counterweight to SearchJev, published last week as the first case of a decision model that was both faster and more accurate. Same model family, different workflow, opposite conclusion. The decision model is not the variable. The harness around it is.
Paper: https://arxiv.org/abs/2610.13092
← Back to all articles
The per-decision result is exactly what the vendors promise. Jev integration substantially reduces decision cost and latency across all five agents. The task-level result is where it gets uncomfortable. On ALFWorld, success stays close to baseline. On two of the mobile agents it declines substantially. The savings get absorbed: agents spend the freed budget on auxiliary reasoning and additional interactions, and the failure modes that eat delivery are failed recovery and premature completion, the agent declaring done before it is. In one agent, mean cost per successful outcome rose even though spending per attempt fell. Cheaper decisions, more attempts, fewer finishes.
The authors' recommendation reads like a checklist for anyone wiring a decision model in this month: preserve progress, support recovery, verify completion, and judge the value by successful deliveries and resources per delivered outcome, not by the per-call price. That is the cost-per-task number this site has been asking vendors for since the September price cuts, measured here by a third party and coming out the wrong way for some workflows.
This is also a direct counterweight to SearchJev, published last week as the first case of a decision model that was both faster and more accurate. Same model family, different workflow, opposite conclusion. The decision model is not the variable. The harness around it is.
Paper: https://arxiv.org/abs/2610.13092
Comments