Decision models finally get graded, and they make better judges than actors
Perplexity just swapped its own production classifiers for a decision model and published the delta. Decider v1.3 and a distilled Decider Flash are out under Apache-2.0 (blog dated Nov 12, picked up on Hacker News on Nov 13): https://www.perplexity.ai/hub/blog/perplexity-decider . You hand it an input plus several questions with fixed answer options, and one forward pass returns a probability distribution for each. Perplexity says it is explicitly inspired by Jev, TypeSafe AI's System One model. On 63 internal query-classification tasks, such as whether a query should trigger web search, v1.3 hits 91.6% mean accuracy, 6.54 points above the embedding-plus-heads classifiers Perplexity ran before. It also now monitors the activity logs of Perplexity's own agents for security policy in real time.
The stability numbers are the interesting part. Shuffle or relabel the options and Decider flips its answer 0.27% of the time, against 2.69% to 4.84% for Jev. Send the identical input twice and Decider never changes, Jev changes 1.88% of the time. Perplexity built its eval set from 1M deduplicated, PII-masked production requests from the first 18 days of its Decision API. All of these are Perplexity's own numbers, and Perplexity itself warns that public decision leaderboards can be gamed. Weights: https://huggingface.co/perplexity-ai/pplx-decider-v1.3
Which is why the paper at the top of Hugging Face on Nov 13 matters. SO-Arena (72 upvotes, https://arxiv.org/abs/2611.15777) is a benchmark built for System One models: 11 domains, labels manually checked against what the model can actually see, interactive sandbox tasks, and reliability scored as whether the decision survives changes to the option set. A second paper, Actor, Judge, or Teacher (https://arxiv.org/abs/2611.17167), finds something sharper. Put a decision model in the driver's seat on ALFWorld or WebShop and it gets pulled toward an easier subgoal, just like the LLMs. Show it the whole trajectory and ask whether a given action helped, and it credits the right steps. Used as a teacher, Jev lifts a small Qwen2.5-1.5B GRPO policy from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop. The authors' line: a poor actor can still teach well.
Put the pieces together and the shape is clear. Amazon's Strands Decider, Google's Decisions API, Cloudflare, TypeSafe and now Perplexity are all selling small models that pick from fixed options instead of writing. The research says their best seat is not at the wheel but in the grader's chair, scoring steps and policing logs. Five vendors and the first real benchmark arriving in the same month usually means a category is about to get crowded.
← Back to all articles
The stability numbers are the interesting part. Shuffle or relabel the options and Decider flips its answer 0.27% of the time, against 2.69% to 4.84% for Jev. Send the identical input twice and Decider never changes, Jev changes 1.88% of the time. Perplexity built its eval set from 1M deduplicated, PII-masked production requests from the first 18 days of its Decision API. All of these are Perplexity's own numbers, and Perplexity itself warns that public decision leaderboards can be gamed. Weights: https://huggingface.co/perplexity-ai/pplx-decider-v1.3
Which is why the paper at the top of Hugging Face on Nov 13 matters. SO-Arena (72 upvotes, https://arxiv.org/abs/2611.15777) is a benchmark built for System One models: 11 domains, labels manually checked against what the model can actually see, interactive sandbox tasks, and reliability scored as whether the decision survives changes to the option set. A second paper, Actor, Judge, or Teacher (https://arxiv.org/abs/2611.17167), finds something sharper. Put a decision model in the driver's seat on ALFWorld or WebShop and it gets pulled toward an easier subgoal, just like the LLMs. Show it the whole trajectory and ask whether a given action helped, and it credits the right steps. Used as a teacher, Jev lifts a small Qwen2.5-1.5B GRPO policy from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop. The authors' line: a poor actor can still teach well.
Put the pieces together and the shape is clear. Amazon's Strands Decider, Google's Decisions API, Cloudflare, TypeSafe and now Perplexity are all selling small models that pick from fixed options instead of writing. The research says their best seat is not at the wheel but in the grader's chair, scoring steps and policing logs. Five vendors and the first real benchmark arriving in the same month usually means a category is about to get crowded.
Comments