JEV-as-a-Judge Answers the Sharpest Critique of Calibrated Decision Models, Halfway
The Jev backlash arrived on Sunday. The Normalization of Inexplicable Failures, a blog post on ihatethefuture.com, hit 224 points on Hacker News with a blunt charge against calibrated decision models like Jev: to know whether one works, you need evals and a ground-truth pipeline, and if you have those you are most of the way to fine-tuning your own. Nobody buying it runs evals, the author argues. Confidence scores get used as cargo cult, with a 0.9 threshold picked over lunch. Even Jev's docs admit the right thresholds depend on your domain.
It is a good post, and the best reply to it is a paper that does exactly the homework the post says nobody does. JEV-as-a-Judge, by Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman, compares a decision-only JEV judge against sixteen generative and reward-model judges, with blinded human adjudication. On ordinary preference and evidence-grounded factuality it lands within three points of the strongest LLM judge at 0.36% of the fee. Where it falls down is equally specific: checking a derivation, and resisting a long, well-written wrong answer.
Here is the part that answers the critique. On several benchmarks JEV's gap to the strong judge sits almost entirely in its low-confidence decisions. So the confidence score is not cargo cult there. It is a routing signal. A frozen cascade that accepts confident verdicts and escalates the unsure ones keeps 99% of the strong judge's accuracy at lower cost.
That is the halfway. The paper proves confidence is useful when someone measures it against human labels first, which is precisely the eval pipeline the blog says buyers skip. Both are right. For teams on the output-space-restriction bandwagon, the takeaway is simple: a calibrated model does not remove the eval work, it tells you where to spend it. Pick the escalation threshold from your own labeled data, not from the docs.
Links: arxiv.org/abs/2609.26550, ihatethefuture.com/2026/09/the-normalization-of-inexplicable.html
← Back to all articles
It is a good post, and the best reply to it is a paper that does exactly the homework the post says nobody does. JEV-as-a-Judge, by Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman, compares a decision-only JEV judge against sixteen generative and reward-model judges, with blinded human adjudication. On ordinary preference and evidence-grounded factuality it lands within three points of the strongest LLM judge at 0.36% of the fee. Where it falls down is equally specific: checking a derivation, and resisting a long, well-written wrong answer.
Here is the part that answers the critique. On several benchmarks JEV's gap to the strong judge sits almost entirely in its low-confidence decisions. So the confidence score is not cargo cult there. It is a routing signal. A frozen cascade that accepts confident verdicts and escalates the unsure ones keeps 99% of the strong judge's accuracy at lower cost.
That is the halfway. The paper proves confidence is useful when someone measures it against human labels first, which is precisely the eval pipeline the blog says buyers skip. Both are right. For teams on the output-space-restriction bandwagon, the takeaway is simple: a calibrated model does not remove the eval work, it tells you where to spend it. Pick the escalation threshold from your own labeled data, not from the docs.
Links: arxiv.org/abs/2609.26550, ihatethefuture.com/2026/09/the-normalization-of-inexplicable.html
Comments