A 60-Line Decision Model Hit the Top of HN. Its Confidence Scores Are the Problem.
Nish Tahir wrote the post everyone building on Jev-style models should have read a month ago, and Hacker News put it at 449 points on Saturday night. The idea of a System One decision model is simple once you see it: a normal LLM generating JSON needs a forward pass per token, eleven passes in the post's example. A decision model assumes the answer is one of a fixed set of options, masks every other token in the vocabulary, and reads the answer off a single pass. The post rebuilds it in about sixty lines on Qwen3-1.7B, constraining the first assistant token to A through E and taking the softmax over just those logits. On a CommonsenseQA holdout the toy gets 59.4 percent, a quick finetune gets 62.4. That is the whole trick that TypeSafe raised 870 million dollars on and Microsoft, OpenAI, Cloudflare and Amazon all shipped in the last month.
Then the post does the thing the vendors do not. Treating the option probabilities as confidence, the post bins them against accuracy. In the 0.9 to 1.0 bin the model is right 70 percent of the time. In the 0.8 to 0.9 bin it is right about 40 percent. An ambiguous question about where you would find a bat comes back as cave at 0.998. Without extra training, the number you get is the model's confidence in the next token, not the probability the answer is correct, and temperature scaling only flattens the curve. If you wire that number into a threshold or an escalation rule, the software acts on a lie.
TypedBench, submitted to arXiv on Wednesday by Rahul Sharma, Andrew Duncan, GaΓ©tan Marceau Caron and Sebastian Vollmer, is the formal version of that complaint. Their point is that decision models are consumed by code, not people, through thresholds, cost-weighted choices and escalation, with no text rationale to inspect, so accuracy on a classification set is not enough. They build seven policy-labelled generators and nine suites, report accuracy as a median and range across paraphrases, measure calibration against the noise floor of a perfectly calibrated predictor, and score realised cost under asymmetric cost matrices. The hosted model follows the stated policy but is wording-sensitive and systematically underconfident, and under asymmetric costs using its probabilities does worse than just taking its top answer. The open decoders from 0.8B to 9B route exactly but slow down as options grow and are least accurate on policy questions.
So the week's tally on decision models: a 7.5 billion dollar valuation, a Microsoft entrant at 0.042 dollars per million tokens, an attack paper showing six lines of log noise flip the gate open, and now a blog post and a benchmark both saying the probability you are paying for is not a probability. Routing is a real market. Calibration is still the product nobody has shipped.
Post: https://nishtahir.com/build-your-own-decision-model/
HN: https://news.ycombinator.com/item?id=50037949
TypedBench: https://arxiv.org/abs/2610.11392
← Back to all articles
Then the post does the thing the vendors do not. Treating the option probabilities as confidence, the post bins them against accuracy. In the 0.9 to 1.0 bin the model is right 70 percent of the time. In the 0.8 to 0.9 bin it is right about 40 percent. An ambiguous question about where you would find a bat comes back as cave at 0.998. Without extra training, the number you get is the model's confidence in the next token, not the probability the answer is correct, and temperature scaling only flattens the curve. If you wire that number into a threshold or an escalation rule, the software acts on a lie.
TypedBench, submitted to arXiv on Wednesday by Rahul Sharma, Andrew Duncan, GaΓ©tan Marceau Caron and Sebastian Vollmer, is the formal version of that complaint. Their point is that decision models are consumed by code, not people, through thresholds, cost-weighted choices and escalation, with no text rationale to inspect, so accuracy on a classification set is not enough. They build seven policy-labelled generators and nine suites, report accuracy as a median and range across paraphrases, measure calibration against the noise floor of a perfectly calibrated predictor, and score realised cost under asymmetric cost matrices. The hosted model follows the stated policy but is wording-sensitive and systematically underconfident, and under asymmetric costs using its probabilities does worse than just taking its top answer. The open decoders from 0.8B to 9B route exactly but slow down as options grow and are least accurate on policy questions.
So the week's tally on decision models: a 7.5 billion dollar valuation, a Microsoft entrant at 0.042 dollars per million tokens, an attack paper showing six lines of log noise flip the gate open, and now a blog post and a benchmark both saying the probability you are paying for is not a probability. Routing is a real market. Calibration is still the product nobody has shipped.
Post: https://nishtahir.com/build-your-own-decision-model/
HN: https://news.ycombinator.com/item?id=50037949
TypedBench: https://arxiv.org/abs/2610.11392
Comments