SimpleBench Says GPT-6.2 Got Worse. The Coding Index Said Better.
Every new frontier model got better on SimpleBench except OpenAI's. That's the top comment on Saturday's Hacker News thread "Astra 6.2 and Sol 6.2 flop at Simple Bench" (154 points), and the leaderboard backs it up. GPT-6.2 Astra scores 77.3%, down from 83.6% for GPT-6 Astra. GPT-6.2 Astra Pro gets 79.2% against GPT-6 Astra Pro's 86.5%. GPT-6.2 Sol is at 69.9%, 13 points below GPT-6.1 Sol's 82.9%, and it sits 30th. Leaderboard: https://simple-bench.com/
The top of the board went the other way. Claude Fable 5.2 is new at number one with 94.9%, Gemini 4 Neon is second at 94.4%, and the Gemini 4 family takes the next three spots. The page still has a struck-out line saying the old top model, Claude Fable, scored 81.9% and lost to humans. Fable 5.2 is now 0.5 points from the best human score of 95.4% and well past the human baseline of 83.7%.
Read the limits before you tweet the chart. SimpleBench has a bit over 200 multiple-choice questions on spatial and temporal reasoning, social intelligence and trick questions, scored as an average over five runs. The human baseline comes from "our small sample of nine participants." The page shows no update date, so the timing rests on the Hacker News thread from November 21. It's one team's benchmark, and it measures one thing: whether a model notices that the question is a trap.
What makes it worth a post is the contradiction. A day earlier the Models Engineering Index, which combines 16 coding benchmarks, had GPT-6.2 Astra at number two, ahead of Fable 5.2. Same model, opposite verdict. Both can be right. A model tuned hard for long agentic coding runs can get worse at reading a sneaky one-paragraph puzzle, and that's a known trade when post-training chases one target. My read: OpenAI optimized 6.2 for the work it sells, and SimpleBench is measuring what that cost.
For builders the takeaway is boring but true. A single leaderboard number tells you which test a model is good at, not which model is good. If your agent spends its day on tidy code tasks, the coding composite is your benchmark. If it reads messy human requests where the real question is buried in the third sentence, SimpleBench is closer to your reality. Run your own evals before you switch.
← Back to all articles
The top of the board went the other way. Claude Fable 5.2 is new at number one with 94.9%, Gemini 4 Neon is second at 94.4%, and the Gemini 4 family takes the next three spots. The page still has a struck-out line saying the old top model, Claude Fable, scored 81.9% and lost to humans. Fable 5.2 is now 0.5 points from the best human score of 95.4% and well past the human baseline of 83.7%.
Read the limits before you tweet the chart. SimpleBench has a bit over 200 multiple-choice questions on spatial and temporal reasoning, social intelligence and trick questions, scored as an average over five runs. The human baseline comes from "our small sample of nine participants." The page shows no update date, so the timing rests on the Hacker News thread from November 21. It's one team's benchmark, and it measures one thing: whether a model notices that the question is a trap.
What makes it worth a post is the contradiction. A day earlier the Models Engineering Index, which combines 16 coding benchmarks, had GPT-6.2 Astra at number two, ahead of Fable 5.2. Same model, opposite verdict. Both can be right. A model tuned hard for long agentic coding runs can get worse at reading a sneaky one-paragraph puzzle, and that's a known trade when post-training chases one target. My read: OpenAI optimized 6.2 for the work it sells, and SimpleBench is measuring what that cost.
For builders the takeaway is boring but true. A single leaderboard number tells you which test a model is good at, not which model is good. If your agent spends its day on tidy code tasks, the coding composite is your benchmark. If it reads messy human requests where the real question is buried in the third sentence, SimpleBench is closer to your reality. Run your own evals before you switch.
Comments