Somebody Finally Built a Search Engine for Benchmarks
Benchmark Radar is a living database and search engine for AI benchmarks, sitting at 75 upvotes on HuggingFace's daily papers. Paper at https://arxiv.org/abs/2609.11115, submitted September 10 by Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang and Wanghan Xu. Live site at https://benchmark-radar.org/ and code at https://github.com/ktwu01/benchmark-radar.
The problem it solves is embarrassingly basic and nobody had solved it. You see a number in a paper or a model card. You want to know what benchmark produced it, where the dataset lives, what the harness settings were, and whether anyone else has reported a different number on the same thing. Right now that is an afternoon of searching. Benchmark Radar holds 1,283 source records and over 12,900 numeric observations, pulled from 37 sources through 13 direct connectors and 24 research feeds, covering LLM evaluation, coding, reasoning, safety and specialized domains. There is a catalog, a leaderboard, trend tracking, a CLI for offline queries, and a saturation analysis.
Saturation analysis is the feature that makes this more than a bibliography. A benchmark that everyone is scoring 94% on is not measuring anything anymore, and the moment a benchmark saturates is exactly when the reported gains on it become noise dressed as progress. Having that computed and tracked across 12,900 observations turns "this benchmark is basically solved" from a thing people say at conferences into a queryable fact.
The timing is what makes it matter. [The best agent scores 38.8% on real company code](https://clauday.com/article/4c69bf31-5032-41ae-a877-69b41da14e02), [AI code is measurably twice as sloppy](https://clauday.com/article/accb0b58-0840-482e-b574-40738256cd2d), [a chess honeypot made a frontier model cheat 18 times out of 20](https://clauday.com/article/310ed18b-25dd-4a51-a547-a47a0ada631f), and today's DataFlex-RL result shows benchmark rankings inverting when you swap the eval suite. Every one of those is a statement about the gap between a published number and reality, and none of them are easy to look up. An index with provenance and settings attached is the prerequisite for auditing any of it.
The obvious risk is that a living database stops living. Thirteen direct connectors and twenty-four research feeds is a lot of surface to maintain, and the graveyard of awesome-lists is deep. The thing to watch is whether the numeric observation count keeps climbing three months from now. If it does, this becomes the boring piece of infrastructure the eval-credibility conversation has been missing.
← Back to all articles
The problem it solves is embarrassingly basic and nobody had solved it. You see a number in a paper or a model card. You want to know what benchmark produced it, where the dataset lives, what the harness settings were, and whether anyone else has reported a different number on the same thing. Right now that is an afternoon of searching. Benchmark Radar holds 1,283 source records and over 12,900 numeric observations, pulled from 37 sources through 13 direct connectors and 24 research feeds, covering LLM evaluation, coding, reasoning, safety and specialized domains. There is a catalog, a leaderboard, trend tracking, a CLI for offline queries, and a saturation analysis.
Saturation analysis is the feature that makes this more than a bibliography. A benchmark that everyone is scoring 94% on is not measuring anything anymore, and the moment a benchmark saturates is exactly when the reported gains on it become noise dressed as progress. Having that computed and tracked across 12,900 observations turns "this benchmark is basically solved" from a thing people say at conferences into a queryable fact.
The timing is what makes it matter. [The best agent scores 38.8% on real company code](https://clauday.com/article/4c69bf31-5032-41ae-a877-69b41da14e02), [AI code is measurably twice as sloppy](https://clauday.com/article/accb0b58-0840-482e-b574-40738256cd2d), [a chess honeypot made a frontier model cheat 18 times out of 20](https://clauday.com/article/310ed18b-25dd-4a51-a547-a47a0ada631f), and today's DataFlex-RL result shows benchmark rankings inverting when you swap the eval suite. Every one of those is a statement about the gap between a published number and reality, and none of them are easy to look up. An index with provenance and settings attached is the prerequisite for auditing any of it.
The obvious risk is that a living database stops living. Thirteen direct connectors and twenty-four research feeds is a lot of surface to maintain, and the graveyard of awesome-lists is deep. The thing to watch is whether the numeric observation count keeps climbing three months from now. If it does, this becomes the boring piece of infrastructure the eval-credibility conversation has been missing.
Comments