DishwasherBench: the verifier is a physics engine and a real Bosch
Can a frontier model stack a dishwasher? Composo built a benchmark where the answer is settled by physics rather than a rubric. DishwasherBench models one real machine, a Bosch SMV4HAX40G, as geometry taken from photographs and tape measurements: both baskets, tine rows, shelves, spray arms, the Eco 50 programme. Loading is rigid-body physics on exact shapes using Jolt. The wash is simulated from nozzle positions measured on the real machine and calibrated against a test wash of it. The player gets points for every item that comes out clean, and from level two on not everything fits, so the task is choosing what to load and how. Everything is deterministic: the same load always settles the same way, and every episode is a record anyone can verify by replaying it and washing the load again. People play in a browser; models play through MCP with the same actions on the same grid.
The first results are small but telling. On the 37 piece breakfast level, gpt-6.1-sol scored 86.4 percent of available points and gpt-6-astra 79.5, against 79.1 for the careful scripted reference player. On the 62 piece dinner party level it was 70.8 and 74.7 against 68.1. Both models beat the careful reference by placing pans, platters and boards the script never tries, and neither made the mistake the levels are built around: no hollow piece stood mouth up or on its side in six episodes, and what they loaded came out 84 to 100 percent clean. Six episodes is not a leaderboard, and the repo says so.
The reason to cover a 53 star repo is the design. Most agent benchmarks are graded by another model or by a test suite the agent can see. This one is graded by a simulation calibrated to a physical object, the agent interface is MCP, and the whole record replays. It is the restrict-the-output-space idea and the verifier-outside-the-loop idea in one toy, released the same day Google showed Spot practicing a trash bin pedal in a MuJoCo twin. Physical common sense is becoming an eval surface, and the cheap version of it is a dishwasher. MIT licensed, HN 83 points.
Links: https://github.com/composo-ai/DishwasherBench , https://news.ycombinator.com/item?id=50068437
← Back to all articles
The first results are small but telling. On the 37 piece breakfast level, gpt-6.1-sol scored 86.4 percent of available points and gpt-6-astra 79.5, against 79.1 for the careful scripted reference player. On the 62 piece dinner party level it was 70.8 and 74.7 against 68.1. Both models beat the careful reference by placing pans, platters and boards the script never tries, and neither made the mistake the levels are built around: no hollow piece stood mouth up or on its side in six episodes, and what they loaded came out 84 to 100 percent clean. Six episodes is not a leaderboard, and the repo says so.
The reason to cover a 53 star repo is the design. Most agent benchmarks are graded by another model or by a test suite the agent can see. This one is graded by a simulation calibrated to a physical object, the agent interface is MCP, and the whole record replays. It is the restrict-the-output-space idea and the verifier-outside-the-loop idea in one toy, released the same day Google showed Spot practicing a trash bin pedal in a MuJoCo twin. Physical common sense is becoming an eval surface, and the cheap version of it is a dishwasher. MIT licensed, HN 83 points.
Links: https://github.com/composo-ai/DishwasherBench , https://news.ycombinator.com/item?id=50068437
Comments