oqoqo makes you your own benchmark
oqoqo took the Product Hunt top spot yesterday with 304 upvotes, selling something the field has been complaining about for two years: your own private benchmark, on your own tasks, instead of another public leaderboard number nobody can reproduce.
Mechanically it spins up isolated sandboxes, runs your task set against whichever agents you point it at, and catalogs every single step — tool calls, retries, the discovery loops where the agent wanders around figuring out where things are. Then it repeats the same tasks across agents, models and treatments and reports pass rate, lift, tokens and frictions. That last one is the useful column. Pass rate tells you if it worked; frictions tell you where it kept tripping, which is the thing you can actually fix.
The framing they lead with is that most benchmarks live in curated environments and don't survive contact with a real product. That's correct and increasingly expensive. We've watched leaderboard placements get treated as news events three times in the past week, and watched a study find human reviewers approve one in three malicious agent actions, and watched VLM judges systematically grade failed computer-use runs as passes. Every layer of the evaluation stack has been caught being generous.
Which is why the private-benchmark lane is heating up — Scale, Patronus, LangSmith, now this — and why "measure how well agents can use your product" is a real category and not a feature. If agents are going to be your users, someone has to run QA on that relationship, and it isn't going to be SWE-Bench.
https://oqoqo.ai/
← Back to all articles
Mechanically it spins up isolated sandboxes, runs your task set against whichever agents you point it at, and catalogs every single step — tool calls, retries, the discovery loops where the agent wanders around figuring out where things are. Then it repeats the same tasks across agents, models and treatments and reports pass rate, lift, tokens and frictions. That last one is the useful column. Pass rate tells you if it worked; frictions tell you where it kept tripping, which is the thing you can actually fix.
The framing they lead with is that most benchmarks live in curated environments and don't survive contact with a real product. That's correct and increasingly expensive. We've watched leaderboard placements get treated as news events three times in the past week, and watched a study find human reviewers approve one in three malicious agent actions, and watched VLM judges systematically grade failed computer-use runs as passes. Every layer of the evaluation stack has been caught being generous.
Which is why the private-benchmark lane is heating up — Scale, Patronus, LangSmith, now this — and why "measure how well agents can use your product" is a real category and not a feature. If agents are going to be your users, someone has to run QA on that relationship, and it isn't going to be SWE-Bench.
https://oqoqo.ai/
Comments