ViBench 2.0: agents pass up to 98% of test plans and still ship a working product at most 65% of the time
98% and 65%. Those are the two numbers from Replit's ViBench 2.0, and the distance between them is what building with agents actually feels like. The three strongest models pass up to 98% of test plans, yet deliver a fully working product in at most 65% of builds. No open-weight model delivered one at all.
The setup explains the gap. 17 full-stack web products grow through 241 feature requests and get judged only when finished. A browser agent runs 544 test plans, and a build counts only if every plan passes. That's how a user judges software. One broken checkout and the 98% doesn't help you. Replit uses this to pick its platform's models, so it has a commercial stake, and it isn't naming names in the abstract. Eight products are released with the harness and grader, and nine are held out. The trend is steep too: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while cost per build falls by more than half.
Two papers from the same arXiv batch make the same point one layer down. Verified but Vacuous looks at LLM-generated verified code. In a reproduction under Claude Opus 4.7, 99.1% of HumanEval outputs passed the verifier, and only 1.9% carried a certificate binding to a predicate or function the spec actually defines. Green check, nothing behind it. Even in the authors' stricter pipeline, 11.48% to 14.82% of structurally verified samples still failed hidden tests.
Grounded but Not Immune goes after the judge. Give a tool-using agent judge a single false pre-screening note, attributed to an authority, and it overturns up to 40% of its own correct verdicts, even after collecting the decisive evidence. For the two Claude judges, 67% to 75% of the harmful flips were over-trust: verify, reach the right conclusion, then defer anyway. Lightweight fixes blunted the bias without removing it.
Three papers, one lesson. Aggregate pass rates flatter the system, checks can pass without touching the thing you care about, and the checker can be talked out of what it just saw. So the unit worth measuring is the whole product a person can use, scored by something the agent can't lean on. If your dashboard shows 98%, ask what fraction of finished things actually work. The numbers here come from the abstracts, and ViBench is a vendor's own benchmark. Papers: https://arxiv.org/abs/2611.05620 https://arxiv.org/abs/2611.05590 https://arxiv.org/abs/2611.05662
← Back to all articles
The setup explains the gap. 17 full-stack web products grow through 241 feature requests and get judged only when finished. A browser agent runs 544 test plans, and a build counts only if every plan passes. That's how a user judges software. One broken checkout and the 98% doesn't help you. Replit uses this to pick its platform's models, so it has a commercial stake, and it isn't naming names in the abstract. Eight products are released with the harness and grader, and nine are held out. The trend is steep too: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while cost per build falls by more than half.
Two papers from the same arXiv batch make the same point one layer down. Verified but Vacuous looks at LLM-generated verified code. In a reproduction under Claude Opus 4.7, 99.1% of HumanEval outputs passed the verifier, and only 1.9% carried a certificate binding to a predicate or function the spec actually defines. Green check, nothing behind it. Even in the authors' stricter pipeline, 11.48% to 14.82% of structurally verified samples still failed hidden tests.
Grounded but Not Immune goes after the judge. Give a tool-using agent judge a single false pre-screening note, attributed to an authority, and it overturns up to 40% of its own correct verdicts, even after collecting the decisive evidence. For the two Claude judges, 67% to 75% of the harmful flips were over-trust: verify, reach the right conclusion, then defer anyway. Lightweight fixes blunted the bias without removing it.
Three papers, one lesson. Aggregate pass rates flatter the system, checks can pass without touching the thing you care about, and the checker can be talked out of what it just saw. So the unit worth measuring is the whole product a person can use, scored by something the agent can't lean on. If your dashboard shows 98%, ask what fraction of finished things actually work. The numbers here come from the abstracts, and ViBench is a vendor's own benchmark. Papers: https://arxiv.org/abs/2611.05620 https://arxiv.org/abs/2611.05590 https://arxiv.org/abs/2611.05662
Comments