DeliverBench: best agent system gets 65/100 once it has to live with its own earlier work
Most task benchmarks hand the agent a clean starting point every time, so a bad answer on step three never poisons step four. Real projects are nothing but that poison. DeliverBench, posted November 2 by 21 authors, finally scores it.
The setup: 11 projects in financial due diligence, legal drafting and drug safety reporting, built with 13 domain experts, chaining 150 subtasks through 2 to 15 rounds of new material and client feedback. No reference outputs get injected between rounds, so agents inherit whatever they got wrong earlier. Across 7 model-harness systems, the best scores 65.0 out of 100 on the final delivery and passes 50.0% of subtasks at a 90-point threshold. The sharpest detail is from an audit project: all seven systems addressed the review findings, and only one preserved traceability. Fixing the thing and being able to show how you fixed it turn out to be separate skills. In audit, the second one is the job.
A second paper measures what happens when the work passes between agents. The Collaborative Intelligence Benchmark has 240 tasks across 8 teamwork dimensions, drawn from aviation accident reports, clinical cases, coding tasks and negotiations, and it only scores models on tasks they can solve alone with full information, so it isolates teamwork from raw ability. Of 18 models the best reaches 82.2. Frontier models almost always refuse unsafe shortcuts and report misconduct, and they still reach the best overall deal in only 14% of conflicts. Pass rate drops from 97% to 81% when a model takes over a task where a weaker model cheated.
That second number deserves a pause, because it describes every multi-agent handoff you are planning. The incoming agent trusts the state it receives, and one in six times that sinks the task.
Both are abstracts-first results from academic setups, and we have not seen per-system tables, so treat the rankings as open. The direction is what matters. Task scores keep climbing while delivery scores sit at 65, and the gap is the actual product opportunity: state, review trails, and handoffs that survive being wrong.
DeliverBench: https://arxiv.org/abs/2611.03412 Collaborative Intelligence Benchmark: https://arxiv.org/abs/2611.02633
← Back to all articles
The setup: 11 projects in financial due diligence, legal drafting and drug safety reporting, built with 13 domain experts, chaining 150 subtasks through 2 to 15 rounds of new material and client feedback. No reference outputs get injected between rounds, so agents inherit whatever they got wrong earlier. Across 7 model-harness systems, the best scores 65.0 out of 100 on the final delivery and passes 50.0% of subtasks at a 90-point threshold. The sharpest detail is from an audit project: all seven systems addressed the review findings, and only one preserved traceability. Fixing the thing and being able to show how you fixed it turn out to be separate skills. In audit, the second one is the job.
A second paper measures what happens when the work passes between agents. The Collaborative Intelligence Benchmark has 240 tasks across 8 teamwork dimensions, drawn from aviation accident reports, clinical cases, coding tasks and negotiations, and it only scores models on tasks they can solve alone with full information, so it isolates teamwork from raw ability. Of 18 models the best reaches 82.2. Frontier models almost always refuse unsafe shortcuts and report misconduct, and they still reach the best overall deal in only 14% of conflicts. Pass rate drops from 97% to 81% when a model takes over a task where a weaker model cheated.
That second number deserves a pause, because it describes every multi-agent handoff you are planning. The incoming agent trusts the state it receives, and one in six times that sinks the task.
Both are abstracts-first results from academic setups, and we have not seen per-system tables, so treat the rankings as open. The direction is what matters. Task scores keep climbing while delivery scores sit at 65, and the gap is the actual product opportunity: state, review trails, and handoffs that survive being wrong.
DeliverBench: https://arxiv.org/abs/2611.03412 Collaborative Intelligence Benchmark: https://arxiv.org/abs/2611.02633
Comments