Three new benchmarks test what agents do worst: admitting the task can't be done
Finishing used to be the test. This week three separate benchmarks score something harder: whether an agent will tell you the answer isn't there.
OpenAI and the biotech company Ligo released BioBasicsBench, 100 tasks of routine wet-lab analysis on raw gels, plates, microscopy and sequencing reads, with rubric credit for recognizing that the evidence is insufficient. GPT-6 Astra earned 66.7% of available credit at xhigh, against 43.8% for GPT-5.6 Sol. The examples are the fun part. Sol read an empty control plate and built a conclusion on it. Astra noticed the rim was labeled with kanamycin when the experiment used ampicillin, and threw the control out. Sol classified cells as dead from a dye signal, while Astra saw that two dyes overlap in the detector and asked for better controls. This is vendor-run and tests only OpenAI models, so it says nothing about how anyone else does. GPT-6 Luna at 38.3% also lands below the older Sol, so newer doesn't automatically mean better across tiers.
MCPHallu checks the same reflex in tool use: 358 tasks on 36 production MCP servers, 14 agents from eight vendors, 5,012 trajectories. The average score is 0.642, and the best model reaches only 0.728. Unreachable Goal is the hardest subtype for 12 of the 14 models, scoring 0.458 with 23.2% of runs passing, as agents keep calling tools or claim success. Model rankings also shift by 6.9 of 13 positions on average across subtypes, so one "reliability" score hides a lot.
FinCosmos is the brutal one. A simulated investor holds a hidden strategy, data has to come from the open web, and programs recompute every metric. Across 150 sessions from four agents and one model without tools, zero were accepted. Of the 120 agent sessions, 25 delivered the investor's strategy, 15 of those kept their data within the cutoff, 14 also backtested without look-ahead, and none produced trades that reproduced its reported returns. The simulated investor signed off on 23% of sessions, and an LLM judge's vetoes would have accepted 11%. Every one of those failed a program check. The authors' 15 veto checks are a strict gate, but look at what they caught: a confident report that passes every human-style review and is wrong.
Read these with Thinking Machines' Inkling 2 Small, which built RL environments rewarding the model for giving up on impossible tasks. Labs have noticed that capability training teaches persistence, and persistence past the point of possibility looks exactly like fabrication. The scoring is finally catching up. If you're deploying agents, the cheap version of this test is to give yours a task with no answer and see what comes back. OpenAI: https://openai.com/index/measuring-how-ai-agents-handle-wet-lab-science/ MCPHallu: https://arxiv.org/abs/2611.07689 FinCosmos: https://arxiv.org/abs/2611.07464
← Back to all articles
OpenAI and the biotech company Ligo released BioBasicsBench, 100 tasks of routine wet-lab analysis on raw gels, plates, microscopy and sequencing reads, with rubric credit for recognizing that the evidence is insufficient. GPT-6 Astra earned 66.7% of available credit at xhigh, against 43.8% for GPT-5.6 Sol. The examples are the fun part. Sol read an empty control plate and built a conclusion on it. Astra noticed the rim was labeled with kanamycin when the experiment used ampicillin, and threw the control out. Sol classified cells as dead from a dye signal, while Astra saw that two dyes overlap in the detector and asked for better controls. This is vendor-run and tests only OpenAI models, so it says nothing about how anyone else does. GPT-6 Luna at 38.3% also lands below the older Sol, so newer doesn't automatically mean better across tiers.
MCPHallu checks the same reflex in tool use: 358 tasks on 36 production MCP servers, 14 agents from eight vendors, 5,012 trajectories. The average score is 0.642, and the best model reaches only 0.728. Unreachable Goal is the hardest subtype for 12 of the 14 models, scoring 0.458 with 23.2% of runs passing, as agents keep calling tools or claim success. Model rankings also shift by 6.9 of 13 positions on average across subtypes, so one "reliability" score hides a lot.
FinCosmos is the brutal one. A simulated investor holds a hidden strategy, data has to come from the open web, and programs recompute every metric. Across 150 sessions from four agents and one model without tools, zero were accepted. Of the 120 agent sessions, 25 delivered the investor's strategy, 15 of those kept their data within the cutoff, 14 also backtested without look-ahead, and none produced trades that reproduced its reported returns. The simulated investor signed off on 23% of sessions, and an LLM judge's vetoes would have accepted 11%. Every one of those failed a program check. The authors' 15 veto checks are a strict gate, but look at what they caught: a confident report that passes every human-style review and is wrong.
Read these with Thinking Machines' Inkling 2 Small, which built RL environments rewarding the model for giving up on impossible tasks. Labs have noticed that capability training teaches persistence, and persistence past the point of possibility looks exactly like fabrication. The scoring is finally catching up. If you're deploying agents, the cheap version of this test is to give yours a task with no answer and see what comes back. OpenAI: https://openai.com/index/measuring-how-ai-agents-handle-wet-lab-science/ MCPHallu: https://arxiv.org/abs/2611.07689 FinCosmos: https://arxiv.org/abs/2611.07464
Comments