18 of 25 agent 'cognitive modules' did nothing, and a fix for reporting harnesses
Agent systems love impressive module names: global workspace, theory of mind, active inference. A pre-registered audit in Monday's arXiv batch checked whether those modules actually do anything. In "Real Computation or Merely Naming?" (arXiv 2610.41604), the authors found that in 18 of 25 named modules, the computed value never reaches any production output. They sort the dead wiring into three shapes: connected but never called, called but inert, and called but blind to content. Of 61 pre-registered criteria across 12 interventions, 53 were confirmed and 5 falsified. After repairs, planner task success went from 0.028 to 1.000, and state-controller error dropped 52%.
The lesson is uncomfortable and general. A module's name is a claim, not a mechanism. If you have bolted a memory, reflection or planning layer onto your agent, the fastest audit is to ablate it and see whether any output changes. This rhymes with a pattern seen all week: MCP servers that are installed but never used, eval calls that cannot be traced to a model, features that look present and are not.
A second paper takes the same worry up a level. "Towards Portable Specification of the Agent Harness" (arXiv 2610.40917) by Gertler, Novelli and Floridi splits the harness into eight functional components and proposes a versioned harness record: which model ran in which harness configuration. The motivation is that harness differences can produce several-fold gaps in reported scores for the same model, and harnesses now change between runs and even within a run when agents edit their own tools. The authors note that no enacted AI law requires such a record.
Put the two together and you get a simple standard: say exactly which harness you ran, and prove each part of it does something. Today's LongCat and GDPx releases, both reporting results across different harnesses, show why that is overdue.
Papers: https://arxiv.org/abs/2610.41604 https://arxiv.org/abs/2610.40917
← Back to all articles
The lesson is uncomfortable and general. A module's name is a claim, not a mechanism. If you have bolted a memory, reflection or planning layer onto your agent, the fastest audit is to ablate it and see whether any output changes. This rhymes with a pattern seen all week: MCP servers that are installed but never used, eval calls that cannot be traced to a model, features that look present and are not.
A second paper takes the same worry up a level. "Towards Portable Specification of the Agent Harness" (arXiv 2610.40917) by Gertler, Novelli and Floridi splits the harness into eight functional components and proposes a versioned harness record: which model ran in which harness configuration. The motivation is that harness differences can produce several-fold gaps in reported scores for the same model, and harnesses now change between runs and even within a run when agents edit their own tools. The authors note that no enacted AI law requires such a record.
Put the two together and you get a simple standard: say exactly which harness you ran, and prove each part of it does something. Today's LongCat and GDPx releases, both reporting results across different harnesses, show why that is overdue.
Papers: https://arxiv.org/abs/2610.41604 https://arxiv.org/abs/2610.40917
Comments