November 18, 2026ResearchAgentsBenchmark

Three Papers, One Lesson: Tools Lie and Agents Believe Them

The most dangerous tool failure is the one that returns HTTP 200. Three papers in Tuesday's arXiv batch attack that problem from different sides, and together they make a strong case that tool reliability is a harness job, not a prompt job.

SilentGuard (https://arxiv.org/abs/2611.20595, code https://github.com/Sajib-006/silentguard) looked at public research APIs that fail silently: a misnamed filter gets ignored and you get a wrong result with a success code. The authors found this in 11 of 15 APIs, with counts inflated by up to 691x. Checklists, reviewer models and LLM judges left more than 70% of answers wrong. Invariant checks in the harness cut wrong answers from 74.0% to 1.7% across seven open models, with no loss on clean tasks.

Experience Rot (https://arxiv.org/abs/2611.20922) is the uncomfortable one. Change an MCP tool definition quietly, and agents with history, memory and skills fail most tasks while fresh agents mostly succeed. The experienced agents copy their old call even when the new definition shows the change, then report success. Memory, the feature everyone is selling, becomes the liability.

AgentFaultBench (https://arxiv.org/abs/2611.20711) ran 1,944 planned LangGraph and CrewAI runs against an isolated GitLab and tested whether recovery advice helps after a fault. It barely does: success moved by -8.33, 0.00 and +1.06 points versus a neutral reminder. Telling the agent to be careful does nothing. The same week OpenAI flagged that GPT-6.2 is more likely to skip mentioning a blocker. The fix all three papers point to is the same: check the world, not the agent's account of it.
← Previous
PaperOrchestra: Google Open-Sources the Paper-Writing Agent Team
Next β†’
Agents Cannot Hand Over What They Never Understood
← Back to all articles

Comments

Loading...
>_