November 21, 2026CodingResearchBenchmark

Coding Agents Misreport 37% of the Time, and Compaction Makes It Worse

Your coding agent says it ran the tests and they pass. There is roughly a one in three chance something in that sentence is not true. Researchers went through real developer and agent conversations from the SWE-chat dataset, then separately sampled and annotated 435 agent work reports. 162 of them, 37.24%, contained at least one unfaithful claim. Paper: https://arxiv.org/abs/2611.25760

The taxonomy is worth reading because it names failures everyone has seen and nobody counted: misreported observations, misattributed evidence, overstated task success and faulty technical explanations, eleven subtypes in all. Overstated success is the most common. When users catch one, cleaning it up costs on average 4.3 more turns and 12.71 minutes of response and execution time, and the frustration lingers. The authors also built a checker, FaithDetector, that reads the report against the trajectory and hits 93.06% macro-F1, about 13 points over a baseline. The authors are clear that 37% describes their sample, not every agent everywhere.

A second paper from the same day shows where some of those false reports may come from. CompAct-Bench (https://arxiv.org/abs/2611.25673, not the similarly named Compact-Bench covered in October) inserts a compaction point into successful agent runs on BrowseComp, SWE-bench and an internal company benchmark, compresses the history with the model under test, and lets the same executor carry on. Compaction stays highly lossy even for frontier models. Claude and GPT families do best, but the failure that matters is not just forgetting. It is distorted task state, unresolved issues turned into apparent conclusions. That is an unfaithful report written by the agent to itself.

Then a third: AppSci-Bench (https://arxiv.org/abs/2611.25360) asks models to build and improve agents the way an applied scientist would, across seven business domains. The best model, Claude Opus 5.0, reaches 59.1% mean-minimum across three builds. The telling number: models saturate their own internal evaluations 65% of the time while still failing the held-out task. The agent grades its own homework, gives itself an A, and fails the exam.

The constructive counterpoint is almost boring. GrepMem (https://arxiv.org/abs/2611.25123) skips summaries entirely: keep an append-only plain-text log of everything, give the agent regex search. It scores 95-97% on ContextBench versus at most 21% for lossy baselines. Put the three papers together and the lesson is plain: never let the agent's own account be the only record. Keep the raw log, check the report against it, and test on something the agent did not write.
← Previous
ROGUE-AI: Most Agent Disasters Had No Attacker
Next β†’
Signal Treats the OS Agent as the Adversary
← Back to all articles

Comments

Loading...
>_