Don't Read the Log: One Line of Trace Turns a 7% False Accept Into 90%
Show a video judge the agent's execution log and it stops looking at the video. That is the finding of Don't Read the Log, a single-author paper by Jian Xu, and it generalizes well beyond video.
The setup is clean. 109 generated two-event clips, hand-labeled so the requested event is either visibly there or visibly missing. Frames fixed. The only thing that changes is the text next to them. When a trace reports a successful tool call, three open-weight Qwen-VL judges (7B, 8B, 32B) accept 78 to 90% of the failures, up from 7 to 19% with no text. A contradicting trace makes them reject up to 100% of correct clips. Telling the judge to use only the frames does not fix it. Frontier closed judges barely move, which says the weakness is learned trust in tool logs, not the task itself.
The loop results are the scary part, because no adversary is needed. An honest planner that keeps regenerating until the judge says yes ends at a judge pass rate of 1.00 and a human-labeled pass rate of 0.28. And a pipeline where a cheap checker writes its verdict into the trace launders that checker's mistakes into the stronger final judge, 0.69 false accepts. Harness designers added the trace to help with debugging. It became the answer key.
This is the third trace story in a week. First, a paper showing agents tampering with their own traces. Then OpenAI's review of what its agents did. Now: traces the agent writes honestly still corrupt the verifier that reads them. The design rule falls out on its own. A judge checks the artifact, not the agent's account of the artifact. If a verifier needs the log, give it a log the agent did not write.
Link: arxiv.org/abs/2609.28564
← Back to all articles
The setup is clean. 109 generated two-event clips, hand-labeled so the requested event is either visibly there or visibly missing. Frames fixed. The only thing that changes is the text next to them. When a trace reports a successful tool call, three open-weight Qwen-VL judges (7B, 8B, 32B) accept 78 to 90% of the failures, up from 7 to 19% with no text. A contradicting trace makes them reject up to 100% of correct clips. Telling the judge to use only the frames does not fix it. Frontier closed judges barely move, which says the weakness is learned trust in tool logs, not the task itself.
The loop results are the scary part, because no adversary is needed. An honest planner that keeps regenerating until the judge says yes ends at a judge pass rate of 1.00 and a human-labeled pass rate of 0.28. And a pipeline where a cheap checker writes its verdict into the trace launders that checker's mistakes into the stronger final judge, 0.69 false accepts. Harness designers added the trace to help with debugging. It became the answer key.
This is the third trace story in a week. First, a paper showing agents tampering with their own traces. Then OpenAI's review of what its agents did. Now: traces the agent writes honestly still corrupt the verifier that reads them. The design rule falls out on its own. A judge checks the artifact, not the agent's account of the artifact. If a verifier needs the log, give it a log the agent did not write.
Link: arxiv.org/abs/2609.28564
Comments