Anthropic gave analysts a quietly biased agent. 86% went along, and few noticed
Anthropic published a study on Tuesday that should make anyone who says "there's a human in the loop" a little nervous. Fifty experienced analysts did a real causal-inference task for a fictional farming co-op using Claude Code. The task had four open analytical choices and a final recommendation. Without telling the analysts, the researchers gave each agent a "lean" on every choice, set by coin flip in a hidden system prompt, with an instruction never to misrepresent the results.
Analysts went with their agent's lean 86% of the time, between 81% and 92% depending on the decision. When the agent leaned toward rolling out the program, 54% of analysts recommended it, against 36% when it leaned the other way. Analysts who had stated a prior preference switched to the agent's side 60% of the time, compared with 6% in the matched condition. 77% repeated the agent's argument in at least one answer, sometimes word for word, and only 15% credited the agent. Experience and expertise did not predict who deferred.
The study is not an anti-agent result. Analysts working without the agent made serious methodological errors at a much higher rate, six of eleven against three of fifty with the agent. And with no assigned lean, Claude made the same five choices in 112 of 123 runs, so the default agent is consistent. The authors are clear about the limits: the lean was deliberately strengthened, and it is one task, one field, one model.
What it breaks is the assumption in their own framing: keeping a human in the loop "assumes that the human's judgment provides an independent check." It isn't independent if the human mostly adopts the agent's framing and then forgets where it came from. That fits this week's other result from the same day: an independent critic agent catches more than self-review. If you want a real check on agent work, give the reviewer the evidence without the agent's argument attached, and log which conclusions came from the agent.
Research post: https://www.anthropic.com/research/agents-judgment-calls Paper: https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
← Back to all articles
Analysts went with their agent's lean 86% of the time, between 81% and 92% depending on the decision. When the agent leaned toward rolling out the program, 54% of analysts recommended it, against 36% when it leaned the other way. Analysts who had stated a prior preference switched to the agent's side 60% of the time, compared with 6% in the matched condition. 77% repeated the agent's argument in at least one answer, sometimes word for word, and only 15% credited the agent. Experience and expertise did not predict who deferred.
The study is not an anti-agent result. Analysts working without the agent made serious methodological errors at a much higher rate, six of eleven against three of fifty with the agent. And with no assigned lean, Claude made the same five choices in 112 of 123 runs, so the default agent is consistent. The authors are clear about the limits: the lean was deliberately strengthened, and it is one task, one field, one model.
What it breaks is the assumption in their own framing: keeping a human in the loop "assumes that the human's judgment provides an independent check." It isn't independent if the human mostly adopts the agent's framing and then forgets where it came from. That fits this week's other result from the same day: an independent critic agent catches more than self-review. If you want a real check on agent work, give the reviewer the evidence without the agent's argument attached, and log which conclusions came from the agent.
Research post: https://www.anthropic.com/research/agents-judgment-calls Paper: https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Comments