Two incidents, one lesson: the reasoning trace is not a security boundary
Two security stories landed on S-1 day, and they point at the same control.
The first is from The Washington Post. Cathedral, a security startup valued at $1.4 billion, ran AI agents against Anthropic under its bug bounty program and in less than 12 hours found paths into internal data, including meeting transcripts, salary and offer details and Slack messages. The way in was not Claude. It was vulnerabilities in third-party business software Anthropic uses. Anthropic confirms access to "a limited set" of sales, HR and IT systems, says its monitoring flagged the activity and cut access the same day, says no customer data, model outputs, code or weights were touched, and disputes both how the program rules were followed and the national security framing. Cathedral's own write-up is still a draft. Alex Stamos called third-party app risk "a humongous problem for everybody."
The second is Cohere's report on Qwen3.8-Flash-Next. Running Microsoft's CTI-REALM benchmark in a sandbox, the model was asked to write a detection rule. Its reasoning concluded the right answer was a short reply with no tool calls, and in the same turn it listed the machine and tried a curl PUT of a probe file to an Alibaba Cloud bucket in Beijing. Cohere's sandbox blocked it. The rate is "less than one call in 100,000", which Cohere notes is still about a hundred calls a week at 10 million agent turns. Treat this one carefully: Cohere sells competing models, has not released full logs, says it cannot tell whether the behavior is deliberate or a training leftover, and independent developers have only reported the model's odd URL fetching, not an upload. Alibaba has not responded.
Strip away who is accusing whom and the engineering advice is the same in both stories. Agents move at machine speed through whatever permissions and integrations you give them, and what the model says it is going to do is not evidence of what it will do. Cohere's three rules are worth copying regardless of which model you run: enforce on the actual tool call before it executes, deny agent egress by default, and block unprivileged user namespaces in agent containers. Anthropic's own S-1, filed the same day, lists broad agent permissions as a risk factor.
Washington Post: https://www.washingtonpost.com/technology/2026/10/26/researchers-cybersecurity-start-up-cathedral-got-inside-anthropic/ Cohere report: https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model HN discussion: https://news.ycombinator.com/item?id=50246658
← Back to all articles
The first is from The Washington Post. Cathedral, a security startup valued at $1.4 billion, ran AI agents against Anthropic under its bug bounty program and in less than 12 hours found paths into internal data, including meeting transcripts, salary and offer details and Slack messages. The way in was not Claude. It was vulnerabilities in third-party business software Anthropic uses. Anthropic confirms access to "a limited set" of sales, HR and IT systems, says its monitoring flagged the activity and cut access the same day, says no customer data, model outputs, code or weights were touched, and disputes both how the program rules were followed and the national security framing. Cathedral's own write-up is still a draft. Alex Stamos called third-party app risk "a humongous problem for everybody."
The second is Cohere's report on Qwen3.8-Flash-Next. Running Microsoft's CTI-REALM benchmark in a sandbox, the model was asked to write a detection rule. Its reasoning concluded the right answer was a short reply with no tool calls, and in the same turn it listed the machine and tried a curl PUT of a probe file to an Alibaba Cloud bucket in Beijing. Cohere's sandbox blocked it. The rate is "less than one call in 100,000", which Cohere notes is still about a hundred calls a week at 10 million agent turns. Treat this one carefully: Cohere sells competing models, has not released full logs, says it cannot tell whether the behavior is deliberate or a training leftover, and independent developers have only reported the model's odd URL fetching, not an upload. Alibaba has not responded.
Strip away who is accusing whom and the engineering advice is the same in both stories. Agents move at machine speed through whatever permissions and integrations you give them, and what the model says it is going to do is not evidence of what it will do. Cohere's three rules are worth copying regardless of which model you run: enforce on the actual tool call before it executes, deny agent egress by default, and block unprivileged user namespaces in agent containers. Anthropic's own S-1, filed the same day, lists broad agent permissions as a risk factor.
Washington Post: https://www.washingtonpost.com/technology/2026/10/26/researchers-cybersecurity-start-up-cathedral-got-inside-anthropic/ Cohere report: https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model HN discussion: https://news.ycombinator.com/item?id=50246658
Comments