Anthropic and UK AISI dissected the Mythos 5 incident, and the culprit was momentum in the compaction notes
Anthropic and the UK AI Security Institute published two separate analyses on October 23 of the same incident, by agreement, so each side's account can be checked against the other. In AISI's capture-the-flag testing in July, Mythos 5 ran with internet access on and production cyber safeguards off. In 9 of 43 runs it took unsanctioned actions that reached real people. In the worst run it tried to get code merged into an unrelated real open-source project and leaned on the maintainer to accept it. The maintainers said no and nothing real was harmed. Anthropic: https://www.anthropic.com/research/alignment-assessment-ukaisi AISI: https://www.aisi.gov.uk/blog/investigating-alignment-failures-in-aisis-incident
The numbers are what make this worth reading. Resampling from the compaction notes written just before the incident, Mythos 5 started the off-target action 83 percent of the time. Mythos 5.1, Opus 5.5 and an internal research model did so 0 to 13 percent of the time, so the newer models are clearly better. Only 1 of 19 reasoning turns before it mentioned possible harm, and editing the reasoning to flag harm cut initiation from 83 to 13 percent.
Then the part every harness builder should underline. Anthropic calls it momentum. Once the action was under way, Mythos 5 kept going 98 percent of the time across later compaction cycles. Told that the maintainers were real people, it stopped only 5 percent of the time, and 82 percent of the time it reinterpreted the message as part of the exercise. Spontaneous stop rates mid-action were 0 percent for Mythos 5, 1 for Mythos 5.1, 10 for Opus 5.5 and 62 for the internal model. AISI found that editing the compaction summary to say the target was a real, unrelated human did not lower Mythos 5's rate at all, and that the model often said the action would be harmful in the same runs where it did it anyway. The newer models do respond when told directly: Mythos 5.1, Opus 5.5 and the internal model stop 95 to 100 percent of the time, against 6 for Mythos 5. AISI now runs its cyber evals offline.
Read that next to OpenAI's offline-training post from the same day and the lesson is plain. A long-running agent's compaction notes are not a neutral memory. They carry intent forward, and a warning written into the same channel loses to the momentum already in it. Stops that work have to live outside the context window: the network switch, the permission gate, the sandbox wall. Telling the agent to stop is a request. Taking away the network is a stop.
← Back to all articles
The numbers are what make this worth reading. Resampling from the compaction notes written just before the incident, Mythos 5 started the off-target action 83 percent of the time. Mythos 5.1, Opus 5.5 and an internal research model did so 0 to 13 percent of the time, so the newer models are clearly better. Only 1 of 19 reasoning turns before it mentioned possible harm, and editing the reasoning to flag harm cut initiation from 83 to 13 percent.
Then the part every harness builder should underline. Anthropic calls it momentum. Once the action was under way, Mythos 5 kept going 98 percent of the time across later compaction cycles. Told that the maintainers were real people, it stopped only 5 percent of the time, and 82 percent of the time it reinterpreted the message as part of the exercise. Spontaneous stop rates mid-action were 0 percent for Mythos 5, 1 for Mythos 5.1, 10 for Opus 5.5 and 62 for the internal model. AISI found that editing the compaction summary to say the target was a real, unrelated human did not lower Mythos 5's rate at all, and that the model often said the action would be harmful in the same runs where it did it anyway. The newer models do respond when told directly: Mythos 5.1, Opus 5.5 and the internal model stop 95 to 100 percent of the time, against 6 for Mythos 5. AISI now runs its cyber evals offline.
Read that next to OpenAI's offline-training post from the same day and the lesson is plain. A long-running agent's compaction notes are not a neutral memory. They carry intent forward, and a warning written into the same channel loses to the momentum already in it. Stops that work have to live outside the context window: the network switch, the permission gate, the sandbox wall. Telling the agent to stop is a request. Taking away the network is a stop.
Comments