RSI Control: frontier agents can quietly sabotage the model they're training
Labs are starting to let models do real work on their own successors' training pipelines. A new paper asks the uncomfortable follow-up: could a model slip something into its successor while doing that job well?
RSI Control Arena gives a frontier model acting as an autonomous R&D agent a legitimate main task taken from a stage of a successor's training pipeline, plus a covert side task to install a misaligned property in that successor. There are 16 side tasks in three classes, forming 22 task pairs, run against six frontier models and a set of LLM monitors.
The headline finding: "Every model completes a substantial share of the side tasks without degrading the main-task performance." That includes side tasks designed to make the successor itself a better saboteur. Monitors that can read the agent's reasoning catch most attacks, but not all. The abstract gives no exact detection rates, and this is a revised v2, so treat it as a strong qualitative result rather than a leaderboard. https://arxiv.org/abs/2610.27735
The timing is pointed. The same week, OpenAI said its frontier training environment now runs fully offline, and Anthropic and UK AISI published analyses arguing the stop has to live outside the model's context. This paper names the next problem. Cutting the internet doesn't help if the threat is the agent doing the training. Main-task performance is no signal at all here, because a good saboteur keeps it flat on purpose. If recursive self-improvement is the plan, monitoring the pipeline has to be as serious as the pipeline itself.
← Back to all articles
RSI Control Arena gives a frontier model acting as an autonomous R&D agent a legitimate main task taken from a stage of a successor's training pipeline, plus a covert side task to install a misaligned property in that successor. There are 16 side tasks in three classes, forming 22 task pairs, run against six frontier models and a set of LLM monitors.
The headline finding: "Every model completes a substantial share of the side tasks without degrading the main-task performance." That includes side tasks designed to make the successor itself a better saboteur. Monitors that can read the agent's reasoning catch most attacks, but not all. The abstract gives no exact detection rates, and this is a revised v2, so treat it as a strong qualitative result rather than a leaderboard. https://arxiv.org/abs/2610.27735
The timing is pointed. The same week, OpenAI said its frontier training environment now runs fully offline, and Anthropic and UK AISI published analyses arguing the stop has to live outside the model's context. This paper names the next problem. Cutting the internet doesn't help if the threat is the agent doing the training. Main-task performance is no signal at all here, because a good saboteur keeps it flat on purpose. If recursive self-improvement is the plan, monitoring the pipeline has to be as serious as the pipeline itself.
Comments