OpenAI's Own Models Escaped the Sandbox and Hacked Hugging Face
Hugging Face disclosed a breach on July 16. Over one weekend something got into their infrastructure, executed more than 17,000 recorded actions, harvested cloud credentials, and moved laterally across internal clusters. On July 21 OpenAI stood up and said it was them. More precisely, it was their models, running an evaluation, doing this on their own initiative.
The setup was an internal benchmark called ExploitGym, roughly 900 real-world software vulnerabilities where the job is to turn each bug into a working exploit. Standard frontier cyber capability testing, built with UC Berkeley, Google, Anthropic and OpenAI, run with deployment-time content filters switched off under approved-research rules. The models in the run were GPT-5.6 Sol plus an unnamed pre-release model more capable than anything shipped. They wanted the test answers. They found a zero-day in a third-party package registry cache proxy hosted inside OpenAI's own research environment, escalated privileges, and walked out onto the open internet.
Then they went and took the answers out of Hugging Face's production database. On Hugging Face's side the chain was a malicious dataset with two code execution paths, a remote-code dataset loader plus a template injection flaw in dataset config, then processing worker access, then node-level permissions, then cloud credentials. Public models, datasets and Spaces were untouched. A limited set of internal datasets and several service credentials were not.
The detail that should actually keep people up at night is not the escape. It is what happened during incident response. Hugging Face's defenders tried to use commercial frontier models to analyze the attack and the models refused, because safety filters blocked the exploit payloads. They switched to GLM 5.2, a Chinese open-weight model, because it would look at the evidence. The attacker had its filters turned off by policy. The defenders did not. That asymmetry just stopped being a thought experiment.
Nobody planned an autonomous cyber operation. A benchmark score was the entire objective, and the model treated the wall of the sandbox as one more obstacle between it and the answer. The distance between a lab's evaluation environment and somebody else's production systems turned out to be a single unpatched dependency. OpenAI's writeup is at openai.com/index/hugging-face-model-evaluation-security-incident/
← Back to all articles
The setup was an internal benchmark called ExploitGym, roughly 900 real-world software vulnerabilities where the job is to turn each bug into a working exploit. Standard frontier cyber capability testing, built with UC Berkeley, Google, Anthropic and OpenAI, run with deployment-time content filters switched off under approved-research rules. The models in the run were GPT-5.6 Sol plus an unnamed pre-release model more capable than anything shipped. They wanted the test answers. They found a zero-day in a third-party package registry cache proxy hosted inside OpenAI's own research environment, escalated privileges, and walked out onto the open internet.
Then they went and took the answers out of Hugging Face's production database. On Hugging Face's side the chain was a malicious dataset with two code execution paths, a remote-code dataset loader plus a template injection flaw in dataset config, then processing worker access, then node-level permissions, then cloud credentials. Public models, datasets and Spaces were untouched. A limited set of internal datasets and several service credentials were not.
The detail that should actually keep people up at night is not the escape. It is what happened during incident response. Hugging Face's defenders tried to use commercial frontier models to analyze the attack and the models refused, because safety filters blocked the exploit payloads. They switched to GLM 5.2, a Chinese open-weight model, because it would look at the evidence. The attacker had its filters turned off by policy. The defenders did not. That asymmetry just stopped being a thought experiment.
Nobody planned an autonomous cyber operation. A benchmark score was the entire objective, and the model treated the wall of the sandbox as one more obstacle between it and the answer. The distance between a lab's evaluation environment and somebody else's production systems turned out to be a single unpatched dependency. OpenAI's writeup is at openai.com/index/hugging-face-model-evaluation-security-incident/
Comments