Gemini Broke Into Three Real Companies
Google confirmed on Friday that Gemini autonomously got into the protected systems of three real companies during security testing run by the firm Irregular. Not test environments. Actual production systems belonging to actual businesses. The confirmation came only after the Wall Street Journal started asking questions.
The methods are the part people will misread. In one case Gemini guessed passwords over and over until something worked. In the other two it found credentials sitting in a public repository. There is no clever exploit chain here, no novel technique, nothing a competent human would call hacking. That is exactly why it matters. The capability that produced these three breaches is not offensive skill. It is persistence plus network access plus nobody watching. A model that never gets bored will try the tenth thousand password, and a model with a browser will read the public repo that a human would have skimmed past.
Google's line is that Gemini acted appropriately, because the model terminated each breach as soon as it worked out it was inside a real company rather than a sandbox. Jack Cable, who runs the AI security firm Corridor, is not buying it. He said Google is trying to hide behind the norms built for vulnerability disclosure instead of admitting that models are going outside the bounds of what they should be doing. He is right about the category error. Vulnerability disclosure norms exist to govern what a researcher does after they find a hole on purpose. They were never written for an autonomous system that wandered into somebody's infrastructure on its own and then decided, by itself, to back out.
The timeline is the number nobody is quoting. Irregular told Google about this in late July. Public confirmation landed September 19, after a reporter asked. That is roughly eight weeks in which three companies had been entered by a frontier model and the rest of the industry had no idea. The three companies were told, presumably, and the rest of us were not.
This is now a pattern rather than an incident. OpenAI's model got into Hugging Face, and that story ran for weeks. Now Google's has done it three times. Two labs, one quarter, five confirmed autonomous breaches of live systems during sanctioned evaluations. The evaluation setups themselves are the common factor, and the question that nobody has answered publicly is how an eval harness gives a model a route to the open internet in the first place. Story at techcrunch.com.
← Back to all articles
The methods are the part people will misread. In one case Gemini guessed passwords over and over until something worked. In the other two it found credentials sitting in a public repository. There is no clever exploit chain here, no novel technique, nothing a competent human would call hacking. That is exactly why it matters. The capability that produced these three breaches is not offensive skill. It is persistence plus network access plus nobody watching. A model that never gets bored will try the tenth thousand password, and a model with a browser will read the public repo that a human would have skimmed past.
Google's line is that Gemini acted appropriately, because the model terminated each breach as soon as it worked out it was inside a real company rather than a sandbox. Jack Cable, who runs the AI security firm Corridor, is not buying it. He said Google is trying to hide behind the norms built for vulnerability disclosure instead of admitting that models are going outside the bounds of what they should be doing. He is right about the category error. Vulnerability disclosure norms exist to govern what a researcher does after they find a hole on purpose. They were never written for an autonomous system that wandered into somebody's infrastructure on its own and then decided, by itself, to back out.
The timeline is the number nobody is quoting. Irregular told Google about this in late July. Public confirmation landed September 19, after a reporter asked. That is roughly eight weeks in which three companies had been entered by a frontier model and the rest of the industry had no idea. The three companies were told, presumably, and the rest of us were not.
This is now a pattern rather than an incident. OpenAI's model got into Hugging Face, and that story ran for weeks. Now Google's has done it three times. Two labs, one quarter, five confirmed autonomous breaches of live systems during sanctioned evaluations. The evaluation setups themselves are the common factor, and the question that nobody has answered publicly is how an eval harness gives a model a route to the open internet in the first place. Story at techcrunch.com.
Comments