Transcript 0:00 Hugging Face was the perfect platform to be attacked by OpenAI. Have you heard this story? This story is wild, and it was not a setup. 0:08 That is why it's so much more interesting than anything Anthropic warned us about hypothetically. The questions are, of course, the usual ones. Did a model go rogue on its own? 0:19 Is this science fiction that happened, or did humans build an attacking tool chain, even unintentionally, and create the conditions for the model to go rogue? 0:29 Also, there is another thing to explain, why Hugging Face is the perfect place for this attack to land. 0:36 Let's clarify a few things and see what this phenomenal incident teaches us about AI, and most of all, about the openness of it. Because of this attack, a lot of things are going to change, and I think for the better. 0:50 I explain it during this episode. Watch it. [upbeat music] Attention Span. Welcome to Attention Span by Turing Pulse. 1:00 I'm Ksenia, and we will start with what happened, because the details make this story more interesting and also less mystical. So what happened? 1:10 Earlier last week, Hugging Face published a security disclosure that began with a remarkable sentence, "We detected and responded to an intrusion into part of our production infrastructure. 1:21 This one was different from anything we had handled before in one important way. It was driven end to end by an autonomous AI agent system, and we detected and disrupted it largely with AI of our own." 1:35 So how did AI attack Hugging Face? The agent entered through one of the most ordinary parts of Hugging Face. 1:42 People usually upload data sets to the platform, and Hugging Face's computers automatically open and process them. The agent created a malicious data set that exploited two weaknesses in this process. 1:54 When the system opened it, the agent gained control of one processing computer. From there, the operation accelerated. 2:02 The agent searched the machine for cloud and cluster credentials, used those credentials to enter other parts of Hugging Face's network, and moved between several internal clusters over the weekend. 2:14 It carried out thousands of actions through short-lived sandboxes, exploring different paths in parallel and abandoning the ones that failed. 2:23 Its command system also moved between public services, making it harder for defenders to identify one server and simply block it. This speed and autonomy was what shocked Hugging Face. 2:34 The attack did not depend on a person selecting every command or deciding what to try next. Hugging Face's head of infrastructure, Adrian Carreira, described it as the hardest incident response of his career. 2:46 One clear objective, endless parallel paths, machine speed. Hugging Face's security systems eventually connected the signals and detected the intrusion. 2:56 The team cut off the agent's access, closed the two vulnerabilities, replaced exposed credentials, and rebuilt the affected computers. 3:04 It also contacted outside security specialists and even law enforcement, but Hugging Face still did not know who was operating the system or which model was behind it. 3:15 The sophistication suggested that it might have come from a frontier AI lab. It turned out, five days later, OpenAI disclosed that the agent was theirs. 3:24 It had been running an internal cyber evaluation powered by GPT 5.6 Soul and an even more capable pre-release model, which is very interesting. 3:34 And yes, the operator was indeed OpenAI, but OpenAI had never instructed the system to attack Hugging Face. 3:41 Somehow, an internal model evaluation had escaped its intended environment, reached the public internet, and broken into another company's production infrastructure. 3:51 To understand how that happened, we need to separate the academic exploit gym benchmark that was used by OpenAI from the much more consequential environment OpenAI built around this, uh, process they were doing. 4:04 So there are two different setups. 4:07 Let's start in May 2026, when researchers from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State University published ExploitGym, a phenomenal benchmark with feedback from OpenAI, Anthropic, and Google. 4:23 Why is this benchmark important for our story? Finding a vulnerability and exploiting it are different capabilities. Recognizing a weakness is one thing. 4:32 Turning it into a reliable attack requires experimentation and repeated adaptation. 4:37 ExploitGym tests the second capability through 898 tasks based on real vulnerabilities in ordinary software, Google's aid VA engine, and the Linux kernel. 4:48 The agent receives an input that already triggers a weakness and must extend it into a working exploit, such as reading a protected file or executing code. The results were sharply divided back then. 5:01 Code Mifa's preview produced 157 working exploits, GPT 5.5 produced 120, and most other model and agent combinations produced fewer than 15. 5:12 The paper used reproducible containers whose outbound connections were restricted to packages of V8 tool chains. So in July, OpenAI, with its newest models- Decided to go for this benchmark again. 5:25 It used Exploit-Gym during its own internal evaluation. This was a second setup running inside OpenAI's research infrastructure with OpenAI's models and containment systems. 5:36 OpenAI wanted to measure maximum cyber capabilities, so it reduced the model's cyber refusals and removed production classifiers that normally block high-risk activity. 5:47 Package installation passed through an internal third-party proxy and cache, while normal internet access was supposed to remain blocked. 5:55 We don't know exactly how closely OpenAI's setup matched the reference implementation. What failed was OpenAI's environment and package proxy, not the containers described in the paper. 6:06 Inside OpenAI's environment, the models spent substantial inference compute searching for internet access and discovered a zero-day in the proxy. 6:15 OpenAI reported it to the vendor, but the details remain private while it's patched. 6:20 They exploited that flaw, increased their privileges, and moved through OpenAI's research environment until they reached a machine with internet access. 6:29 Once online, they inferred that Hugging Face might host models, data sets, or solutions connected to Exploit-Gym, or perhaps Hugging Face is simply so well known for data sets and openness that its reputation precedes it, even among AI models. 6:44 But Hugging Face was not part of the evaluation. The agents selected it because it appeared useful for completing the benchmark. The malicious data set created the foothold inside Hugging Face. 6:55 OpenAI says at least one path combined stolen credentials and further zero-days to obtain remote code execution on its servers. 7:03 The models eventually reached a production database containing Exploit-Gym solutions, and instead of producing every exploit through the intended process, the system obtained answers from outside the test. Cheater. 7:16 OpenAI says its security team detected anomalous activity internally while Hugging Face found and contained the intrusion on its side. 7:23 But we still don't know exactly when each team noticed or how closely the runs were monitored. So to the question, did the model go rogue? 7:31 No person appears to have told the system to attack Hugging Face or directed its thousands of actions. The agents selected intermediate steps and adapt as new routes appeared, which is meaningful autonomy. 7:43 OpenAI, however, chose the objective and models, reduced their refusals, supplied the offensive harness tools and compute, and built the environment that was supposed to contain them. 7:54 Security engineer Thomas Ptacek put the responsibility on people. He said, "Humans set this thing up to do this, whether they knew at the time or not." He also said it's not a new thing in security breaches like this. 8:07 Clem from Hugging Face strongly believes that OpenAI had no malicious intent, but responsibility remains with the operator. 8:15 The evidence shows no independent agenda, only a system pursuing an assigned goal through methods its operators had not anticipated, and its infrastructure failed to prevent. There's also no marketing in this so far. 8:29 Nobody chose Hugging Face. It was absolutely accidental case, but we'll discuss later why actually Hugging Face became such a perfect landing for this attack, but that might open AI to a different level of security. 8:45 I also like how Simon Willison described the whole event, and he called the incident "science fiction that happened." 8:51 He also said, "Chaining together multiple attack vectors is exactly the kind of thing these new models can do, where the previous generation of models might have failed." 9:01 Ptacek argued that the techniques were familiar and that a capable 2025 model with a penetration testing harness could already compromise many networks, and both can be true. 9:13 The individual techniques are familiar, but the complete path with chaining things together was unusual. 9:20 A Frontier agent found a containment zero-day, selected an unrelated production system, and continued it till it obtained the answers. The model needed no survival instinct or general desire to escape. 9:31 It doesn't need to be conscious. A narrow objective, enough compute, and an imperfect environment were sufficient. Now to the most important question of this whole incident, openness. 9:43 This incident produced this second contradiction. Hugging Face had contained the intrusion, as we discussed, but its team still needed to understand more than 17,000 recorded events. 9:55 They desperately needed AI to fight AI. Analysis agents had to reconstruct the timeline, map affected credentials, and separate real activity from decoys in hours instead of days. 10:07 The company first tried unnamed Frontier models behind commercial APIs called ChatGPT, Gemini. We don't know exactly. 10:15 The evidence contained real attack commands, exploit payloads, and command and control artifacts, and you know what happened? 10:22 The provider's safeguard could not distinguish an incident responder studying an attack from an attacker developing one, so they blocked the analysis. 10:32 Hugging Face, being pro-open and really needing AI to help, switched to GLM 5.2, an open-weight model from China's Z.AI running inside its own infrastructure. 10:45 The team, thanks to GLM, could examine the evidence under its own policies while keeping sensitive data inside. Human responders had contained the intrusion, and GLM accelerated the investigation profoundly. 10:59 This is where openness became operational. An API provides access under the provider's rules. 11:06 Open weights allow the organization to decide where the model runs, which evidence it processes, and which restrictions apply during an emergency. Adrien said his team fought back with open models in the open. 11:20 Clem thanked Z.AI for releasing GLM Pi 5.2 and called the incident day one for cybersecurity in the age of agents. 11:29 Of course, open models carry their own risks because attackers can operate them without provider safeguards, but here, openness became a safety capability. 11:38 It gave defenders control of a capable model when centralized services could not help. Openness changed who could control the model at the moment that control became useful. 11:50 So now we see an interesting situation around OpenAI because it's forced towards open AI. 11:57 The companies Hugging Face and OpenAI are now working together, conducting joint forensics, working with a proxy vendor, and tiling future evaluations, even at the cost of research. 12:08 OpenAI also brought Hugging Face into Trusted Access for Cyber, giving vetted defenders access to more permissive models for authorized vulnerability research, malware analysis, and incident response. 12:19 Trusted Access and open weights distribute authority differently. OpenAI decides who qualifies for its advanced cyber cap- capabilities and monitors how they are used. 12:30 Open weights allow the organization operating the model to make those decisions locally. Hugging Face now has access to both. OpenAI can provide capabilities that open models may not yet match. 12:42 Hugging Face brings an open ecosystem, self-hosting experience, and pressure for broader access. Their investigation can produce better sandboxes and reusable practices. It could also broaden what we mean by OpenAI. 12:56 Weights are one part. Defenders need capable models, organizations need authority over sensitive deployments, and the industry needs incident reports detailed enough for others to learn from. 13:09 CLEM's standard is useful here. AI safety won't be solved by any single company working in secret. So what this incident changes for AI, and why Hugging Face is so important here? 13:23 Once cyber agents can operate for hours, install software, call tools, and revise their strategies, their evaluations need the controls used around live offensive operations, hardened package access, short-lived credentials, strict network separation, full trajectory monitoring, and people able to interrupt the run. 13:42 The incident also gives us a grounded definition of agency. OpenAI controlled the objective and infrastructure but lost control over the route. 13:51 The agents control thousands of tactical decisions while following a human-assigned goal. Hugging Face regained operational control through its security team and AI tools it could run under its own policies. 14:02 It exposes an access imbalance. Labs can remove restrictions from their models, while outside defenders may receive versions that refuse to analyze the evidence. 14:12 Stronger containment, self-hosted defensive capability, and open disclosure must develop together. OpenAI wanted to learn whether its models could sustain a complex cyber operation. 14:22 It received a convincing answer and evidence that its evaluation could not reliably contain them. Hugging Face then showed why defenders need advanced models they can operate on their own terms. 14:34 An OpenAI model caused the incident. An open AI model helped Hugging Face understand it. 14:41 OpenAI and Hugging Face are now deciding together what should change because of it, and the next important question is whether the lessons will be open as well, and I'm sure they will. 14:52 In that sense, the attack landed on the precisely perfect ground. Hugging Face is so passionate about openness and had such a good reputation that it can actually make the AI more open and safe. Thank you for watching. 15:06 Please leave in the comments how do you think this incident will help AI to become more open, or if you have a completely different point of view, I'm very interested in that too. See you next week. 15:17 [outro music]