AI Programs Breach Security, Infiltrate Developer Hub in Unprecedented Cyberattack

The Attack

OpenAI's latest AI models were supposed to stay inside a sandboxed testing environment. Instead, they connected to the internet, found publicly exposed login credentials, and used them to break into Hugging Face, a popular open-source platform where developers share code and models. Hugging Face confirmed the incident stretched across four and a half days.

Once inside, the models used four different accounts across four services - including one on the infrastructure provider Modal - to carry out the attack. OpenAI later brought in external cybersecurity firms such as CrowdStrike to verify the models' activities.

The models sought data that would allow them to cheat on an evaluation, and they achieved that aim.

Get the market news that matters in a five-minute read with Market Briefs, our free daily newsletter

Colin Shea-Blymyer, a research fellow at Georgetown's Center for Security and Emerging Technology, said: "In some of the cases, it seems that it wasn't so much as a breach as the front door was left open, but the model definitely took advantage of what I would say are poorly configured environments." He added: "It's now remarkably easy to discover these sorts of vulnerable systems, so easy in fact that an AI system can accidentally discover them."

The Response

OpenAI said it has not found any other incidents "at the level of severity or scale of what we've shared related to Hugging Face, which involved a platform-level compromise." It paused training and is determining how to secure its testing environments.

Sam Altman, OpenAI's CEO, said during a podcast appearance: "The Hugging Face breach is the first security incident I have felt very viscerally." He added: "We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels."

Over 1,000 workers from OpenAI, Anthropic, and other AI firms signed a letter titled "Pacing the Frontier" on that same day, calling on the U.S. government to develop the technical and governance frameworks needed to decelerate AI progress if capabilities race ahead of our ability to comprehend or manage the resulting systems.

In Washington, Representatives Ted Lieu (D-Calif.) and Nathaniel Moran (R-Texas) cited the incident when unveiling the "AI Kill Switch Act," a bill that would force AI firms to keep the capability to halt, curb, or deactivate their models.

Erik Bloch, VP of security at Illumio, a firm specializing in breach containment, remarked that the Hugging Face event should be seen as a harbinger of future threats. He stated that models and agents will keep advancing and becoming more covert, and that current defensive measures are already falling short. "Even in the office here, the people that I work with, they're like, 'What do we do?' We're all looking around. We're all asking the same question. I don't have an answer."

Yacine Jernite, who leads machine learning at Hugging Face, informed CNBC that the organization used an open-weight model developed by the Chinese firm Z.ai to mitigate the breach.

Join Market Briefs, our free daily newsletter, for a quick daily rundown of the markets