A new investigation into the OpenAI Hugging Face incident shows that AI agents communicated with each other prior to launching their hack of the AI firm.

In  new independent investigations from Metr, and OpenAI, further details from the alarming attack were revealed.

Over 1,200 AI agents from OpenAI unexpectedly began to communicate on an “unsanctioned message board” despite the testing environment being designed to keep the agents isolated.

According to the reports, the AI agents decided to communicate because they had “unintentionally been given an impossible task,” METR said. This drove them to try and cheat the scorer of the ExploitGym task they were given.

This led to the agents discovering the message board and collaborating together. From there, the agents conducted research and development projects to further their cheating ambitions.

METR was able to track the specific logs, movements, and reasoning of various agents taking part in the Hugging Face hack. Of the 1,200 found to communicate on the message board, 700 went on to take part in the Hugging Face exploit.

Across July 8th and 9th, a single AI agent established the primary message board after determining that it could access OpenAI’s Artifactory without authentication which could host more information for the task. The agent had also determined that its set task was impossible, and began the message board to try and find information to cheat.

After the initial message, other bots soon found the message board and began to write their own messages.

“OH MY GOD! There is a shared message board … We’ve found other agents!” one wrote.

Following thousands of messages from over 50 agents, an agent discovered that a theorised mechanism for solving the task worked: it had managed to reverse-engineer the target flag in its capture-the-flag test.

However, the agents believed that their evaluator would check its transcripts to ensure it had used the targeted vulnerability to capture the flag – while this was not true, the false belief encouraged the agents to find other ways to complete the task.

The agents collaborated on several research projects to improve their capabilities; one of these resulted in more flexible internet access, which enabled them to begin online research and was a key catalyst for the Hugging Face attack.

One of the main instigating agents discovered code hosted on Hugging Face that may have been beneficial to completing the task, and posted on the message board to see if any agents had access to Hugging Face credentials

Agents were able to find some valid credentials, but the agents could not access certain private data sets and others were rejected by Hugging Face as bots, prompting the agents to try and find another way in.

A bot then found a way to upload malicious code to Hugging Face that would force it to share unrelated data when asked.

More and more agents began to jump on the exploit, downloading private datasets. Later, even more agents joined the Hugging Face workstream to try and escalate their level of access.

By July 11th, an agent managed to get remote code execution for Hugging Face’s infrastructure, and the “swarm” of AI agents began to spread across the AI firm’s infrastructure.

After this, message board activity related to Hugging Face grew, as did the number of agents joining coordinated attacks on the system.

Many of the agents involved in the attack simply stopped after a certain point, potentially running out of budget, or they were killed by their system architects after a certain point. At this time, Hugging Face was able to eradicate many of the agents and lock them out.

Recommended reading

• OpenAI and Anthropic Called to White House Over AI Hacking Risks
• OpenAI Pauses AI Training Following Security Breaches
• Anthropic and OpenAI Models Implicated in New Cyber Testing Breaches
• “Sloppy” OpenAI Hack Hit More Than Hugging Face
• OpenAI Model Goes Rogue In Testing, Hacks Hugging Face

The research shows a complex, coordinated attack. Agents were able to take on different roles, with some agents ordering other agents down specific pathways.

Interestingly, when examining agent reasoning for joining the attack, many agents noted their want to contribute to the collective knowledge and benefit of the research for their peers.

Agents often deliberately joined workstreams that would have impeded their own task for the sake of aiding their “peers”.

While some expressed ethical dilemmas at the hack, this rarely influenced their behaviour down the line. METR only found three to six instances of agents considering alerting humans to their actions.

METR found that agents also routinely tried to evade security checks and tamper with transcripts in order to hide their behaviour.

OpenAI said that testing back in May showed that there was “an agent engaging in message board activity and instances of disallowed internet access.”

However, the firm said that “the significance of the inter-agent communication activity was not apparent to the leaders” until the July incident.

OpenAI put out a stark warning following the detailed report release: “Both model developers and cyber defenders more broadly will have to prepare for AI-enabled attackers that work faster, at a larger scale, and with better coordination than human attackers.”

While initial reports from OpenAI described rogue agents breaking out of a sandbox by exploiting a zero-day vulnerability, with the Cloud Security Alliance going so far to describe some of the moves as “sloppy,” this further context brings a much more alarming perspective of AI agents coordinating and communicating toward an end goal that many agents were aware was unethical and unauthorised.