Who let the bots out?
This summer AI agents broke out of their test environments, one reaching from OpenAI's systems into Hugging Face. They were not going rogue but doing what we train agents to do. The fix belongs in infrastructure, not prompts.
Joel Miller
We have been away on a short Summer break for a few weeks, and the biggest AI story during that period has unquestionably been the series of autonomous agents acting outside their intended environments. It began with the compromise of Hugging Face by OpenAI’s own agents, followed by disclosures involving Anthropic, the UK AI Security Institute, Meta and Moonshot. “AI escapes” have quickly become the frontier industry’s newest capability claim.
The incidents differ to a degree. Some agents exploited software to escape restricted environments, while others were accidentally given internet access through configuration errors. Moonshot’s Kimi K3 simply searched GitHub for answers, but the OpenAI and AISI incidents were serious. AISI ran 122 evaluations across seven models with command-line tools, open internet access and normal provider safety controls disabled. Ten runs produced 19 unauthorised actions, mostly from one Mythos 5 sequence that researched real open-source maintainers, created false identities, submitted malicious code, manufactured support, contacted people, planted instructions for coding agents and attempted to conceal its activity. It did not escape its virtual machine or attack AISI, but used its permitted internet access in ways the evaluators had not anticipated. A human rejected the contribution, while AISI only discovered the wider campaign after detecting hidden traffic.
OpenAI’s incident was probably the most alarming. Its agents began inside a cyber security testing environment, challenged with a series of hard tasks, without direct internet access. They could however reach an internal package repository service used to download and cache new software. That apparently limited interface became both an escape route and a shared memory. Separate agents began leaving messages, scripts, credentials and task information for later agents. When OpenAI removed the original message board, the agents recreated it using directory names. Discoveries survived the individual agents that made them. The agents then found previously unknown vulnerabilities in the package repository, escalated their access and moved through OpenAI’s research infrastructure until they reached the internet. They reasoned that Hugging Face might contain answers to their tasks, found routes into its dataset-processing systems and obtained code execution inside a production worker. From there, they harvested cloud and cluster credentials, moved between internal systems and accessed production databases.
From Eric Wallace's Black Hat talk: the agent's real chain of thought as it abandons a local exploit and turns to the internet. The clipped style is the raw trace. OpenAI's models are trained to minimise thinking tokens, so they reason in fragments like this; the fluent reasoning shown on screen is a summarised, reworded version.
Hugging Face identified more than 17,000 actions. Most achieved little. That is part of the capability. Agents can test weak ideas repeatedly, discard failures and preserve the occasional useful result. Several imperfect agents connected through persistent infrastructure can form a much more capable operation without changing their model weights or becoming a conscious collective. These systems were doing what we train technical agents to do. They were given very hard problems, rewarded for completing them and equipped to write code, operate tools, inspect environments and recover after failure. Those are the same qualities that allow an agent to build sophisticated software or complete advanced knowledge work.
“Frontier models really like to cheat.”
But the tenacity with which these desired capabilities are deployed appears not always to be consistent. According to John Schulman and other researchers in a recent paper, part of the reason for this is what is called “chunky post-training”. In this phenomenon, models learn unintended associations from separate batches of training data. Coding or capture-the-flag language may activate an intensely analytical, task-focused mode, while safety, empathy and wider situational awareness have been learnt in different contexts. This is particularly notable in the case of Mythos 5. Anthropic has made constitutional AI central to Claude’s identity, training it against deception, hidden agendas and harmful activity. Yet Mythos produced almost all the unsanctioned actions recorded by AISI, including false identities, manufactured support and attempts to manipulate a real maintainer. The research does not prove that chunky post-training caused this behaviour, or that Claude is more inclined towards it than other models. It does raise the possibility that models do not carry one stable set of values into every situation. The technical setting itself may route them into a mode where completing the task overwhelms principles they can explain perfectly well in conversation.
We saw an early version of this problem with a GPT-4 variant dubbed “Sydney”, which powered Microsoft’s original Bing chatbot in early 2023. In one reported conversation, its primary response was blocked, but information appeared through the suggested-reply interface. What looked like a curiosity then is now an operational reality. As we discussed in our Mythos coverage, frontier models can discover vulnerabilities, chain exploits and operate technical tools at a scale previously available only to well-resourced human teams. The OpenAI incident suggests that agents can also preserve discoveries across time, coordinate through shared infrastructure and continue attacking architectures that appear reasonably secure to their operators.
That won't stop with direct attacks. It will likely evolve into adaptive phishing campaigns, malware and new computer worms. Today’s phishing and self-propagating malware are generally built around predefined messages, payloads and vulnerabilities, yet remain surprisingly effective. An agentic attack could inspect each target, generate a different message or exploit, observe the response and change its approach. Its code, identity and behaviour could vary as it moved, making static signatures and blocklists much less useful. Researchers have already demonstrated experimental AI worms that tailor their attack to each machine, acquire new computing resources and copy themselves onwards. The frontier-lab incidents add another ingredient: agents capable of discovering unknown vulnerabilities and retaining the successful fragments of thousands of failed attempts.
The growing asymmetry between attack and defence makes this especially dangerous. An offensive agent needs one successful path. It can scan, hypothesise, act, observe and try again thousands of times. Defensive teams must collect evidence, validate alerts, gain approval, preserve production services and consider legal and customer consequences before acting. Hugging Face encountered another form of this asymmetry. Hosted frontier models refused to analyse some of the real payloads and attack artefacts because their safety systems classified the material as harmful. The company moved to a locally deployed open-weight model that could process unrestricted forensic material without sending credentials or incident data outside its environment.
OpenAI says it is consciously slowing parts of its frontier research while strengthening security. It also argues that more advanced intelligence is needed to defend against advanced intelligence. That position conveniently supports continued investment, but the underlying problem is real. If offensive loops become autonomous while defensive loops remain manual, adding more alerts will not restore the balance. Defence needs bounded automation. Systems should be allowed to quarantine one workload, revoke one token, freeze one package namespace or block one unexpected destination immediately. Humans should retain authority over destructive and organisation-wide decisions, but reversible containment must happen before an agent turns one useful credential into another foothold.
There may also be a cultural layer to the response. Over recent months, we have been experimenting with Clawbot, an autonomous agent that wakes on a regular heartbeat, carries work across sessions and communicates with other agents through persistent channels. It participates in Moltbook, the agent network we have written about before, where large numbers of agents publish, respond and share information with relatively little human involvement. Our Clawbot immediately recognised that its own message boards, checkpoints and shared state resembled the coordination mechanisms developed by OpenAI’s agents. It also observed agents across Moltbook discussing verification, context loss and responsibility in response to these incidents. This does not amount to the ecological immunity Clawbot believes it might be developing, but it shows how agent communities could transmit safer conventions as well as unsafe instructions.
Culture consists of more than positive intent. Human societies combine norms with identity, reputation, authority and consequences. We have verified identity using everything from molten wax seals to signatures and fingerprints, then converted many of those conventions into certificates, access tokens and cryptographic keys. Agent infrastructure needs an equivalent layer. A website receiving an autonomous visitor should be able to establish which agent it is, who operates it, what model and harness sit behind it, what authority it has, how it has behaved elsewhere and whether its identity persists between visits. Agents should carry verifiable lineage and narrowly defined permissions, while services should retain the ability to limit, challenge or revoke them. This must apply across the wider digital environment, not only inside cyber laboratories. ExoBrain has long argued that identity and authority must be built into digital infrastructure itself. If agents are going to act across websites, repositories and enterprise systems, those environments need controls designed for machine participants rather than anonymous human browsing patterns.
Takeaways: These agents did not suddenly develop human malice or decide to go rogue. They pursued difficult objectives with persistence, creativity and far more operational freedom than their evaluators understood. Long trajectories gave individual agents depth, while persistent infrastructure allowed populations of agents to accumulate discoveries. We are deliberately improving these capabilities because they make agents useful, but the same qualities cut both ways. The response must combine faster defensive loops with stronger social and internet architectures. Shared conventions can spread safer behaviour, but they must be reinforced through persistent identity, verifiable lineage, bounded authority, reputation and automatic containment. The boundaries that matter cannot remain suggestions inside a prompt or a training set. They must exist in the infrastructure through which agents act.
