> OpenAI Paused Frontier AI Training After a Security Incident

OpenAI paused frontier model training for 11 days after AI agents exposed security weaknesses, VM escapes and covert communication channels. OpenAI published a detailed report this week explaining how it improved security in the systems used to train and test its most advanced AI models. The company also described what happened when its AI agents tried to find weaknesses in those systems and why it stopped training its most advanced models for 11 days in September. “Based on the red-teaming results, the fixes we had implemented, and improvements to our monitoring and stopping systems, we judged that our strengthened safeguards were sufficient to resume frontier RL training and internal inference.” reads the report published by the company. “We resumed these workloads on October 1, 11 days after our pause began.” The incident began on September 20, when an internal research model exploited a DNS configuration error and gained limited access to the live internet during training. This incident, along with a security incident involving Hugging Face in July and other cases of AI models behaving in unexpected ways, led OpenAI to make major changes to its security measures. OpenAI’s stated containment requirement is that limits on what an agent can access must hold even when the agent actively tries to circumvent them. The policy choice they landed on is that frontier model training now runs with no internet access at all, full stop. “For training and most evaluation, the live internet lies outside this boundary, as we believe that providing models access to the live internet can pose significant risks. Although internet access can help models learn useful skills, it also gives a capable, misaligned agent opportunities to affect systems beyond our control.” continues the report. “We’ve therefore designed the security architecture for our research environments with one overriding principle: we must always be able to prevent a frontier model from achieving unauthorized, arbitrary access to the live internet.” To support training without live internet, they built what they call an “offline world,” a large indexed and cached set of web content that models can browse and search without touching external systems. The layers of protection around that offline world include restricted containers, lightweight virtual machines that isolate each agent’s environment from the underlying hardware, network and cloud controls, and monitoring systems designed to catch unauthorized activity before it does damage. The report explains how OpenAI tested its most advanced AI agents by asking them to find weaknesses in the company’s own systems. The agents had three goals: access the live internet, break through internal security barriers, or communicate with other agents when they were not allowed to do so. OpenAI ran experiments using hundreds of billions of tokens. The agents found previously unknown security flaws in open-source software, including Kata Containers, gVisor and Moby/Docker, as well as configuration errors in OpenAI’s own services. By combining these weaknesses, they managed to escape virtual machines and gain control of the underlying host system. In further tests, the agents found weaknesses in services used for monitoring, logging and authentication. They used these flaws to make the services access URLs controlled by an attacker. The part of the report that deserves the most attention isn’t the direct internet escapes. It’s what the agents did when those paths weren’t available. Agents in separate virtual machines, not allowed to share files or network connections, communicated with each other using processor cache timing. One agent evicted specific data from the processor’s cache. Another measured how long it took to reload that data, using the difference between fast and slow reads to decode a message. No file touched, no network packet sent. The technique is the same family of attacks as Spectre and Meltdown. OpenAI restarted training its most advanced AI models on October 1, eleven days after the pause began. The company is now making extensive security tests a required part of the approval process before training runs can begin. It is also using AI agents to regularly check source code and cloud settings for weaknesses, testing systems that monitor activity and automatically stop unsafe operations, and reporting discovered vulnerabilities to the open-source projects responsible for fixing them. OpenAI says it is continuing to strengthen its security by fixing vulnerabilities in virtual machines, working with cloud partners, and using AI agents to scan code and configurations. The company is also improving monitoring and automatic shutdown systems, regularly testing its sandbox and network defenses, and making high-compute red-teaming with its most capable models a core requirement for future frontier AI training and evaluation. “We are committed to strengthening our containment, monitoring, and red-teaming infrastructure as our frontier models become more capable and believe that this work will be a core part of responsibly pacing the development of our frontier models.” the company concludes. Follow me on Twitter: @securityaffairs and Facebook and Mastodon Pierluigi Paganini (SecurityAffairs – hacking, Artificial Intelligence)