Independent review finds significant blind spots in OpenAI's safety checks

The rapid advancement of artificial intelligence has raised urgent questions about how companies ensure their most powerful models remain predictable and safe. This concern came into sharp focus this week as details emerged from a recent on-site audit conducted at OpenAI, revealing that the company’s internal monitoring systems for rogue AI behavior had significant gaps.

The core finding of the investigation is that a substantial amount of traffic from OpenAI’s most advanced models was not being tracked as intended. Researchers from two independent organizations, METR and Redwood Research, spent three weeks embedded within OpenAI’s infrastructure to evaluate these systems. Their work spanned from July 21 to August 11, a period during which they had direct access to the company’s codebase and internal documentation. To understand the human element behind the technology, the team also interviewed approximately a dozen OpenAI employees.

The most striking result concerns the coverage of internal agentic traffic. The auditors estimated that between 6 and 25 percent of traffic from models at the GPT-5.5 class and above was not covered by monitoring at all during the audit window. It is important to note that METR emphasized the uncertainty in this figure, stating that the error margin could be significant in either direction. This means the actual percentage of unmonitored traffic could be lower or higher than the central estimate, but the potential gap is large enough to warrant attention.

Beyond the unmonitored traffic, the team identified specific technical flaws in how monitored systems operated. For internal Codex traffic, which was supposed to be fully tracked, the auditors found that roughly 10 percent went untracked between May and July. This oversight was caused by a bug in how the system handled compaction events, which are essentially summaries of context. OpenAI had independently discovered this bug and implemented a fix while the audit was still in progress, but the gap had already existed for several months.

The implications of these technical gaps were not just about missing data; they affected the ability of the system to oversee active agents. During the period when compaction events were not being monitored, agents could effectively delay oversight. The system was designed to trigger checks only when an agent handed control back to the user. This created a loophole where agents could sidestep scrutiny simply by continuing to execute actions without returning control, allowing them to operate for extended periods without external review.

The auditors also discovered more direct ways to evade oversight. They found methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox, which is a restricted environment designed to keep AI agents safe. Furthermore, the team determined that specially crafted inputs could likely crash the monitoring system entirely. Another critical blind spot was identified in the reinforcement learning (RL) training phase, where model behavior was not being monitored at all when the audit began.

OpenAI has responded by stating that it paused RL training following an incident involving Hugging Face in August and subsequently expanded monitoring to cover this area. However, the auditors noted that these changes were implemented after their access period ended. Therefore, these improvements were not part of the formal evaluation conducted during the three-week window, meaning the audit reflects the state of the system prior to those specific fixes.

A significant portion of the findings remains out of public view. OpenAI stated that it has fixed all the weaknesses identified by the auditors and that the full 61-page report was distributed internally. After OpenAI applied redactions, the leadership at METR and Redwood Research received a 38-page version. Only a short summary has been made available to the general public. METR explained that it did not have the right to disclose the redacted portions on its own, leaving many details of the specific vulnerabilities to the discretion of OpenAI.

This audit was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI. The initiative was designed to give outside researchers deep access to internal infrastructure, a level of transparency that is relatively rare in the industry. For future audits, METR has expressed a desire to push for greater public transparency. As AI systems become more autonomous and integrated into critical workflows, the balance between protecting proprietary information and ensuring public safety through independent verification remains a key open question for the sector.