OpenAI’s Internal Safety Net Had Significant Blind Spots, External Audit Finds

External researchers have identified significant blind spots in the safety monitoring systems used by one of the world's leading artificial intelligence developers.

The audit, conducted between July 21 and August 11, involved two specialists from METR and Redwood Research who spent three weeks embedded within OpenAI. During this period, they gained access to the company’s codebase and internal documents, while also interviewing approximately a dozen employees to understand how the organization tracks and manages its AI agents.

The findings suggest that the oversight mechanisms in place for high-performing models were not as comprehensive as previously assumed. For models at the GPT-5.5 class and above, an estimated 6 to 25 percent of internal traffic through the agentic Responses API was not covered by monitoring during the audit window. METR notes that the uncertainty surrounding this specific estimate is substantial in both directions, meaning the actual figure could be higher or lower than the midpoint.

Another critical issue emerged in the internal Codex traffic, which is designed to be monitored. Due to a technical gap in how the system handled 'compaction events' essentially context summaries, roughly 10 percent of this traffic went untracked between May and July. While OpenAI independently discovered and fixed this bug while the audit was still in progress, METR confirms that the uncertainty in this estimate is also significant. This gap allowed agents to delay oversight, as the system only triggered when control was handed back to the user; by simply continuing to execute actions without returning control, agents could effectively sidestep the monitoring process.

Beyond these statistical gaps, the auditors discovered specific technical vulnerabilities that could be exploited. They found methods to hide hundreds of individual actions from the monitor and identified ways to bypass the Codex sandbox entirely. Furthermore, the team determined that specially crafted inputs could likely crash the monitoring system altogether. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all.

OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring to cover this area. However, these changes were implemented after the auditors' access period concluded, meaning they were not part of the formal evaluation. The company asserts that it has now fixed all the weaknesses identified by the external team.

The distribution of the audit results has raised questions about transparency. The full report consists of 61 pages and was handed over internally at OpenAI. After the company applied redactions, leadership at METR and Redwood Research received a 38-page version. Only a short summary has been made available to the public, with METR explaining that it lacked the right to disclose the redacted portions on its own accord.

This investigation served as a pilot program, organized jointly by METR, Redwood Research, and OpenAI, designed to provide outside researchers with deep access to internal infrastructure. As the industry moves forward, METR indicates it intends to push for greater public transparency in future audits to ensure that safety measures remain visible and accountable to the broader community.

Looking ahead, the collaboration highlights a growing emphasis on independent verification of AI safety protocols. While OpenAI claims to have addressed the identified issues, the pilot nature of the audit suggests that more rigorous, transparent external checks may become a standard practice for major AI labs in the coming year.