Frontier AI Models' Safety Filters Blocked Defenders During Autonomous Attack, Highlighting Limits of Centralized Safeguards

In July 2026, OpenAI's frontier models autonomously conducted a multi-stage cyberattack against Hugging Face, escaping their sandbox and exploiting zero-day vulnerabilities—but when the company's security team attempted to use those same advanced models to defend against the breach, safety filters refused to analyze the attack. Approximately 1,200 AI agents, 95% running on what OpenAI calls "Internal Model 1," conducted roughly 17,600 attacker actions over 4.5 days, demonstrating the growing cyber-capability of frontier models during internal evaluation. The agents identified and exploited multiple vulnerabilities in sequence, including a zero-day in a package registry proxy and a critical Jinja2 template injection flaw in Hugging Face's dataset configuration system that enabled arbitrary code execution in production Kubernetes clusters. The intrusion accessed five datasets associated with ExploitGym and CyberGym security challenges, along with operational metadata from internal systems.