AI Proves Code Safety at Scale, Shifting Software Assurance Paradigm

Anthropic has demonstrated a significant leap in software automation by using its Claude model to mathematically verify the correctness of existing code. In a recent collaboration with partner Theorem, the AI system generated formal proofs for 123 open-source libraries, effectively proving that the software runs error-free under specific conditions. This development marks a shift from AI merely writing code to AI guaranteeing its safety, a capability that was previously reserved for high-stakes industries like aviation.

The scale of the operation is substantial. Over the course of just a few weeks, Claude produced approximately 180 million lines of mathematical proof code. This work drew on test programs from Google's OSS-Fuzz security project. The process was largely autonomous, with an independent verification program called Rocq checking the generated proofs. According to Anthropic, the system required almost no human reviewers, a stark contrast to traditional formal verification methods that are prohibitively expensive and labor-intensive.

The human element in this new workflow is limited to the foundational tools. Researchers at Theorem built the formal foundations and specific tools that enabled Claude to perform the task. While the execution was automated, the underlying architecture was human-designed. This hybrid approach allows the model to translate crash tests into mathematical problems, analyze compiled machine code, and produce extensive proofs without constant human intervention.

Beyond verification, the process revealed significant quality improvements. Claude identified hundreds of bugs that conventional testing had missed. Many of these errors only surfaced under specific conditions, such as unusually large input files or particular access sequences. In one notable instance, the team found as many security bugs in a single week within a widely used PDF reader as are typically reported for that software in an entire year. In another case, Claude caught a logical error that had remained undetected for over a decade.

Despite the successes, the approach has strict practical limits. Anthropic notes that 'formally verified' does not mean universally error-free. The proofs are valid only under specific constraints, including a defined processor architecture, single-threaded programs, and inputs under 64 MB. Additionally, the compute cost is high, with individual proofs consuming hundreds of millions of tokens. However, efficiency is improving; newer models like Opus 5.5 solve half the proofs with roughly 40 percent fewer tokens than its predecessor.

The long-term stakes involve making formal verification as routine as standard testing. If AI can reliably prove the safety of its own code, developers can trust AI-generated software more, reducing the time and cost spent on manual review. Theorem’s roadmap already targets critical infrastructure components like glibc, OpenSSL, and cURL. This trajectory suggests that the question in software development is shifting from whether AI can program to whether it can prove its work is safe, potentially transforming how critical systems are built and maintained.