AI Formal Verification Redefines Software Safety Standards

Anthropic has demonstrated that its Claude model can automatically generate mathematical proofs to verify the safety of existing software code. This development marks a significant shift in how developers approach software reliability, moving beyond traditional testing toward rigorous mathematical validation.

The initiative, conducted by Anthropic's Frontier Red Team in partnership with Theorem, focused on verifying 123 open-source libraries from Google's OSS-Fuzz security project. Over a few weeks, the system produced approximately 180 million lines of proof code. This volume of automated verification work would have previously required vast amounts of human specialist labor, highlighting the potential for efficiency in the software development lifecycle.

Historically, formal verification has been prohibitively expensive and time-consuming. For example, verifying the seL4 operating system kernel required 20 person-years of specialist effort. Consequently, this method was largely reserved for critical applications such as Airbus avionics. By automating the process, Anthropic aims to make formal verification as routine as standard software testing, thereby lowering the barrier to entry for high-stakes code assurance.

The process does not involve writing new software. Instead, Claude translates crash tests into mathematical problems and analyzes compiled machine code to generate proofs. An independent verification program named Rocq automatically checks these proofs. Anthropic describes the workflow as fully autonomous, meaning it requires minimal human review, although the underlying formal foundations and tools were developed by humans at Theorem.

A critical benefit of this approach is the discovery of latent bugs that conventional testing often misses. During the verification process, Claude identified hundreds of bugs and was authorized to fix them. Many of these defects only manifested under specific conditions, such as unusually large input files or particular access sequences. In one instance, the team found as many security vulnerabilities in a widely used PDF reader in a single week as are typically reported for that software in an entire year. Additionally, the system detected a logical error that had remained undetected for over a decade.

Despite these successes, Anthropic emphasizes that formal verification does not guarantee that software is entirely error-free. The proofs are valid only under strict constraints, including a specific processor architecture, single-threaded execution, and input sizes under 64 MB. In real-world scenarios, these libraries often run on diverse hardware and configurations, which limits the universal applicability of the proofs.

The computational cost of generating these proofs is substantial, with individual tasks consuming hundreds of millions of tokens. However, the project suggests that reusing proofs and leveraging more efficient models can reduce costs. For instance, the newer Opus 5.5 model solved half of the proofs using 40 percent fewer tokens than its predecessor, Opus 5. This efficiency gain supports the viability of scaling the technology.

The broader implication of this work is a shift in the question from whether AI can program to whether it can prove that its code is safe. If AI can guarantee correctness under defined conditions, developers can place greater trust in automated software generation. Anthropic expects this method to become standard practice, with Theorem's roadmap targeting critical infrastructure components like glibc, OpenSSL, and cURL for verification in the coming years.