Artificial Intelligence1 min read

Formalizing CPython

By · Published by Everything Blog

In short

Software researchers utilized the Opus 5.5 artificial intelligence model to translate the reference implementation of Python directly into machine-checked proofs within the Lean language. Because the popular language lacks a precise written specification, edge-case bugs frequently evade detection in its underlying C code. This rigorous verification effort forces exact definitions for every execution path and has already exposed hundreds of hidden flaws, including severe memory-corruption bugs.

Key points

  • Converting CPython to the Lean language eliminates functional ambiguity by strictly defin…: Converting CPython to the Lean language eliminates functional ambiguity by strictly defining arithmetic overflow, error handling, and other hidden edge cases.
  • Candidate functions undergo testing across tens of thousands of inputs prior to proof gen…: Candidate functions undergo testing across tens of thousands of inputs prior to proof generation to keep the AI from creating trivial proofs.
  • Mathematical proofs can diverge from real-world execution if initial model parameters err…: Mathematical proofs can diverge from real-world execution if initial model parameters erroneously assume operations like memory allocation cannot fail.

Sources checked by our analysis pipeline: 1

AI agents wrote 613,318 lines of Lean that copy Python's internals line by line, tested every piece against the real Python, had a computer check every proof, and turned up 418 bug reports in Python along the way.

TL;DR

- We pointed Opus 5.5 at CPython, the program that runs almost all Python code, and had it rewrite its functions in Lean, a language for writing machine-checked proofs.

- Each rewrite copies the original line by line and is tested against real Python on tens of thousands of inputs before anyone tries to prove anything about it.

- A function only counts once a second agent, which didn't write it, checks the work and agrees it holds up. 4,071 functions have passed that check so far.

- The result is 613,318 lines of Lean across 1,539 files, with 18,898 theorems and lemmas, all checked by Lean's proof checker on every build.

- Along the way the proofs turned up 418 bug reports. Most are small, but about 55 can crash the interpreter or corrupt memory from ordinary Python code.

Why…

Artificial Intelligence