Artificial Intelligence1 min read

I had AI read my field

By · Published by Everything Blog

In short

Initiated from thousands of papers by Caltech's TAPIR group, the project processed a citation network of 158,059 papers divided into 32 distinct subfields. Full-text screening covered 50,269 accessible papers comprising 421 million words, while the automated system recomputed 659 core publications to verify their findings. Beyond tracking errors passed along through private email revisions, the project evaluated 155 open research questions and found that eight could be solved right now with existing data and codes. Integrity checks across the literature identified just 13 retractions among 184,733 papers with digital object identifiers.

Key points

  • Theoretical astrophysics research appears largely free from systemic fraud, displaying no…: Theoretical astrophysics research appears largely free from systemic fraud, displaying no evidence of paper-mill text or chatbot residue.
  • Uncorrected mistakes frequently cascade through scientific literature because formal revi…: Uncorrected mistakes frequently cascade through scientific literature because formal revisions are often limited to private email discussions.
  • Dozens of outstanding astronomical questions can be advanced immediately without waiting…: Dozens of outstanding astronomical questions can be advanced immediately without waiting for new physical telescopes or detectors.

Sources checked by our analysis pipeline: 1

I had AI read my entire field.

158,059 papers and 554,670 citations. Every paper with full text screened, 659 of the most important recomputed and checked, and every verified error followed through the papers that cite it. This is the map, and what it says about whether science corrects itself.

Errors that nobody fixed

I spent five years in this field doing my PhD at Caltech, and a few months of that went to errors in the literature that were already known. The equations my code was built on came from Bishop, Gómez, Lehner, Maharaj & Winicour, “High-powered gravitational news” (1997), the paper the subfield's characteristic codes descend from. People knew it had mistakes. The fixes were scattered through some 10,000 lines of intersecting email threads that contradicted each other, and the paper itself was never corrected. The best I could do was correct the record in my own papers: an appendix setting out the full set of equations, made consistent, and a parenthetical “with a sign error in s3 corrected” for anyone reading closely.

That is the limit of…

Artificial Intelligence