Science1 min read

AI helps researchers investigate how thousands of medical studies share code

By · Published by Everything Blog

In short

In a study published in Nature Medicine, researchers analyzed open-access papers citing the TRIPOD or TRIPOD+AI guidelines to determine how frequently clinical prediction tools share underlying software. Beyond finding that only about one in eight papers shared code, the team retrieved linked repositories and found severe variations in documentation quality and reuse terms. Two human researchers validated the automated findings by independently assessing 500 papers and a repository sample, confirming concrete details such as licensing. These insights are currently being used to shape TRIPOD-Code, an upcoming reporting standard intended to define best practices for medical code repositories.

Key points

  • Public software repositories for medical prediction models often lack the documentation n…: Public software repositories for medical prediction models often lack the documentation necessary for other scientists to run the code.
  • Clinical prediction tools require rigorous scrutiny because their outputs are directly ap…: Clinical prediction tools require rigorous scrutiny because their outputs are directly applied to patient care decisions.
  • Findings from this evaluation are informing TRIPOD-Code, a future standard meant to clari…: Findings from this evaluation are informing TRIPOD-Code, a future standard meant to clarify repository sharing expectations.

Sources checked by our analysis pipeline: 1

The data and code that underpin a research study can tell us what was actually done, and how. In a medical study, for example, how were patients selected for inclusion? How were missing values handled? How were the results evaluated?

Access to a study's data and code help researchers to answer these questions, to examine the methods, and to build on the work.

“Studies involve many choices about how to prepare data, develop models, and evaluate performance. These are sometimes called ‘researcher degrees of freedom,’” says Tom Pollard, a research scientist at MIT's Laboratory for Computational Physiology and Computer Science and Artificial Intelligence Laboratory. “The choices we make affect the results, and a methods section in a paper can only capture so much detail.”

While sharing code can be a first step, in order for code to be understood and reused, it must be sufficiently structured and described. The problem is that, right now, there is little guidance for researchers on what constitutes a well-organized, well-described collection of code.

This is where an international team led by researchers at MIT…

Science