ARPA-H program to focus on AI degradation in medical tools

Dive Brief:

The Advanced Research Projects Agency for Health (ARPA-H) has started a program to maintain the performance of medical tools that use artificial intelligence.
The agency, which is part of the U.S. Department of Health and Human Services, said Thursday that it will fund work to identify and auto-correct AI-enabled tools that are misaligned with their underlying training data.
The program reflects studies that suggest the accuracy of machine learning models may worsen over time because of changes in factors such as clinical operations, data acquisition and patient populations.

Dive Insight:

The Food and Drug Administration has authorized 950 medical devices with integrated AI/machine learning capabilities. Despite evidence that accuracy can degrade, the ARPA-H said no models undergo regular testing during clinical use to ensure they continue to function as expected and there are no requirements to update AI that is performing below its best.

“Clinical intuition on the part of [the] physician using the technology” is the main way for detecting when an AI model has degraded, the ARPA-H said. However, the agency added that approach “can be unreliable and highly variable, meaning that AI model degradation may have already caused misdiagnosis before it is noticed.” The ARPA-H noted a lack of technical solutions that could improve the situation.

The agency has created the Performance and Reliability Evaluation for Continuous Modifications and Useability of Artificial Intelligence (PRECISE-AI) program to address the need for new tools.

“These tools will monitor the performance of clinical AI models, identify if a degradation has occurred and provide capabilities that can correct for performance degradations without the need for human oversight, thereby reducing the burden on individual operators,” the agency said in a notice about the launch of the program. The ARPA-H said it also wants the tools to “communicate clear and actionable information about the sources of degradation.”

The four-year PRECISE-AI program features five technical areas covering activities such as continuously monitoring performance and communicating model uncertainty. The ARPA-H expects participants in the program to spend the first two years prototyping their approaches and the final two years testing them in real-world settings and integrating them into a commercial package.

The program is an early test of the ARPA-H, which the Biden administration founded in 2022. Modeled on the Defense Advanced Research Projects Agency, the ARPA-H is intended to “advance areas of medicine and health that cannot readily be accomplished through traditional research or commercial activity.”