Summary: The American College of Radiology (ACR) has launched Assess-AI, the first AI quality registry to monitor and analyze the real-world performance of imaging AI algorithms, providing radiology facilities with performance insights, national benchmarks, and tools to ensure safe, effective, and transparent AI integration in clinical practice.
Key Takeaways
- Assess-AI is the first AI quality registry for radiology, providing facilities with tools to monitor and evaluate the performance of imaging AI algorithms in real-world clinical settings and benchmark them against national standards.
- Participating radiology facilities gain access to continuous AI oversight, including monitoring algorithm stability, analyzing result concordance with radiology reports, and generating actionable insights for AI governance teams.
- Assess-AI aims to ensure safe, effective, and transparent clinical AI adoption, helping radiologists manage growing imaging demands while maintaining high standards of patient care.
———————————————————————————————————————————————————————————————
The American College of Radiology (ACR) launched Assess-AI , the world’s first artificial intelligence (AI) quality registry designed to monitor ongoing performance of an array of imaging AI algorithms in real world clinical settings.
Monitoring Clinical AI Performance
Assess-AI is the newest ACR National Radiology Data Registry, capable of monitoring AI results and collecting an array of contextual information such as patient demographics, modality/exam meta data, and output from radiology reports. Its purpose is to provide radiology facilities with analytics on how clinical AI is operating in their own practices over time and compare their results against aggregated national performance benchmarks from other sites using identical or similar products.
“Users of AI technology in radiological care need to ensure that algorithms perform sufficiently in their local environment,” says Christoph Wald, MD, PhD, MBA, FACR, vice chair of the ACR Board of Chancellors and chair of the ACR Commission on Informatics. “Most legacy radiology systems were not built to support activities to ensure algorithms used in clinical settings are operating as expected. Assess-AI assists facilities with insights into the performance of deployed AI products. We look forward to growing this program with ACR Data Science Institute (DSI) facilitated oversight, offering a tangible real-world approach to address a challenge radiologists are increasingly faced with today.”
Assess-AI is the latest AI-related resource offered by ACR. ACR’s portfolio of AI products includes AI Central—a tool that helps facilities make informed algorithm purchasing decisions to care for their patients—and the ACR Recognized Center for Healthcare-AI (ARCH-AI) , the first national AI facility quality assurance program.
Inside Assess-Ai
Utilizing Assess-AI, clinical sites and AI developers will be able to obtain performance reports of deployed AI derived from local practices. This data will also help developers make improvements to future versions of algorithms.
Radiology facilities that participate in Assess-AI will have access to services including:
- Monitoring the stability of algorithm inputs over time, including imaging equipment, protocols and software version.
- Continuous oversight of AI result concordance/discordance with radiology reports.
- Aggregation of observations into reports and dashboards to share with a facility’s AI governance team, empowering further action as needed.
“Assess-AI will play a critical role in safely and effectively accelerating the clinical adoption of AI in radiology by ensuring AI products perform optimally in clinical settings so that radiologists can focus on what’s most important – providing high-quality care to their patients,” says Woojin Kim, MD, chief medical officer of the ACR DSI. “Expanding on its commitment to the radiology profession, the launch of Assess-AI is ACR’s latest step in helping empower radiologists to implement AI safely, effectively, and transparently.”
Shaping the Future of AI in Radiology
With the rising demand for imaging outpacing the supply of radiologists, AI is seen as an essential tool to help bridge the gap and enable radiologists to maintain high standards of care while meeting increasing demands. ACR, working with leadership at the U.S. FDA and Congress, is positioned to guide the future of AI in radiology.
