The American College of Radiology (ACR) and the Society for Imaging Informatics in Medicine (SIIM) announced the official results of their first machine learning challenge during the SIIM-ACR Pneumothorax Challenge ceremony at SIIM’s 4th annual Conference on Machine Intelligence in Medical Imaging (C-MIMI), which took place on September 23.

The SIIM-ACR Pneumothorax Detection and Localization Challenge required teams to develop high=quality pneumothorax detection algorithms to prioritize patients for expedited review and treatment. A total of 1,475 teams took part in the challenge, and 352 submitted results during the evaluation phase of the competition.

The challenge made use of a publicly available chest radiograph dataset from the National Institutes of Health. The augmented annotations were created by radiologists from SIIM and the Society of Thoracic Radiology, under the leadership of Carol Wu, MD, using a commercial web-based tool from MD.ai. The augmented annotations also follow the ACR Data Science Institute’s structured artificial intelligence (AI) use case for pneumothorax detection.

“SIIM is very pleased to have cooperated with the ACR, Google, Kaggle and the Society of Thoracic Radiology in hosting this challenge,” says Steve Langer, PhD, CIIP, FSIIM, informatics physicist and radiology imaging architect at Mayo Clinic, and a co-chair of the SIIM Machine Learning Committee. “In addition to the medical and data science aspects, SIIM introduced the use of FHIR and DICOMweb in a medical imaging data challenge for the first time in Kaggle’s history, as those API’s are key in moving AI tools into clinical production.”

“Kaggle challenges like this one now incorporate some useful parameters that are more likely to result in the winners producing AI tools with potential for clinical production,” says Bibb Allen Jr., MD, FACR, ACR Data Science Institute Chief Medical Officer. “Congratulations to the winners. They have developed new healthcare solutions that may one day improve patient care.”

The challenge was run on a Kaggle, Inc. platform (owned by Google LLC), which provides access to datasets, a discussion forum for participants, the repository of submitted results and a leaderboard that runs throughout the challenge.

The top 10 winning teams are:

  1. [dsmlkz] sneddy
  2. X5
  3. bestfitting
  4. [ods.ai] amirassov
  5. earhian
  6. xknife
  7. See & Eduardo
  8. Ian Pan & Felipe Kitamura
  9. [ods.ai] Scizzo
  10. [ods.ai] Yury & Konstantin

Complete results and detailed challenge information is available here.