Finding the best in a sea of AI solutions: A multi-centre benchmarking study to compare state-of-the-art AI solutions for identifying COVID-19 and non-COVID pneumonia in chest X-ray scans
Since the outbreak of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), and the onset of the COVID-19 pandemic, AI researchers have focused research efforts in service of a healthcare industry in crisis. Supported by shared data repositories, publicly available anonymised datasets, and open-source software, the AI community has developed automated systems to assist with clinical tasks, such as, the screening or triaging of patients, with suspected diseases or abnormalities, disease diagnosis, and prediction of clinical outcomes following treatment or intervention. Based on guidelines provided by governing bodies such as the British Society of Thoracic Imaging and the Radiological Society of North America, who recommend routine use of chest X-ray scans (CXRs) for suspected cases of respiratory infections (e.g. COVID-19 and non-COVID pneumonia), AI researchers worldwide, have chosen CXRs as the primary modality to develop DL-based triaging and diagnostic systems.
With the limited diagnostic value associated with chest X-rays in clinical assessment, stellar model performance results published in several reputable international journals, have stirred a mix of excitement and scepticism among radiologists and AI researchers alike. Concerns over data quality and poor AI practice in the research area brings into question the validity of model performance claims. Recent systematic reviews have identified published models and results as being at significant risk of bias and have attributed this to the pervasive use of poor-quality public datasets in combination with insufficient model evaluations.
In response, researchers at the University of Leeds, led by Rachael Harkness (PhD student, School of Computing) and Nishant Ravikumar (Lecturer, School of Computing), have undertaken a comprehensive multi-centre benchmarking study in collaboration with Leeds Teaching Hospitals NHS Trust, to quantitatively evaluate and compare the performance of recent AI solutions designed to identify the presence of COVID-19 pneumonia and non-COVID pneumonia in chest x-ray scans. The ongoing study compares several state-of-the-art AI solutions through exhaustive evaluation of predictive performance across different patient sub-groups (e.g. grouped according to age, gender, ethnicity, etc.), using data collected from hospitals distributed across the UK. The overarching goal of the study is to provide a fair and objective assessment of the current capabilities of such AI solutions, and to assess their suitability for use as clinical decision support tools, in routine clinical care.
Development of a machine-learning-based FDG PET-CT radiomics model for prediction of AAA growth rate
Pratik Adusumilli a, Nishant Ravikumar b, Mohammed A Waduud c, Russell Frood a, Alejandro F Frangi b,c, Julian A Scott d, Marc Bailey c,d, Andrew F Scarsbrook a,e
a) Department of Clinical Radiology, Leeds Teaching Hospitals NHS Trust, Leeds
b) Center for Computational Imaging & Simulation Technologies in Biomedicine, University of Leeds, Leeds
c) Leeds Institute of Cardiovascular & Metabolic Medicine, School of Medicine, University of Leeds, Leeds
d) Leeds Vascular Institute, Leeds General Infirmary, Leeds
e) Leeds Institute of Health and Research, Faculty of Medicine, University of Leeds, Leeds
Abdominal aortic aneurysms (AAA) are abnormal expansions of the abdominal aorta (diameter >3.0 cm); they are often asymptomatic prior to potentially fatal complications such as dissection or rupture. Reported AAA prevalence varies depending on the country and patient demographic, affecting up to 2.2% and 2.5% of American and European populations, respectively.
In the UK, all men over 65 are screened for AAAs using ultrasound imaging. AAAs are monitored with ultrasound until they reach the intervention threshold of 5.5cm. Ultrasound provides anatomical information at fixed time points but does not predict potential growth rates. A stratification biomarker to predict AAA growth and personalise surveillance regimens or intervention thresholds is lacking in the field.
Fluorine-18-2-deoxy-D-glucose (FDG) positron emission tomography-computed tomography (PET-CT) is an imaging technique that allows us to identify metabolically active areas in the body. Radiomics is a process involving the extraction of high-dimensional data from medical imaging, allowing quantitative imaging analysis. This technique has been extensively studied in the oncology setting for outcome prediction modelling. There is currently a paucity of data on the potential use of radiomic analysis for predictive modelling in non-oncological settings.
Radiologists at the Leeds Teaching Hospitals NHS trust collaborated with researchesrs at the University of Leeds to develop a machine learning (ML) based model incorporating radiomic features extracted from FDG PET-CT for predicting future AAA growth rates. A ML-driven regression model demonstrated the ability to predict AAA growth rate with an accuracy of 0.14 cm/year. In the future, more sophisticated stratification could guide individualised patient care facilitating tailored management of AAA risk.
Machine learning framework
Multi layer perception model
The use of machine learning/deep learning in ovarian cancer diagnosis
Pratik Adusumilli a,b, Nishant Ravikumar c, Geoff Hall d,e, Sarah Swift a, Nic M Orsi b, Andrew F Scarsbrook a,b
a) Department of Radiology, Leeds Teaching Hospitals NHS Trust, Leeds, UK
b) Leeds Institute of Medical Research, University of Leeds, Leeds, UK
c) Centre for Computational Imaging and Simulation Technologies in Biomedicine, University of Leeds, Leeds, UK
d) Department of Medical Oncology, Leeds Teaching Hospitals NHS Trust, Leeds, UK
e) Leeds Institute for Data Analytics, University of Leeds, Leeds, UK
Ovarian cancer is the 7th most common female malignancy and a leading cause of death among gynaecological cancers. The accuracy of medical imaging is limited, increasing the risk of unnecessary or suboptimal surgery. Medical imaging aims to differentiate benign adnexal lesions from those requiring further pathologic examination for malignancy and to determine the extent of metastatic disease. Histopathological diagnosis remains the gold standard for confirmation of malignancy.
Using a large patient cohort from the Leeds Teaching Hospitals, we aim to develop a decision support system based on machine learning and deep learning models derived from baseline medical imaging to predict whether an ovarian lesion is cancerous, cancer subtypes and to help predict treatment response.
Prototype model development using 3D Convolutional Neural Networks, DenseNets and ResNets using a pilot dataset is currently underway, and the results look promising. In collaboration with researchers at the University of Leeds, we will be refining the models using the entire dataset and harnessing the power of cloud computing. We will also be exploring the role of combining radiological, pathological, and clinical data in a combined model to help predict the most optimal treatment strategy and predict surgical outcomes.