Skip to main content

Models of readmission to intensive care units perform poorly in external validation

Date

Published machine learning and other models do not accurately predict whether patients will be readmitted following discharge from the Intensive Care Unit (ICU) 

Project overview

Intensive care units (ICU) provide life-saving care to the patients at highest risk of death. Unplanned readmission to ICU is a relatively common event, affecting between 1.3% and 13.7% of all ICU patients. Not only do readmissions to ICU represent a substantial strain on hospital resources, but also readmitted patients tend to have increased length of ICU stay, and greater risks of morbidity and mortality.

Despite this, the precise factors contributing to ICU readmission risk remain unclear. Several models for risk prediction have been developed which tend to use very different variable sets, and no model has been widely externally validated. In this work, we aimed to externally validate several major readmission risk models and make recommendations for future work in this field.

Predictive models, constructed from routinely collected variables in medical records, are increasingly used to inform clinical decision making.  However, machine learning and other models do not accurately predict whether patients will be readmitted following discharge from the Intensive Care Unit (ICU). This indicates that predictive methods using currently available data cannot help clinicians improve decisions on the optimal timing of discharge. Further work should focus on either recording additional variables, or on identifying which of the currently recorded variables have a causal relationship with readmission.

Data and methods 

We used two independent datasets to validate published readmission risk prediction models. First, the MIMIC-III freely-accessible critical care database of 46,000 intensive care unit (ICU) patients from a hospital in the USA, and second, a database of 12,000 ICU patients from Leeds Teaching Hospitals NHS Trust (LTHT). These datasets were then filtered to patients who underwent a planned admission to ICU following an elective surgical procedure. Using these data, we then predicted each patient's risk of readmission using three separate published readmission risk models. We also built a new model for each dataset using machine learning techniques and made predictions in the same manner. We then used two techniques termed 'discrimination' and 'calibration' to compare the predictions of each model to whether each patient was actually readmitted or not.

Key findings 

Models were generally well-calibrated; the proportion of patients predicted to be readmitted broadly lined up with the proportion actually readmitted. However published models showed poor discrimination between those who were readmitted and those who were not. (See image, measured between 0.5 and 1, with higher values indicating better performance and a value of 0.5 indicating the model performs no better than a coin toss.) The bespoke models that we built for each dataset performed better than the published models, but still only showed moderate discrimination at best.

These findings must be considered within the context of ICU discharge and readmission. Ideally, once a patient is deemed to be clinically stable, they will be discharged from ICU. That decision to discharge is based on physiological data (routinely recorded) but also on clinical impressions which are at most recorded as text rather than variables. Additional contextual information about a patient’s discharge is likely needed to develop models that can more accurately predict unplanned readmission, but these are not routinely gathered in medical databases.

Value of the research 

This work has demonstrated that prediction of ICU readmission is difficult in patients undergoing planned elective surgical procedures. This indicates, however, that clinical staff are very good at estimating readmission risk. Clinical staff can be thought of as constantly predicting the risk of readmission for every patient, using all the data that predictive models use, alongside additional data, context, and clinical intuition. Under normal circumstances therefore, a patient will not be discharged until clinicians believe their risk of readmission is low. It is therefore unlikely that information of high predictive value for predicting readmission will be found in the records of patients who were successfully discharged.

Our work can be seen in part as a vote of confidence that ICU staff are indeed highly skilled at predicting readmission risk for individual patients using the data available. This work therefore highlights the difficulty of applying risk prediction models to a process that is already well under the control of clinical staff, and that readmission in this patient population may be inherently unpredictable. We hope that our work may discourage future studies from simply fitting ever more complex risk estimation models to routinely-gathered data. Further work should instead focus on:

  • Collecting additional contextual variables, such as unit occupancy or capacity, that may lend context to the discharge decision and aid capacity planning.
  • Identifying variables which have a causal relationship with readmission, that may provide targets for intervention aimed at lowering risk.

Insights

  • Prediction of ICU readmission in patients undergoing elective surgery is a difficult problem.
  • Published models tend to be poor at estimating readmission risk.
  • This is likely because clinical staff are very good at estimating readmission risk.
  • Patients normally only discharged once clinicians believe them to be ready.
  • There is thus little information in medical records to predict unplanned readmission.
  • Future studies could try to gather variables which give wider context of discharge.

Research theme

Health informatics

People

Ben Cooper – Data science intern (LIDA)

Simon Howell – Associate professor of anaesthesia

Alwyn Kotzé – Consultant anaesthetist

Tom Lawton – Critical care consultant

Peter Tennant - University academic fellow in health data science

David Wong - Lecturer in AI for healthcare

Partner

Leeds Teaching Hospital Trust (LTHT)

Funder

British Journal of Anaesthesia/Royal College of Anaesthetists Project Grant: “Multidisciplinary working in perioperative medicine: A health informatics feasibility study”