### Data collection and statistical analysis

Medical records of patients with COPD as the primary diagnosis were queried from the METEOR (Methodist Environment for Translational Enhancement and Outcomes Research) clinical data repository from all eight Houston Methodist Hospitals.^{17}. Because the Rothman Index score is not stored as raw data, the score was manually pulled from the monitoring panel. The Rothman Index (PeraHealth, Charlotte NC) is a regularly updated integrated health score that uses a range of twenty-six physiological measures, including laboratory test results, vital signs, and nursing assessments.^{18}. It is an automated, proprietary third-party algorithm integrated into commercial electronic medical record systems. The Rothman index has been shown in multiple trials to be a valuable metric for predicting mortality in hospitalized patients. However, it has not been used as a component to predict readmission. Supplementary Figure S3 shows the twenty-six variables and the corresponding Rothman index for one patient in the monitoring panel.

The following available parameters that were considered to be likely to affect readmission included demographic data, index admission type, first-day data on severity of illness, comorbidities, laboratory data, admission medications, admission Rothman index, procedures, and Main complaint for admission.^{19.20}. A univariate analysis was used for each parameter to assess its association with subsequent readmission.^{twenty-one}. A logistic regression Wald test was performed on the parameters and the parameters were ranked by p-value. Highly predictive parameters of subsequent readmission were identified from the training sample and subsequently validated against a second population of COPD patients.

All aspects of this study were conducted in accordance with relevant guidelines and regulations, including the preservation of patient privacy. The Houston Methodist Hospital Institutional Review Board approved this study, and this study is one of several process quality improvement projects commissioned by Houston Methodist Hospital management to improve the quality of patient care. Retroactive accruals of patient data were used in the development of the model and a waiver was granted by the Houston Methodist IRB for patient informed consent.

### Model building and training.

The task of predicting early patient readmissions was formulated as a binary classification task: readmission yes/no to any of the Houston Methodist system hospitals within 30 days. Two mathematical modeling approaches were built: logistic regression^{22.23} and artificial neural network^{24.25}.

### Training and testing the neural network

The artificial neural network has the advantage of modeling complex nonlinear functions. A classic neural network includes an input layer, some hidden layers, and an output layer. Each layer contains some nodes or neurons. Each node in the hidden layer is a mathematical function to transfer information from input to output. The connections between two nodes of two adjacent layers are called weights. The logistic regression model is considered as a simple form of the neural network with a single node in the hidden layer and one output unit. A model of components in a simple neutral network is presented in Supplementary Figure S4.

The artificial neural network can be described by a mathematical formula such as:

$$y_{i}^{\left( 1 \right)} = f\left( {\mahop \sum \limits_{j = 1}^{{m_{0} }} w_{i,j}^{ \left( 1 \right)} *y_{j}^{\left( 0 \right)} + w_{0,j} } \right)$$

while \({y}_{j}^{\left(0\right)}\) is he *j*th input in input layer, \({w}_{i,j}^{\left(1\right)}\) the weight of the connection *j*entry node to *Yo*the node in the first hidden layer, \({w}_{0,j}\) the bias, and \(f(x)\) function called the activation function, which is a predefined function, such as the hyperbolic tangent, sigmoid function, softmax function, or Gaussian function. Information from each layer is passed to the next layer based on the formula up to the output layer.

The best artificial neural network model was established using a triple cross validation method; The same procedure was carried out to train the logistic regression model for the comparison proposals. For each round of triple cross validation, the neural network model was trained one hundred times with different randomly initialized coefficients, since the neural network easily gets stuck in the local optimal solution due to incorrect initialization. The network with the best prediction performance with the test data set was selected as the best model for each round. The final performance was estimated on the average performance of the triple cross validation. That process was repeated four times with 2, 3, 4, or 5 nodes in the hidden layer in order to determine the optimal number of hidden nodes. After the number of nodes was established, we compared the artificial neural network model with the logistic regression model based on their average triple cross-validation performance. In addition, the final neural network model exit scores across all training samples were converted to percentiles as the final risk score for readmission. Percentiles equal to or greater than 50% are considered high risk of readmission.

The training of the neural network prediction model was performed using the ‘neuralnet’ package.^{26} in R^{27}. The neural network was trained by resilient backpropagation (RPROP)^{28} with the weight tracking method.

### Model validation

For validation purposes, logistic regression and artificial neural network algorithms were prospectively applied to predict readmission in COPD patients admitted at different time periods. The readmission prediction performance was then compared between the neural network model and the logistic regression model. Area under the receiver operating characteristic curve (AUCROC), sensitivity, specificity, and positive predictive value (PPV) were calculated to compare readmission prediction in the two models. Model calibration was assessed by plotting the predicted versus observed 30-day COPD readmission rate.

### Implementation of high-risk readmission protocols for the COPD cohort

Since 2017, patients with a primary diagnosis of COPD have been evaluated weekly by the hospital’s Readmission Reduction Committee, which reviews the diagnosis and care plan of patients in the context of the risk of readmission determined by the predictive model. Patients identified as high risk for readmission received the following: specialist consultation or notification of risk status, medical educational visits by clinical pharmacists and respiratory therapists, and early medical and home health follow-up visits scheduled before discharge. The implementation of post-discharge transition phone calls (CONNECT) ensured that all aspects of discharge planning went smoothly.

All COPD patients admitted between January 2015 and July 2020 to VIZIENT were evaluated to compare readmission rates before and after the introduction of the prediction app and subsequent interventions. VIZIENT is an external dataset that provides readmission data, including readmissions outside of the Houston Methodist system. Ultimately, the Re-Admit app is available on doctors’ bedside smartphones.

### statistic analysis

Model performance was evaluated based on AUC with standard variance and 95% CIs for sensitivity, specificity, and positive predictive value (PPV). The average AUC of the models in the training set was calculated in triple cross validation. The 95% CIs of the metrics for the neural network model in the validation dataset were calculated with 2000 bootstrapping. A calibration analysis was performed to compare the alignment of the logistic regression model and the artificial neural network model on the validation data.