Introduction

In December 2019, the Wuhan Municipal Health Commission (China) reported a cluster of pneumonia cases, later identified as being caused by a novel coronavirus initially named 2019 n-CoV and then SARS-CoV-2¹; progressively, it reached high levels of intercontinental propagation until March 11 of 2020, when the World Health Organization categorized it as a pandemic². In 2020, Colombia experienced 12 784 confirmed COVID-19 deaths, with a significant excess mortality of 66 % above historical averages and 72 % of these deaths occurring in hospitals³. As a reflection of the most critical phases of the pandemic, by October 2020, 57,7 % of the intensive care unit (ICU) beds in the country were occupied. By early 2021, Colombia had doubled its capacity, increasing from 5539 ICU beds in 2019 to 11 905⁴. In Bogotá, the situation was even more critical: by January 2021, ICU bed occupancy for COVID-19 patients was around 93 %⁵. The scientific community has focused its efforts on the knowledge of the natural evolution of the disease and the recognition of risk factors that can predict adverse clinical outcomes such as admission to the ICU and/or death.

The clinical manifestations and severity of the resulting disease, known as COVID-19 (Coronavirus disease), exhibit wide heterogeneity⁶, leading to challenges in forecasting its outcomes. These manifestations can range from mild symptoms to severe pneumonia, which can rapidly progress to acute respiratory failure, septic shock, and multiple organ dysfunctions that could result in death. This variability accentuates the complexity and unpredictability of the disease's impact on individuals. It has been postulated that robust models capable of predicting the prognosis of COVID-19 are urgently needed to support decisions about protection, hospital admission, treatment, and population-level interventions⁷. A systematic review conducted shortly after the pandemic was declared identified 31 predictive models for diagnosis and prognosis⁸, an update in February 2021 already counted 232 models among which the Jehi group's model for diagnosis and the 4C Mortality Score for prognosis are considered promising⁹.

Ji et al. developed the CALL score, a clinical prediction rule for the risk of progression in 208 patients from two hospitals in Beijing (China)¹⁰. The resulting nomogram included age, comorbidities, lymphocytes, and lactate dehydrogenase, and has become one of the most widely utilized and validated to date. Liang et al. developed and validated the COVID-GRAM on 1590 patients from 575 hospitals in 31 provinces in China¹¹, the model incorporates variables such as: X-ray abnormalities, age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil to lymphocyte ratio, lactate dehydrogenase, and direct bilirubin, each weighted differently in the formula. Subsequently, two separate models were developed by the ISARIC-4C investigators group: the ISARIC-4C Deterioration model¹² and the ISARIC-4C Mortality Score¹³ on 73 948 and 35 463 patients respectively from 260 hospitals in England, Scotland, and Wales. It predicts COVID-19 severity based on age, sex, comorbidities, respiratory rate, oxygen saturation, Glasgow Coma Scale, urea/BUN, and C-reactive protein levels. The supplementary material presents the nomograms and formulas of the different prediction rules.

SARS-CoV-2 virus infection has a natural evolution different from other previously known types of pneumonia, with a high capacity for contagion and high morbidity and mortality; it is for this reason that clinical prediction rules capable of predicting unfavorable outcomes are needed. Currently, there is limited information regarding the performance of these predictive tools in our context, which makes the validation of the CALL score, COVID-GRAM, and ISARIC-4C models a priority task, especially given the risk of resurgence of the disease. The aim of this study is to validate the clinical prediction rules specifically designed for COVID-19 prognosis: CALL score, COVID- GRAM, and ISARIC-4C for the prediction of clinical worsening given by admission to intensive care or death in hospitalized patients in a cohort of 3 hospitals in Bogotá, Colombia, during 2020.

Materials and methods

Type of study: a prospective observational analytical multicenter study was carried out.

Population: inclusion criteria were: 1) adult individuals hospitalized for COVID-19 confirmed by a Real-Time Polymerase Chain Reaction (RT-PCR) in nasal swab during the period between april 15 and november 30, 2020, in three fourth-level care hospitals in Bogotá, Colombia: El Tunal Hospital, San José Hospital, and University Children's Hospital San Jose. 2) Patients with direct admission to intensive care, those referred after 72 hours of stay in another institution, pregnant women, and those with any condition that seriously affected their short-term survival, such as advanced neoplasms, poorly controlled hepatopathies, renal disease, or patients who had declined advanced resuscitation measures, were excluded from the study.

Screening: after identifying COVID-19 compatible cases from the institutional censuses, the process involved weekly reviews of the hospitalization rosters from the internal medicine services within the designated COVID-19 areas. Potential study participants were identified during these reviews. Subsequently, the inclusion criteria for each identified case were meticulously verified to ensure eligibility.

Information collection: the information for each patient was recorded in the study format only after confirming discharge, transfer to the ICU, or death. Was completed a virtual data collection form, capturing variables recommended by the International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) of the World Health Organization, based on clinical observations and laboratory data from the patient's admission¹³. Additionally, patients were followed up after their discharge to determine the outcome regarding transition to intensive care or death, ensuring a comprehensive understanding of the patient journey and the disease's progression.

Variables: the study analyzed a comprehensive set of demographics, clinical, and laboratory variables to understand their impact on COVID-19 outcomes. The key demographic and clinical variables included the number and percentage of female patients, the average age in years with Standard Deviation (SD), and the number and percentage of patients with obesity. Additionally, the presence of comorbidities was documented, including hypertension, diabetes, chronic heart disease (excluding hypertension), chronic kidney disease, smoking, chronic lung disease, chronic neurological disease, and cirrhosis. The duration of illness before hospital admission was also recorded, with the mean number of days and SD.

Laboratory variables were meticulously documented to provide a detailed clinical picture. These included the mean white blood cell count in x10³ cells per liL with SD, lymphocyte count in x10³ cells per liL with SD, and the number and percentage of patients with lymphocytes smaller than 1,2 x10³ cells per |lL. The mean platelet counts in x10³ cells per liL with SD, lactate levels in mmol/L with SD, and creatinine levels in mg/dL with SD were also recorded. Furthermore, liver enzyme levels, such as aspartate aminotransferase and alanine aminotransferase, were measured, along with the number and percentage of patients with prolonged clotting time. Inflammatory markers such as high-sensitivity C-reactive protein, ferritin, D-dimer, and lactate dehydrogenase were noted, along with the number and percentage of patients with positive high-sensitivity cardiac troponin I results. Clinical prediction scores, including the CALL score, COVID-GRAM, and ISARIC-4C, were also included to assess their association with ICU admission and mortality.

Measures to control potential biases: to mitigate potential biases, several measures were implemented throughout the study. First, strict inclusion and exclusion criteria were applied to ensure the homogeneity of the patient population, focusing on adults hospitalized with confirmed COVID-19 via RT-PCR tests. Weekly reviews of hospitalization rosters and meticulous verification of inclusion criteria helped identify eligible participants accurately. Additionally, was used a virtual data collection form to capture variables recommended by ISARIC of the World Health Organization, based on clinical observations and laboratory data from the patient's admission. Follow-ups were conducted after patient discharge to determine outcomes regarding transition to intensive care or death, ensuring comprehensive data collection and minimizing information bias. Despite these efforts, some potential biases could not be completely avoided, such as the inherent severity of cases in fourth-level hospitals and the impact of other conditions on patient outcomes.

Statistical analysis: the description was made taking the qualitative variables reported with absolute frequencies, percentages and the quantitative variables with measures of central tendency and dispersion according to the distribution of the variables. The score for each patient was calculated for each of the study scales (CALL score, COVID-GRAM, ISARIC-4C in its versions for deterioration and mortality). Based on this score, the respective receiver operating characteristic (ROC) curves were constructed for each prediction rule, taking into account the two outcomes evaluated: transfer to the ICU and in hospital death. The area under the ROC curve (AUC) with its respective 95 % confidence interval (CI) was calculated. The AUC ranges classify as follows: AUC = 0,5 indicates no better discrimination than chance, 0,7 < AUC < 0,8 indicates acceptable discrimination, 0,8 < AUC < 0,9 indicates good discrimination, and AUC >0,9 indicates excellent discrimination. The optimal cut off value was calculated using the Youden technique, assuming normal distribution of the variables, from which contingency tables were constructed and the operative characteristics (sensitivity, specificity, accuracy, positive and negative predictive values, and positive and negative likelihood ratios) were calculated for the outcomes described.

Calibration of each of the prediction rules was performed by plotting the scores obtained (x-axis) against the proportion of observed events (y-axis); regarding the prediction of ICU admission or death using LOESS (locally estimated scatterplot smoothing) curves, then was calculated the Brier Score. In order to incorporate clinical decision reasoning into the model evaluation, was used the decision curve analysis framework, where predictive models are compared with common strategies to treat all or none of the patients; the net benefit for each strategy was calculated by subtracting the proportion of false positives from true positives weighted by the relative harm of a false positive and false negative result. Analyses were performed in the statistical program R version 4.0.2 (R Foundation, Vienna, Austria) using the packages "pROC", "ROCit", and "rmda".

Sampling: a consecutive sampling (non-probabilistic) method was used, including patients in the order of their admission who met the inclusion criteria during the study period. Although no formal sample size calculation was performed prior to the start of the study, a post hoc estimation based on an expected AUC of 0,75, an event rate of 26 %, a significance level of 5 %, and a statistical power of 80 % suggests that approximately 514 patients were required.

Measures to control potential biases: given the importance of maintaining the integrity and accuracy of the analysis, it was deemed inappropriate to perform data imputation for the missing values.

Ethical considerations: the study was approved by the ethics and research committees of each of the institutions (act number 138 of the Health Services Unit from El Tunal Hospital, 0498-2020 of the San José Hospital and SDM-026-20 of the University Children's Hospital San José) and informed consent was not required. Funding was obtained from the Health Sciences University Foundation from the call for research promotion number DI-I-0631-20.

Results

Between april 15 and november 30 a total of 1150 patients were screened, of whom 793 met the inclusion criteria and were enrolled in the study.

However, only 358 patients had complete data available to calculate all three clinical prediction scores evaluated. 98 patients came from El Tunal Hospital, 142 from San José Hospital, and 118 from University Children's Hospital San José. Clinical worsening indicating transfer to the ICU primarily due to complications related to COVID-19 occurred in 186 (60 %) patients and 128 (35,7 %) patients died.

Table 1 describes the general characteristics of the population, 136 (38,0 %) were women, the average age was 58,9 years (SD 15,7) and the duration of symptoms before admission was 8 days (SD 6,3). The most frequent comorbidities were: arterial hypertension in 138 (38,5 %), obesity in 71 of the 213 patients (33,3 %), and smoking in 83 patients (23,2 %); other comorbidities less frequently identified were diabetes in 67 (18,7 %), chronic lung disease in 49 (13,7 %), chronic heart disease in 37 (10,3 %), chronic kidney disease in 18 (5 %), chronic neurological disease in 26 (7,3 %), and cirrhosis in 1 (0,3 %). The mean lymphocyte count was 1060 cells/LiL (SD 800), with 246 (68,7 %) cases below 1200; creatinine was 1.1 mg/dL (SD 1,4), C Reactive Protein 68,7 mg/dL (SD 92,4), ferritin 1091 ng/mL (SD 1124), D-dimer 1252 m g/ml (SD 2515), lactate dehydrogenase 520 U/L (SD 316); troponin I was positive in 69 of 327 patients in whom it was measured (21,1 %).

Figure 1 and Table 2 present the ROC curves of the different clinical prediction rules for the outcomes evaluated. For the prediction of admission to the ICU, the performance was moderate to poor, obtaining an AUC of 0,68 (95 % CI: 0,62-0,73) for the COVID-GRAM, 0,62 (95 % CI: 0,55-0,67) for the ISARIC-4C and 0,55 (95 % CI: 0,48-0,60) for the CALL score. Regarding the prediction of death, an improvement is shown, obtaining an AUC of 0,76 (95 % CI: 0,71-0,82) for the COVID-GRAM, 0,76 (95 % CI: 0,70-0,82) for the ISARIC-4C and 0,66 (95 % CI: 0,59-0,73) for the CALL score.

*AUC: Area Under the ROC Curve. *FPR: False Positive Rate Source: authors.

Figure 1 ROC curves with the diagnostic performance of the different clinical prediction rules. a) Intensive care admission. b) Death. 

Table 3 shows the results of the operational characteristics of each of the clinical prediction rules evaluated according to the optimal cut-off values obtained by the Youden technique. In all cases, very poor results are shown for the prediction of ICU admission, while for death the COVID-GRAM and ISARIC-4C obtain high specificities and negative predictive values (96,9 % and around 84 % respectively) at the expense of very low sensitivities (14,3 % and 12,7 %).

The calibration analysis of the different prediction rules shows how the CALL score obtained a poor result for mortality demonstrated by a high Brier Score (0,48) and the lack of linearity in the LOESS curve while it is adequate for the COVID-GRAM and ISARIC-4C rules (Figure 2 and Table 2). The three rules evaluated showed poor calibration for the prediction of ICU admission, with high Brier Score scores (Table 2). These results are in accordance with Table 4, which shows the percentages of the outcomes found in each of the rules evaluated compared to the expected result, categorized according to the risk groups set out in the original documents; the result would only indicate a good calibration for ISARIC-4C in the assessment of mortality.

*ICU: Intensive Care Unit. Source: authors.

Figure 2 Calibration using LOESS curves of the various clinical prediction rules for predicting admission to intensive care or death.  

Source: authors.

Figure 3 Decision curves for each of the clinical prediction rules evaluated. a) Intensive care admission. b) Death. 

Discussion

In our study, the evaluated outcomes, including clinical worsening leading to ICU transfer due to COVID-19 complications in 186 (52,0 %) patients and death in 128 (35,8 %) patients, align with those reported in other databases¹⁴. These findings underscore the critical importance of predicting which patients are at risk of progression to severe disease states. However, our results do not provide definitive guidance supporting the routine use of any of the evaluated prediction rules in our clinical settings. Furthermore, we encountered a significant gap in the literature, as we were unable to locate studies from our region that validate these prediction models in our local context.

The CALL score was probably one of the first rules developed specifically for the identification of clinical worsening in patients with COVID-19, reporting an AUC of 0,91 in the cohort in which it was developed¹⁰. It has been validated in several cohorts in which, despite an early report with an AUC of 0,85 among 252 patients¹⁵. Other studies show rather poor results, such as one that included 210 Italian patients with an AUC of 0,62¹⁶, another with 411 English patients with 0,56¹⁷, and what is perhaps the largest validation cohort with 1363 patients from Sâo Paulo and Barcelona with an AUC of 0,52¹⁸; while a Dutch cohort of 403 patients with an AUC of 0,70¹⁹. This poor performance is in line with that reported in this study with an AUC of 0,55.

Although it was not developed to predict mortality, three previously mentioned studies evaluated the CALL score in this sense, finding AUC of 0,77 in Italy¹⁶, 0,77 among 1363 patients from Spain and Brasil¹⁸ and 0,76 among 403 patients in The Netherlands¹⁹. This is in line with the moderate performance obtained in this study with AUC of 0,66.

The COVID-GRAM prediction rule obtained in the derivation cohort an AUC of 0,88 for clinical worsening¹¹, the various validations performed have documented a wide variability with AUC of 0,52¹⁸ in patients from Sâo Paulo and Barcelona, 0,88 among 523 Spanish patients²⁰, 0,72 in another Spanish cohort of 306 patients²¹, 0,68 among 481 Turkish patients²², and 0,70 among 14 343 French patients²³. The result of the present cohort barely reached an AUC of 0,62.

As with the CALL score, the performance of the COVID-GRAM for in-hospital death was evaluated in three different studies documenting an AUC of 0,78 among 210 elderlies in Italy²⁴, 0,77 among 1363 Brazilian and Spanish¹⁸, and 0,70 among 481 Turkish patients²² results quite close to the 0,76 achieved in the present cohort.

The ISARIC-4C scale for death in its derivation and initial validation study showed AUC of 0,75 and 0,77 respectively¹³, it has validation studies with large cohorts of patients, achieving quite consistent results of AUC to the one reached in this study of 0,76 with reports of 0,79 among 6802 American patients²⁵, 0,78 among 14 343 French patients²³ and 0,78 among 76 588 patients in the United Kingdom²⁶.

Ultimately, the ISARIC-4C rule for clinical deterioration obtained an AUC of 0,77 in the development cohort with the largest number of patients at 73 948¹³. It has undergone several validations in which an AUC of 0,80 was found among 481 Turkish patients²¹, 0,76 among 76 588 UK patients²⁶, 0,77 in the Dutch cohort of 403 patients¹⁸ and 0,76 in 410 Singaporean patients²⁷. These good results contrast with the moderate 0,62 obtained in this analysis.

Thus, the results could be grouped as moderate to poor for the prediction of clinical worsening by the three rules evaluated, with poor calibration and poor net benefit; this situation improves substantially for the prediction of death, except in the case of the CALL score. Among the limitations recognized in this study, the first is that the populations included are from fourth-level hospitals, which may induce a selection bias in patients with greater baseline severity and cannot be applied to second or first-level entities, nor to patients with COVID-19 in an outpatient setting. An additional limitation is that the estimated sample size was not reached, as more than half of the eligible cases lacked sufficient data to calculate all three risk scores, this may have reduced the statistical power of the comparisons and limited the generalizability of the findings. As a limitation or potential bias, the outcome of the patients (death or ICU admission) could be due to a reason other than COVID-19, even though exclusion criteria were applied for conditions that affect short-term survival. In this study, it is recognized as a strength the multicenter character, the large number of outcomes to make comparisons, and the calibration and net benefit analysis of the different scales.

More external validation studies are needed to ensure the generalization of its use.

Conclusions

The clinical prediction rules evaluated showed poor to moderate performance for ICU admission and good performance for death in a Colombian population hospitalized by COVID-19. Specifically, the CALL score exhibited the weakest performance, while the ISARIC-4C and COVID- GRAM showed potential utility in assessing the risk of death. Despite these findings, the results do not unequivocally support the routine application of any evaluated prediction rules in our clinical settings, echoing the broader discussion which highlights the vital need for accurate risk stratification in managing COVID-19 progression.