SciELO - Scientific Electronic Library Online

 
vol.54 número3Supervivencia de pacientes con VIH y Cancer en Cali, ColombiaCaracterización clínica y desenlaces de una cohorte de pacientes colombianos con amiloidosis AL índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

  • En proceso de indezaciónCitado por Google
  • No hay articulos similaresSimilares en SciELO
  • En proceso de indezaciónSimilares en Google

Compartir


Colombia Médica

versión On-line ISSN 1657-9534

Colomb. Med. vol.54 no.3 Cali jul./set. 2023  Epub 25-Sep-2023

https://doi.org/10.25100/cm.v54i3.5580 

Original article

Clinimetric properties of the Perme Intensive Care Unit Mobility Score -a multicenter study for minimum important difference and responsiveness analysis

Ricardo Kenji Nawa1 
http://orcid.org/0000-0002-0852-7013

Marcio Luiz Ferreira De Camillis2 
http://orcid.org/0000-0001-5566-6422

Monique Buttignol1  3 
http://orcid.org/0000-0002-7195-6400

Fernanda Machado Kutchak4  5 
http://orcid.org/0000-0002-5717-0380

Eder Chaves Pacheco1 
http://orcid.org/0000-0001-7367-854X

Louise Helena Rodrigues Gonçalves3 
http://orcid.org/0000-0002-2386-9847

Leonardo Miguel Correa Garcia2 
http://orcid.org/0000-0002-5730-3147

Karina Tavares Timenetsky1 
http://orcid.org/0000-0002-4176-2445

Luiz Alberto Forgiarini Junior6 
http://orcid.org/0000-0002-6706-2703

1 Hospital Israelita Albert Einstein, São Paulo, SP, Brazil

2 Hospital Moinhos de Vento, Porto Alegre, RS, Brazil

3 Hospital Municipal da Vila Santa Catarina Dr. Gilson de Cássia Marques de Carvalho;São Paulo, SP, Brazil

4 Universidade Vale dos Sinos, Porto Alegre, RS, Brazil

5 Universidade Federal do Rio Grande do Sul, Postgraduate Program in Health Sciences: Cardiology and Cardiovascular Sciences, Porto Alegre, RS, Brazil:

6 Universidade Católica de Pelotas - UCPel, Medicine Course, Postgraduate Program in Health and Behavior, Pelotas, RS, Brazil.


Abstract

Background:

The use of instruments in clinical practice with measurement properties tested is highly recommended, in order to provide adequate assessment and measurement of outcomes.

Objective:

To calculate the minimum clinically important difference (MCID) and responsiveness of the Perme Intensive Care Unit Mobility Score (Perme Score).

Methods:

This retrospective, multicentric study investigated the clinimetric properties of MCID, estimated by constructing the Receiver Operating Characteristic (ROC). Maximizing sensitivity and specificity by Youden's, the ROC curve calibration was performed by the Hosmer and Lemeshow goodness-of-fit test. Additionally, we established the responsiveness, floor and ceiling effects, internal consistency, and predictive validity of the Perme Score.

Results:

A total of 1.200 adult patients records from four mixed general intensive care units (ICUs) were included. To analyze which difference clinically reflects a relevant evolution we calculated the area under the curve (AUC) of 0.96 (95% CI: 0.95-0.98), and the optimal cut-off value of 7.0 points was established. No substantial floor (8.8%) or ceiling effects (4.9%) were observed at ICU discharge. However, a moderate floor effect was observed at ICU admission (19.3%), in contrast to a very low incidence of ceiling effect (0.6%). The Perme Score at ICU admission was associated with hospital mortality, OR 0.86 (95% CI: 0.82-0.91), and the predictive validity for ICU stay presented a mean ratio of 0.97 (95% CI: 0.96-0.98).

Conclusion:

Our findings support the establishment of the minimum clinically important difference and responsiveness of the Perme Score as a measure of mobility status in the ICU.

Keywords: Intensive care unit; outcome measures; clinimetric properties; rehabilitation; ROC Curve; Calibration; Outcome Assessment; Health Care

Resumen

Antecedentes:

Se recomienda encarecidamente el uso de instrumentos en la práctica clínica con propiedades de medición probadas, con el fin de proporcionar una evaluación y medición adecuada de los resultados.

Objetivo:

Calcular la diferencia mínima clínicamente importante (MCID) y la capacidad de respuesta de la puntuación de movilidad de la unidad de cuidados intensivos de Perme (Perme Score).

Métodos:

Este estudio multicéntrico retrospectivo investigó las propiedades clinimétricas de MCID, estimadas mediante la construcción de la característica operativa del receptor (ROC). Maximizando la sensibilidad y especificidad mediante la prueba de Youden, la calibración de la curva ROC se realizó mediante la prueba de bondad de ajuste de Hosmer y Lemeshow. Además, establecimos la capacidad de respuesta, los efectos suelo y techo, la consistencia interna y la validez predictiva del Perme Score.

Resultados:

Se incluyeron un total de 1,200 registros de pacientes adultos de cuatro unidades de cuidados intensivos (UCI) generales mixtas. Para analizar qué diferencia refleja clínicamente una evolución relevante calculamos el área bajo la curva (AUC) de 0.96 (95% CI: 0.95-0.98); y se estableció el valor de corte óptimo de 7.0 puntos. No se observaron efectos suelo (8.8%) o techo (4.9%) sustanciales al alta de la UCI. Sin embargo, se observó un efecto suelo moderado al ingreso en la UCI (19.3%), en contraste con una incidencia muy baja del efecto techo (0.6%). El Perme Score al ingreso en UCI se asoció con la mortalidad hospitalaria, OR 0.86 (95% CI: 0.82-0.91), y la validez predictiva de estancia en UCI presentó una relación media de 0.97 (95% CI: 0.96-0.98).

Conclusiones:

Nuestros hallazgos respaldan el establecimiento de la diferencia mínima clínicamente importante y la capacidad de respuesta de el Perme Score como medida del estado de movilidad en la UCI.

Palabras clave: unidad de cuidados intensivos; medidas resultado; propiedades clinimétricas; rehabilitación; Curva ROC; Calibración; Evaluación de resultados; Asistencia sanitaria

Remark

1) Why was this study conducted?
This study was conducted to evaluate the clinimetric properties of the Perme Intensive Care Unit Mobility Score..
2) What were the most relevant results of the study?
This study established the minimum clinically important difference (MCID), responsiveness, the ceiling and floor effects, the internal consistency, and predictive validity of the Perme Score..
3) What do these results contribute?
The results of this study will allow a better understand and interpretation of the results of the Perme Score as an outcome assessment instrument in clinical practice and research purposes.

Introduction

A systematic review of physical function endpoints in critical care patients has found significant variability in measurement instruments utilized for clinical and research purposes 1. The evaluation of physical function is essential to deliver an effective and assertive therapeutic plan, able to predict individuals at high risk of developing functional impairments after critical care episodes and hospitalization 2,3. The Perme Intensive Care Unit Mobility Score (Perme Score) was designed to assess patients' mobility status when admitted to the intensive care unit (ICU) 1,4. The Perme Score consists of seven domains 4. Starting with the assessment of patients' alertness observed upon arrival and initial contact with the rater and ending with the total distance walked over 2 minutes, the total score ranges from 0 to 32 points, with higher scores indicating higher mobility status 4. Since the initial publication of the proposal, several studies have been published using the Perme Score as an assessment tool to objectively measure the mobility level of ICU patients 5-11.

However, measurement instruments must be sensitive to detect clinical changes and have robust measurement properties tested for validity and reliability 12. The Consensus-based Standards for selecting health Measurement Instruments (COSMIN) 13 initiative developed a taxonomy and three quality domains (e.g., reliability, validity, and responsiveness) 13, the greater the number of clinimetric properties tested, the more robust the measurement instrument tends to be. The interrater reliability of the Perme Score was previously tested, showing an excellent agreement between observers 5. The minimum clinically important difference can be defined as the most minor difference in score for an outcome of interest that would lead the patient or clinician to consider a change (e.g., better or worse) in the patient's status 14. The present study aims to establish the minimum clinically important difference (MCID) and the responsiveness of the Perme Score in generalizable patients admitted to the ICU. Secondly, we aimed to analyze the measurement properties: (1) floor and ceiling effect of the Perme Score to ICU admission and ICU discharge; (2) internal consistency; (3) predictive validity for in-hospital and hospital stay after ICU discharge.

Materials and Methods

Study design and ethics approval

This study is reported following the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement 15, and the Consensus-based Standards for selecting Health Measurement Instruments (COSMIN) methodology 13. A multicenter retrospective study including a convenience sample of patients from four private ICUs. All sites classified their ICUs as 'mixed general ICUs'. The local Institutional Review Board (IRB) of each hospital approved the study (Table 1S on the Supplementary Material for local IRB approval). Local institutional review boards approved this study, as follows: Hospital Israelita Albert Einstein Ethics Committee, Hospital Municipal Vila Santa Catarina Ethics Committee, Hospital Moinhos de Vento Ethics Committee and Hospital Nossa Senhora da Conceição Ethics Committee approved this study. The informed consent was waived for all sites for this retrospective study.

Participants

Records were eligible for inclusion if they had been admitted to the ICU between January 01, 2019, to June 30, 2019. The patients included in the study should be required mechanical ventilation support for a period of more than 24 hours. The exclusion criteria adopted were: (1) patients under 18 years old; (2) diagnosis of neurodegenerative diseases; (3) patients with cognitive impairment prior to hospital admission; (4) patients on comfort measures (e.g., palliative care); and (5) primary diagnosis of myopathic or neurological processes associated with muscle weakness (e.g., Guillain-Barré syndrome). All patients were followed up until discharge from the intensive care unit.

Data collection and study variables

All data were extracted by an independent research assistant who did not participate in this study and were fully anonymized before being made available to researchers. Collected variables included demographics, comorbidities, Simplified Acute Physiology Score (SAPS III) at ICU admission, with higher scores indicating more severe illness and higher risk of death 16, Sequential Organ Failure Assessment score (SOFA) at ICU admission, with higher aggregate scores indicating more severe organ dysfunction 17, Charlson Comorbidity Index (CCI), with a score of zero indicating that no comorbidities were found 18. The higher the score, the more likely the predicted outcome will result in mortality or higher resource use 18, Modified Frailty Index (MFI) - categorized frailty using MFI values into non-frail (MFI= 0), pre-frail (MFI= 1-2) or frail (MFI ≥3) 19, resource use and organ support duration of MV, ICU length of stay, and ICU and in-hospital mortality.

Statistical analysis

Continuous variables were reported as median [interquartile range (IQR)], since none of the variables presented symmetry or normality, assessed by the Shapiro-Wilk test. Categorical variables were presented as absolute and relative frequencies. Analyses were performed using 'R software', version 4.0.3 (R Foundation) 20, and 0.05 was set as an indicator of statistical significance throughout.

Minimum clinically important difference

The MCID was established by constructing the Receiver Operating Characteristic (ROC) curve with Perme Score at ICU admission, and the subsequent calculation of the area under the curve (AUC) determined the best cut-off point to discriminate between patient groups classified as 'little better' and 'much better' 21. Patients classified as 'worsening' were excluded from the analysis. The MCID is the value that results in a greater AUC while maximizing sensitivity and specificity. Youden's criterion was utilized to define this point.

Responsiveness

Responsiveness was assessed by considering three mobility milestones, as follows: (1) sitting activity; (2) standing activity, and (3) walking activity, by using the items of the Perme Score at ICU discharge, which were considered essential for defining clinical improvement (e.g.: item '9' supine to sit; item '11' sit to stand; and item '14' gait, respectively). We assessed change over time 22 based on the capacity of the Perme Score to predict these questions as an outcome by calculating the differences between 'ICU admission' and 'ICU discharge' values. The ROC curves were constructed and the AUC were calculated, with 95% CI, testing for the null hypothesis of an AUC equal to 0.5. The calibration of the ROC curves were assessed by the Hosmer and Lemeshow goodness-of-fit test (H-L test), with results presented by test statistics (Ji 2), degrees of freedom (Df) and p-value.

Floor and ceiling effects

The floor and ceiling effects were calculated by assessing the proportion of patients who obtained the lowest and highest Perme Score values. Floor and ceiling effects of less than 15% were considered acceptable 23,24.

Internal consistency

Internal consistency was assessed using Cronbach's alpha coefficient (α-Cronbach) 25, with a 95% CI, which was calculated between items with similar responses (e.g., Perme Score items '2 to 6', '7 and 8', and '9 to 15') 26. Internal consistency was considered adequate if α was between 0.70 and 0.95 27.

Predictive validity

The predictive validity of the Perme Score at ICU admission and the in-hospital mortality were performed using a generalized estimating equation model with binomial distribution, considering the expected correlation between data from the same center 28. The adequacy of the score in the prediction was evaluated by ROC curve and by calculating the AUC and the calibration of the ROC curve, assessed by H-L test 29. Model results were presented by (OR, 95% CI) and p-value. The relationship between the Perme Score at ICU admission and the ICU stay or hospital stay after ICU discharge was investigated using a mixed linear model applied to the length of stay log-transformed and considering the expected correlation between data from the same center. The results are presented by differences or mean ratios, 95% CI, and p-values. For the length of hospital stay after ICU, the constant 0.5 was added to all observations before the log transformation to deal with cases equal to zero.

Results

During the observation period, 1,209 patients were initially screened from a mix of medical and surgical ICUs for inclusion in this study. Of these, 1,200 (99.3%) patients met the inclusion criteria and were included in subsequent analyses (Figure 1).

Figure 1 Consolidate flowchart of included patients Definition of abbreviations: ICU = intensive care unit; LOS = length of stay. 

Nine patients (0.7%) were excluded due to missing core data. Baseline characteristics with demographic details of pooled patients' data are shown in Table 1 (Table 2S on the Supplementary Material for additional information about baseline characteristics and clinical outcomes of the included patients). The median [IQR] age of patients was 67 (55-79) years, 57.1% were male, and the total Perme Score at ICU admission mean 9.37 (95% CI: 8.84-9.90), and 17.20 (95% CI: 17.25-18.31) at ICU discharge of a total n = 1,200 patients. Excluding deaths and transfers for other hospitals the mean of 9.75 (95% CI: 9.21-10.29), and 19.05 (95% CI: 18.51-19.59) at ICU discharge of a total n= 1,089 patients (Table 3S on the Supplementary Material for additional information about patients' organ support during ICU stay).

Table 1 Baseline characteristics and clinical outcomes of the included patients 

Overall HIAE1 HMVSC2 HMV3 HNSC4
no. (%) 1,200 (100) 455 (37.9) 268 (22.3) 305 (25.4) 172 (14.3)
Age, years, median (IQR) 67 (55-79) 75 (62-86) 60 (48-68) 70 (60-81) 61 (49-70)
Male gender - no (%) 685 (57.1) 259 (56.9) 159 (59.3) 171 (56.1) 96 (55.8)
Perme Score§, total points, median (IQR)
ICU admission 8 (2-13) 8 (4-11) 9 (8-19) 7 (2-18) 0 (0-8)
ICU discharge 21 (9-27) 18 (8-26) 22 (11.5-26) 24 (20-28) 8 (0-18)

Data are median and interquartile range [quartile 25% - quartile 75%] or n (%). Percentages may not total 100 because of rounding.

Definition of abbreviations: 1HIAE = Hospital Israelita Albert Einstein, São Paulo, Brazil; 2 HMVSC = Hospital Municipal Vila Santa Catarina, São Paulo, Brazil; 3 HMV = Hospital Moinhos de Vento, Porto Alegre, Brazil; 4 HNSC = Hospital Nossa Senhora da Conceição, Porto Alegre, Brazil.

§ Perme ICU mobility score range from 0 to 32, with higher scores indicating better mobility level.

||The Perme Mobility Index (PMI) is calculated by the difference between the total Perme Score at ICU discharge and the total Perme Score at ICU admission, divided by the ICU length of stay (ICU LOS) [PMI = ΔPerme Score (ICU discharge - ICU admission) / ICU LOS]. The result is a dimensionless number and it can be either positive or negative. Positive values are associated with patients that improve the mobility status during ICU stay, whereas negative values are associated with patients that decrease mobility status during ICU stay.

Minimum clinically important difference

The MCID was determined by constructing a ROC curve that showed an AUC= 0.96; 95% CI: 0.95-0.98, and the optimal cut-off value of 6.5, by Youden criteria, with a specificity of 95.5% and a sensitivity of 88.9%, OR= 1.78; 95% CI: 1.62-1.97 (p <0.001) and H-L test χ2= 15.6; Df (7), p= 0.029] (Figure 1S on the Supplementary Material for additional information). The summary of clinical mobility improvement and mobility milestone occurrence are presented in Tables 2 and 3, respectively. When analyzing clinical mobility improvement, 723 patients were included in this analysis, corresponding to 60.2% of the sample. Mobility milestone occurrence was measured at ICU discharge, all patients were included in the study analysis.

Table 2 Clinical mobility improvement 

Overall HIAE1 HMVSC2
1,200 (100) 455 (37.9) 268 (22.3)
Clinical improvement - no.(%)
Worse 83 (6.9) 55 (12.1) 28 (10.5)
Stable 269 (22.4) 173 (38.0) 96 (35.8)
Little improvement 120 (10) 67 (14.7) 53 (19.8)
Much better 249 (20.8) 158 (34.7) 91 (34)

Data are n (%), percentages may not total 100 because of rounding.

Definition of abbreviations: 1 HIAE = Hospital Israelita Albert Einstein, São Paulo, Brazil; 2 HMVSC = Hospital Municipal Vila Santa Catarina, São Paulo, Brazil.

Table 3 Mobility milestone occurrence rate at ICU discharge 

Overall HIAE1 HMVSC2 HMV3 HNSC4
Sit, no. (%) 733 (68.4) 296 (65.1) 123 (45.9) 280 (91.8) 34 (19.8)
Stand, no. (%) 661 (60.6) 258 (56.7) 104 (38.8) 268 (87.9) 31 (18)
Walk, no. (%) 494 (41.2) 146 (32.1) 97 (36.2) 237 (77.7) 14 (8.1)

Data are n (%), percentages may not total 100 because of rounding.

Definition of abbreviations: 1 HIAE = Hospital Israelita Albert Einstein, São Paulo, Brazil; 2 HMVSC = Hospital Municipal Vila Santa Catarina, São Paulo, Brazil; 3 HMV = Hospital Moinhos de Vento, Porto Alegre, Brazil; 4 HNSC = Hospital Nossa Senhora da Conceição, Porto Alegre, Brazil.

Responsiveness

The overall responsiveness of the Perme Score showed a good capacity to detect changes over time, from ICU admission to ICU discharge, as follows: (1) sit position- the ROC curve showed an AUC= 0.89 (95% CI: 0.87-0.90), and the optimal cut-off value of 5.5, a specificity of 89.7%, and a sensitivity of 74.9%, OR= 1.37 (95% CI: 1.31-1.44; p <0.001) and H-L test χ2= 3.83 (Df 8), p= 0.872) (Figure 2-A ); (2) stand position - the ROC curve, which showed an AUC = 0.87 (95% CI: 0.85-0.89) and the optimal cut-off value of 9.5, with a specificity of 96.5%, and a sensitivity of 65.4%, OR= 1.30 (95% CI: 1.26-1.35; p <0.001) and H-L test χ2= 8.77 (Df (8), p= 0.362) (Figure 2-B ); and (3) walk - the ROC curve showed an AUC= 0.79 (95% CI: 0.77-0.82) and the optimal cut-off value of 13.5 specificity of 85.6%, and a sensitivity of 59.5%, OR= 1.15 (95% CI: 1.13-1.17; p <0.001) and H-L test χ2= 33.56 (Df (8), p <0.001) (Figure 2-C ). Additional information about the optimal cut-off value is shown at Table 4S, 5S, and 6S on the Supplementary Material to summarize the Perme Score responsiveness.

Figure 2 Responsiveness of the Perme Score. (A) Sit position - the ROC curve a the AUC= 0.89, 95% CI: 0.87-0.90, and the optimal cut-off value of 5.5 specificity of 89.7%; 95% CI: 86.14-92.63, and a sensitivity of 74.9%; 95% CI: 71.76-78.04 and calibration by H-L test χ2= 3.83 (Df (8), p= 0.872); (B) Stand position - the ROC curve shows the AUC= 0.87; (95% CI: 0.85-0.89), and the optimal cut-off value of 9.5 specificity of 96.5% (95% CI: 94.64-98.14), and a sensitivity of 65.4% (95% CI: 61.57-68.99) and calibration by H-L test χ2= 8.77 (Df (8), p= 0.362); and (C) Walk - the ROC curve shows the AUC= 0.79 (95% CI: 0.77-0.82), and the optimal cut-off value of 13.5 specificity of 85.6% (95% CI: 82.86-88.10), and a sensitivity of 59.5% (95% CI: 55.26-63.77) and calibration by H-L test χ2= 33.56 (Df (8), p <0.001). Definition of abbreviations: AUC= area under the curve, H-L test= Hosmer & Lemeshow test, ROC= receiver operator characteristics. 

Floor and ceiling effects

The Perme Score presented a significant floor effect at ICU admission of 19.3% (scored '0' points), and a ceiling effect of 0.6% (scored '32' points). Nevertheless, the Perme Score presented acceptable floor and ceiling effects at ICU discharge of 8.8% (scored '0' points) and 4.9% (scored '32' points), respectively.

Internal consistency

The internal consistency was based on the Cronbach's alpha coefficient (Cronbach's α) value. For the Perme Score items' 2 to 6', the internal consistency was moderate at ICU admission and discharge, as follows: Cronbach's α= 0.70 (95% CI: 0.67-0.73) and 0.65 (95% CI: 0.61-0.69), respectively. For items 1, 7 and 8, the internal consistency was excellent at ICU admission and discharge, as follows: Cronbach's α= 0.93 (95% CI: 0.93-0.94) and 0.89 (95% CI: 0.88-0.90) respectively. For the Perme Score items' 9 to 15', the internal consistency observed was also excellent at ICU admission and discharge, as follows: Cronbach's α= 0.96 (95% CI: 0.95-0.96) and 0.97 (95% CI: 0.96-0.97), respectively.

Predictive validity

The total Perme Score at ICU admission predictive validity for hospital mortality showed OR= 0.86 (95% CI: 0.82-0.91). We also constructed a ROC curve, which showed an AUC = 0.792 (95% CI: 0.67-0.78) and an optimal cut-off value of 8.5, by Youden criteria, with a specificity of 0.53 (Figure 2S on the Supplementary Material for a summary of the predictive validity). The predictive validity between the Perme Score at ICU admission and the ICU length of stay showed that the addition of one unit in the Perme Score is expected to decrease 3.1% in the average ICU length of stay, and a mean ratio of ICU stay MR= 0.969 (95% CI: 0.96-0.97). However, the predictive validity between the Perme Score at ICU discharge and hospital stay after ICU discharge showed no evidence of a relationship between the variables and a mean ratio of hospital stay after ICU discharge MR= 0.99 (95% CI: 0.98-1.0) for hospital stay after ICU discharge.

Discussion

This study has shown the Perme Score establishment of the MCID and the predictive validity of hospital mortality and ICU length of stay. The Perme Score was also responsive to change; no floor or ceiling effects were present at ICU discharge in a mixed general ICU population. This is the first study to assess the predictive validity of the Perme Score about in-hospital mortality.

When selecting a measurement instrument, it is important to guarantee that it is being utilized only in the population for which they are intended or validated and at the time they are responsive 22,23,27,30 The clinimetric properties test is necessary to use outcome measures with strong evidence 30. This study calculating the MCID of the Perme Score will allow for a better understanding and interpretation of the results reported in previous studies already published using the Perme Score as an outcome measure 6-10,31-34. This should help clinicians and researchers provide better accuracy in physical therapy and rehabilitation interventions. However, this is not the first study to calculate the MCID of the Perme Score. A previous single-center study conducted with patients admitted to the ICU established an MCID of 1.36 points attributed to improved functional mobility 11. This value can be considered questionable because the value seems to be too narrow once the Perme Score total points range from '0' to '32' points. Additionally, the MCID found in this study might be appropriate for the convenience sample included in the study, not considering a higher number of patients from multicenter studies, which may better represent a population of a general ICU cohort. The strengths of our study were the multicenter design with a large number of patients included and a wide range of admission diagnoses, allowing generalizability of the results. The literature supports using different methodologies to calculate the MCID (e.g., anchor and distribution-based) 21,35-37. The study published by Wilches-Luna et al. (2020) 11, calculated the MCID using a distribution-based approach 37,38, based on the calculation of the standard error of measurement (SEM) index. The SEM index is calculated with the square root of two times the cut-off value of 1.96, representing a normal standard curve associated with a 95% CI 11. Our study adopted the methodology resulting in the greatest AUC, maximizing sensitivity and specificity, according to Youden's criterion, which tends to be more consistent when compared with the distribution-based approach. In order to establish the more appropriate value of MCID, we chose the data point closest to the upper left corner of the curve, with the best balance of sensitivity and specificity 35.

The results of this study also showed good responsiveness of the Perme Score when applied to a general mixed ICU population. The score detected changes in patients' mobility status over time during ICU stay. The responsiveness of a measurement instrument is a relevant psychometric item of a clinical scoring system. It determines whether the measure can detect changes it was designed to measure in the environment and cohort it was intended for 22 other measures of physical function have been designed for use in the ICU, and studies have been published testing the clinimetric properties of these instruments 1. We found a moderate floor effect at ICU admission. However, this is very likely to be related to the deep sedation of participants included in the study, yet close to values less than 15% considered acceptable by the literature 23,24. This finding is lower when compared with 31% found in a previous study 11. Similarly to the Wilches-Luna et al. 11, study, the ceiling effect was shallow at ICU admission and can be considered nonexistent. None of the previous studies have tested the predictive validity of the Perme Score, although this has been shown in this study. The findings of our study presented consistent data on the predictive validity of the Perme Score for hospital mortality and ICU length of stay.

Since the Perme Score is available in different languages 4,6,7,39, , we believe the results of this study will allow for extrapolation of the findings to interpretations of clinical conditions and research purposes, thus improving its utility. However, it is important to emphasize that no physical function assessment scoring system will ever be able to remain responsive to patients' clinical progress during the hospital stay linearly.

Our study has limitations. Although this was a multicentric study conducted in four different centers, the retrospective nature of this study should be considered a limitation. First, due to the significant number of enrolled patients, we truly believe our findings can contribute to clinical practice and a better understanding of mobility status assessment for hospitalized patients. Second, the time frame established was limited only during the ICU stay; we did not follow patients' mobility until hospital discharge due to the lack of the validation process of the use Perme Score outside the ICU setting. Another possible limiting factor would be that some analyzed variables, mainly in relation to the characterization of the sample, are not presented by all the centers included. However, we believe that this does not compromise the primary outcome of the present study, which was presented by all centers, which ensures adequate analysis as well as the outcomes obtained. Because the test can have a switching effect and is modifiable by therapeutic interventions, studies with a prospective design are required to evaluate these effects.

Conclusion

The establishment of the Perme Score minimum clinically important difference will improve its clinical use and the interpretation of the total score of points. A limited overall ceiling and moderate floor effect at ICU admission were observed. The Perme Score presented strong evidence of its capability to detect changes in patients' mobility status over time, showing responsiveness in a mixed general ICU population.

Acknowledgments

The authors acknowledge Raquel Afonso Caserta Eid and Dr Thiago Domingos Corrêa for providing administrative support. We also thank Elivane da Silva Victor for supporting statistical analysis and Helena Spalic for proofreading this manuscript.

References

1. Parry SM, Grander CL, Berney S, Jones J, Beach L, El-Ansary D, et al. Assessment of impairment and activity limitations in the critically ill: a systematic review of measurement instruments and their clinimetric properties. Intensive Care Med. 2015; 41: 744-762. doi: 10.1007/s00134-015-3672-x [ Links ]

2. Parry SM, Knight LD, Baldwin CE, Sani D, Kayambu G, Da Silva VM, et al. Evaluating physical functioning in survivors of critical illness: development of a new continuum measure for acute care. Crit. Care Med. 2020; 48: 1427-1435. doi: 10.1097/CCM.0000000000004499. [ Links ]

3. Rathore S, George P. Physical function in critical care tool bridges the waters of icu and post acute care physical functioning assessments. Critical Care Medicine. 2020; 48: 1532-1533. doi: 10.1097/CCM.0000000000004537. [ Links ]

4. Perme C, Nawa RK, Winkelman C, Masud F. A tool to assess mobility status in critically ill patients: the Perme Intensive Care Unit Mobility Score. Method Debakey Cardiovasc J. 2014; 10: 41-49. doi: 10.14797/mdcj-10-1-41. [ Links ]

5. Nawa RK, Lettvin C, Winkelman C, Evora PRB, Perme C. Initial interrater reliability for a novel measure of patient mobility in a cardiovascular intensive care unit. J Crit Care. 2014; 29: 475.e1-5. doi: 10.1016/j.jcrc.2014.01.019 [ Links ]

6. Wilches LEC, Hernández NL, Oliveira AS, Nawa RK. Perme C, Gastaldi AC, Perme ICU Mobility Score (Perme Score) and the ICU Mobility Scale (IMS): translation and cultural adaptation for the Spanish language. Colomb Med (Cali). 2018; 49: 265-272. doi: 10.25100/cm.v49i3.4042. [ Links ]

7. Kawaguchi YMF, Nawa RK, Figueiredo TB, Martins L, Pires-Neto RC. Perme Intensive Care Unit Mobility Score and ICU Mobility Scale: translation into Portuguese and cross-cultural adaptation for use in Brazil. J Bras Pneumol. 2016; 42: 429-434. doi: 10.1590/S1806-37562015000000301 [ Links ]

8. Pereira CS, Carvalho AT, de Bosco AD, Forgiarini JLA. The Perme scale score as a predictor of functional status and complications after discharge from the intensive care unit in patients undergoing liver transplantation. Rev Bras Ter Intensiva. 2019; 31: 57-62. doi: 10.5935/0103-507X.20190016 [ Links ]

9. Timenetsky KT, Serpa NA, Lazarin AC, Pardini A, Moreira CRS, Corrêa TD, et al. The Perme Mobility Index: A new concept to assess mobility level in patients with coronavirus (COVID-19) infection. PLoS One. 2021; 16: e0250180. doi: 10.1371/journal.pone.0250180 [ Links ]

10. Luna ECW, Perme C, Gastaldi AC. Relationship between potential barriers to early mobilization in adult patients during intensive care stay using the Perme ICU Mobility score. Can J Respir Ther. 2021; 57: 148-153. doi: 10.29390/cjrt-2021-018 [ Links ]

11. Luna ECW, de Oliveira AS, Perme C, Gastaldi AC. Spanish version of the Perme Intensive Care Unit Mobility Score: Minimal detectable change and responsiveness. Physioth Res Internat. 2021; 26(1): e1875. doi: 10.1002/pri.1875 [ Links ]

12. Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC. 2010; 10: 22. doi: 10.1186/1471-2288-10-22 [ Links ]

13. Mokkink LB, Prinsen CAC, Bouter LM, de Vet HCW, Terwee CB. The Consensus-based Standards for the selection of health Measurement INstruments (COSMIN) and how to select an outcome measurement instrument. Braz J Phys Ther. 2016; 20: 105-113. doi: 10.1590/bjpt-rbf.2014.0143 [ Links ]

14. Schünemann HJ, Guyatt GH. Commentary-goodbye M(C)ID! Hello MID, where do you come from? Health Serv Res. 2005; 40: 593-597. doi: 10.1111/j.1475-6773.2005.00374.x [ Links ]

15. von Elm E, Altman DG, Egger M, Pocok SJ, Gøtzsche PC, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007; 370: 1453-1457. doi: 10.1016/S0140-6736(07)61602-X [ Links ]

16. Moreno RP, Metnitz PGH, Almeida E, Jordan B, Bauer P, Campos R A, et al. SAPS 3-From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005; 31: 1345-1355. doi: 10.1007/s00134-005-2763-5 [ Links ]

17. Vincent JL, Moreno J, Takala J, Willatts S, de Mendonça A, Bruning H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996; 22: 707-710. doi: 10.1007/BF01709751 [ Links ]

18. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987; 40: 373-383. doi: 10.1016/0021-9681(87)90171-8 [ Links ]

19. Zampieri FG. Iwashyna TJ, Viglianti EM, Taniguchi LU, Viana WN, Costa R, et al. Association of frailty with short-term outcomes, organ support and resource use in critically ill patients. Intensive Care Med. 2018; 44: 1512-1520. doi: 10.1007/s00134-018-5342-2. [ Links ]

20. R Core Team. RA Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. References-Scientific Research Publishing; 2019. [ Links ]

21. Wyrwich KW, Tierney WM, Wolinsky FD. Further evidence supporting an SEM-based criterion for identifying meaningful intra-individual changes in health-related quality of life. J Clin Epidemiol. 1999; 52: 861-873. doi: 10.1016/s0895-4356(99)00071-2 [ Links ]

22. Deyo RA, Centor RM. Assessing the responsiveness of functional scales to clinical change: an analogy to diagnostic test performance. J Chronic Dis. 1986; 39: 897-906. doi: 10.1016/0021-9681(86)90038-x [ Links ]

23. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007; 60: 34-42. doi: 10.1016/j.jclinepi.2006.03.012 [ Links ]

24. McHorney CA, Tarlov AR. Individual-patient monitoring in clinical practice: are available health status surveys adequate? Qual Life Res. 1995; 4: 293-307. doi: 10.1007/BF01593882 [ Links ]

25. DeVellis RF. Scale development: theory and applications. SAGE Publications ; 2016. [ Links ]

26. Knapp TR. Focus on psychometrics. coefficient alpha: conceptualizations and anomalies. Res Nurs Health. 1991; 14: 457-460. doi: 10.1002/nur.4770140610 [ Links ]

27. Terwee CB, Mokkink LB, Knol DL, Ostelo RWJG, Bouter LM, de Vet HCW. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012; 21: 651-657. doi: 10.1007/s11136-011-9960-1 [ Links ]

28. Liang K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986; 73: 13-22. Doi: 10.1093/biomet/73.1.13 [ Links ]

29. Hosmer DW Jr, Lemeshow S, Sturdivant RX. Chapter 8 Logistic regression models for multinomial and ordinal outcomes. Applied Logistic Regression. Wiley Series in Probability and Statistics. 3rd. Wiley; 2013. doi: 10.1002/9781118548387.ch8 [ Links ]

30. Tipping CJ, Young PJ, Romero L, Saxena MK, Dulhunty J, Hodgson CL. A systematic review of measurements of physical function in critically ill adults. Crit Care Resusc. 2012; 14: 302-311. [ Links ]

31. Nawa RK, Serpa NA, Lazarin AC, da Silva AK, Nascimento C, Midega TD, et al. Analysis of mobility level of COVID-19 patients undergoing mechanical ventilation support: A single center, retrospective cohort study. PLoS One. 2022; 17: e0272373. doi: 10.1371/journal.pone.0272373. eCollection 2022 [ Links ]

32. Yen H-C, Han Y-Y, Hsiao W-L, Hsu P-M, Pan G-S, Li M-H, et al. Functional mobility effects of progressive early mobilization protocol on people with moderate-to-severe traumatic brain injury: A pre-post intervention study. NeuroRehabilitation. 2022; 51(2):303-313. doi: 10.3233/NRE-220023.. [ Links ]

33. Gatty A, Samuel SR, Alaparthi GK, Prabhu D, Upadya M, Krishnan S, et al. Effectiveness of structured early mobilization protocol on mobility status of patients in medical intensive care unit. Physiother Theory Pract. 2022; 38(10): 1345-13571-13 doi:10.1080/09593985.2020.1840683. [ Links ]

34. Ceron C, Otto D, Signorini AV, Beck MC, Camillis M, Sganzerla D, et al. The effect of speaking valves on ICU mobility of individuals with tracheostomy. Respir Care. 2020; 65: 144-149. doi: 10.4187/respcare.06768 [ Links ]

35. Deyo RA, Diehr P, Patrick DL. Reproducibility and responsiveness of health status measures. Statistics and strategies for evaluation. Control Clin Trials. 1991; 12: 142S-158S. doi: 10.1016/s0197-2456(05)80019-4 [ Links ]

36. Guyatt G, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Dis. 1987; 40: 171-178. doi: 10.1016/0021-9681(87)90069-5 [ Links ]

37. King MT. A point of minimal important difference (MID): a critique of terminology and methods. Expert Rev Pharmacoecon Outcomes Res. 2011; 11: 171-184. doi: 10.1586/erp.11.9 [ Links ]

38. Beaton DE, Boers M, Wells GA. Many faces of the minimal clinically important difference (MCID): a literature review and directions for future research. Curr Opin Rheumatol. 2002; 14: 109-114. doi: 10.1097/00002281-200203000-00006 [ Links ]

39. Nydahl P, Wilkens S, Glase S, Mohr LM, Richter P, Klarmann S, et al. The German translation of the Perme Intensive Care Unit Mobility Score and inter-rater reliability between physiotherapists and nurses. Europ J Physioth. 2018; 20: 109-115. doi: 10.1080/21679169.2017.1401660 [ Links ]

Notas:

Funding No funding was received for this work.

Material supplementary

Table 1S Institutional Review Board approval 

Local CPEA IRB local number
HIAE1 30797520.6.0000.0071 4.333.***
HMVSC2 38589220.6.2003.0071 4.483.***
HMV3 38589220.6.2001.5330 4.876.***
HNSC4 38589220.6.2002.5530 4.498.***

Definition of abbreviations: CPEA = certificate of presentation of ethical appreciation, IRB = institutional review board. 1HIAE = Hospital Israelita Albert Einstein, 2HMVSC = Hospital Municipal da Vila Santa Catarina Dr. Gilson de Cássia Marques de Carvalho; 3HMV = Hospital Moinhos de Vento; 4HNSC = Hospital Nossa Senhora da Conceição.

Table 2S Baseline Characteristics and Clinical Outcomes of the Included Patients (cont.) 

Overall HIAE1 HMVSC2 HMV3 HNSC4
Diagnosis at ICU admission - no. (%)
Cardiac 113 (15.6) 66 (14.5) 47 (17.5) - -
Endocrine 3 (0.4) 2 (0.4) 1 (0.4) - -
Gastric 69 (9.5) 9 (2) 60 (22.4) - -
Infectious 98 (13.6) 75 (16.5) 23 (8.6) - -
Neurologic 68 (9.4) 62 (13.6) 6 (2.2) - -
Oncologic 10 (1.4) 1 (0.2) 9 (3.4) - -
Orthopedic 38 (5.3) 38 (8.4) 0 (0) - -
Renal 24 (3.3) 12 (2.6) 12 (4.5) - -
Respiratory 120 (16.6) 84 (18.5) 36 (13.4) - -
Other 180 (24.9) 106 (23.3) 74 (27.6) - -
Body mass index (BMI)*, kg/m2, median (IQR) 25.4 (22.5-29.2) 26.3 (23.6-30) - - 24.0 (21-27)
Modified Frailty Score - no. (%)
0 374 (84.4) 374 (84.4) - - -
1 69 (15.6) 69 (15.6) - - -
Comorbidities - no. (%)
Asthma 12 (2.7) 12 (2.7) - - -
COPD 35 (7.9) 35 (7.9) - - -
Dementia 43 (9.7) 43 (9.7) - - -
Stroke 16 (3.6) 16 (3.6) - - -
Systemic arterial hypertension 198 (44.7) 198 (44.7) - - -
Severity of illness, median (IQR)
SAPS III score 51 (41-60) 50 (39-59) - - 52 (43-64)
SOFA score 4 (2-7) 4 (2-7) - - -
CCI, median (IQR) 2 (1-3) 2 (1-3) - - -
MFS, median (IQR) 2 (1-3) 2 (1-3) - - -
Duration of ventilation, days, median (IQR) 0 (0-1) 0 (0-0) - - 2 (0-7)
ICU length of stay, days, median (IQR) 3 (2-7) 3 (2-5) - 4 (3-6) 5.5 (3-13)
Hospital length of stay, days, median (IQR) 14 (8-26) 14 (8-26) - - -
Readmission, mean (SD) 1 (0.2) 1 (0.2) - - -
Discharge location at ICU - no. (%)
Ward 282 (39.1) 49 (10.8) 233 (86.9) - -
Step-down unit 195 (27) 195 (43) 0 (0) - -
Other hospital 20 (2.8) 13 (2.9) 7 (2.6) - -
Death 67 (9.3) 39 (8.6) 28 (10.5) - -
Other 158 (21.9) 158 (34.8) 0 (0) - -
Discharge location - no. (%)
Home 354 (75) 354 (79.7) 0 (0) - -
Home-care 7 (1.5) 7 (1.6) 0 (0) - -
Other hospital 12 (2.5) 12 (2.7) 0 (0) - -
Death 99 (21) 71 (16) 28 (100) - -
PMI, points, median (IQR) 1.6 (0-4.9) 1.5 (0-5) - 2.6 (0.5-5.7) 0.4 (0-2.5)

Data are median and interquartile range (quartile 25% - quartile 75%) or n (%). Percentages may not total 100 because of rounding.

Definition of abbreviations: CCI = Charlson comorbidity index; COPD = chronic obstructive pulmonary disease; ICU = intensive care unit; MFS = modified frailty score; PMI = perme mobility index; SAPS = simplified acute physiology score; SOFA = sequential organ failure assessment.

1 HIAE = Hospital Israelita Albert Einstein, São Paulo, Brazil; 2 HMVSC = Hospital Municipal Vila Santa Catarina, São Paulo, Brazil; 3 HMV = Hospital Moinhos de Vento, Porto Alegre, Brazil; 4 HNSC = Hospital Nossa Senhora da Conceição, Porto Alegre, Brazil.

* The body-mass index (BMI) is calculated by weight in kilograms divided by the square of the height in meters (Kg/m2). The categories are the same for men and women of all body types and ages, as follows: below 18.5 - underweight, 18.5-24.9 - normal or healthy weight, 25.0-29.9 - overweight, and 30 and above - obese.

† The SAPS III score ranges from 0 to 217, with higher scores indicating more severe illness and higher risk of death.

‡ SOFA scores range from 0 to 4 for each organ system, with higher aggregate scores indicating more severe organ dysfunction.

Table 3S Organ Support During ICU Stay 

Overall HIAE1 HMVSC2 HMV3 HNSC4
no. (%) 1200 (100) 455 (37.92) 268 (22.33) 305 (25.42) 172 (14.33)
During ICU stay - no. (%)
Non-invasive ventilation 61 (5.08) 61 (13.41) - - -
Invasive mechanical ventilation 183 (15.25) 62 (13.63) - - 121 (70.35)
Tracheostomy 12 (1) 12 (2.64) - - -
High-flow nasal cannula 8 (0.67) 8 (1.76) - - -
Renal replacement therapy 31 (2.58) 31 (6.81) - - -
Arterial catheter 109 (9.08) 109 (23.96) - - -
Bladder catheter 223 (18.58) 223 (49.01) - - -
Central venous catheter 135 (11.25) 135 (29.67) - - -
Transvenous pacemaker 4 (0.33) 4 (0.88) - - -
ECMO 1 (0.08) 1 (0.22) - - -
Use of vasopressor 133 (11.08) 133 (29.23) - - -

Data percentages may not total 100 because of rounding. Definition of abbreviations: ECMO = extracorporeal membrane oxygenation; ICU = intensive care unit. 1HIAE = ***; 2HMVSC = ***; 3HMV = ***; 4HNSC = ***.

Table 4S Perme Score Responsiveness for Sitting Mobility Milestone 

Cut-off Specificity Sensitivity
-16.5 1.18 100.00
-9.5 5.61 99.86
-8.5 6.20 99.73
-6.5 10.32 99.59
-5.5 12.98 99.32
-4.5 15.04 99.18
-3.5 16.22 98.91
-2.5 17.70 98.50
-1.5 20.94 97.82
-0.5 25.96 97.00
0.5 64.01 86.63
1.5 73.75 83.77
2.5 79.35 81.17
3.5 83.78 78.72
4.5 87.61 76.81
5.5 89.68 74.90
6.5 91.15 69.30
7.5 94.69 66.30
8.5 97.64 63.17
9.5 100.00 60.98

Table 5S Perme Score Responsiveness for Standing Mobility Milestone 

Cut-off Specificity Sensitivity
-8.5 5.83 100.00
-6.5 9.09 99.85
-5.5 11.42 99.70
-4.5 13.05 99.55
-2.5 15.38 99.24
-1.5 18.41 98.64
-0.5 22.61 97.88
0.5 54.55 87.14
1.5 63.40 84.72
2.5 69.46 82.75
3.5 73.89 80.64
4.5 78.55 79.27
5.5 82.52 78.06
6.5 86.25 73.37
7.5 89.74 70.05
8.5 93.24 67.32
9.5 96.50 65.43
10.5 97.90 62.93
11.5 98.37 58.70
12.5 98.60 56.28
13.5 99.07 54.31
14.5 99.53 49.55
15.5 100.00 42.51

Table 6S Perme Score Responsiveness for Walking Mobility Milestone 

Cut-off Specificity Sensitivity
-8.5 3.54 100.00
-4.5 8.22 99.80
-2.5 10.06 99.39
-1.5 12.32 99.19
-0.5 15.16 98.58
0.5 39.38 88.06
1.5 46.18 86.44
2.5 50.71 84.62
3.5 54.53 82.39
4.5 57.93 80.97
5.5 61.76 80.57
6.5 65.01 74.70
7.5 68.13 71.26
8.5 70.96 68.42
9.5 74.08 67.00
10.5 76.77 64.98
11.5 80.31 62.96
12.5 83.43 60.32
13.5 85.55 59.51
14.5 87.82 54.66
15.5 92.92 50.81
16.5 94.90 47.57
17.5 95.61 44.74
18.5 96.32 40.89
19.5 97.59 36.64
20.5 98.16 31.58
21.5 98.58 26.32
22.5 98.87 19.03
23.5 99.15 13.16
24.5 99.72 10.32
25.5 100.00 6.07

Figure 1S ROC Curve for Minimum Clinically Important Difference Establishment. Definition of abbreviations: AUC = area under the curve. 

Figure 2S Predictive Validity of the Perme Score for In-hospital Mortality 

Received: April 04, 2023; Revised: June 09, 2023; Accepted: September 21, 2023

Corresponding author: Ricardo Kenji Nawa, MSc PhD. Hospital Israelita Albert Einstein. 627/701 Albert Einstein Ave. São Paulo, SP - Brazil. ZIP: 05652-900. Tel: +55 11 2151-0603 | Fax: +55 11 2151-1000. Email: ricardo.nawa@einstein.br

Conflicts of Interest

The authors declare no conflicts of interest.

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License