SciELO - Scientific Electronic Library Online

 
vol.46 issue2Cannabis: A cognitive illusionDid Mozart suffer from Gilles de la Tourette syndrome? author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Indicators

Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google

Share


Revista Colombiana de Psiquiatría

Print version ISSN 0034-7450

rev.colomb.psiquiatr. vol.46 no.2 Bogotá Apr./June 2017

https://doi.org/10.1016/j.rcpeng.2017.05.009 

Original article

Validity and internal consistency of the Maslach Burnout Inventory in dental students from Cartagena, Colombia*

Validez y consistencia interna del Inventario Maslach para burnout en estudiantes de Odontología de Cartagena, Colombia

Miguel Angel Simancas-Pallares a   **  

Natalia Fortich Mesa b  

Farith Damián González Martínez a  

a Departamento de Investigación, Facultad de Odontología, Universidad de Cartagena, Cartagena, Colombia

b Área de Investigación, Programa de Odontología, Corporación Universitaria Rafael Núñez, Cartagena, Colombia


Abstract

Objective:

To determine the internal consistency and content validity of the Maslach Burnout Inventory-Student Survey (MBI-SS) in dental students from Cartagena, Colombia.

Materials and methods:

Scale validation study in 886 dental students from Cartagena, Colombia. Factor structure was determined through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Internal consistency was measured using the Cronbach's alpha coefficient. Analyses were performed using the Stata v.13.2 for Windows (Statacorp., USA) and Mplus v.7.31 for Windows (Muthén & Muthén, USA) software.

Results:

Internal consistency was α = 0.806. The factor structure showed three that accounted for the 56.6% of the variance. CFA revealed: x 2 = 926.036; df=85; RMSEA = 0.106 (90% CI, 0.100-0.112); CFI = 0.947; TLI = 0.934.

Conclusions:

The MBI showed an adequate internal consistency and a factor structure being consistent with the original proposed structure with a poor fit, which does not reflect adequate content validity in this sample.

Keywords: Factor analysis; Burnout; Scales; Validation studies

Resumen

Objetivo:

Determinar la consistencia interna y la validez de contenido del Inventario Maslach para burnout-Encuesta para estudiantes (IMB-EE) en estudiantes de Odontología de Cartagena, Colombia.

Material y métodos:

Estudio de validación de una escala en 886 estudiantes de Odontología de Cartagena (Colombia). Se determinó la estructura de factores a través de análisis factorial exploratorio y confirmatorio, y la consistencia interna, con el coeficiente alfa de Cronbach. El análisis estadístico se realizó empleando Stata v.13.2 para Windows (StataCorp., Estados Unidos) y Mplus v.7.31 (Muthén & Muthén, Estados Unidos).

Resultados:

La consistencia interna general fue α = 0,806. La solución factorial mostró tres factores que explican el 56,6% de la varianza. El análisis factorial confirmatorio mostró: X 2 =926,036; gl = 85; RCEMA = 0,106 (intervalo de confianza del 90%, 0,100-0,112); ICA = 0,947; ITL = 0,934.

Conclusiones:

El IMB muestra adecuada consistencia interna y una estructura factorial coherente con la propuesta original, pero con ajuste pobre, lo cual no refleja adecuada validez de contenido en esta muestra.

Palabras clave: Análisis factorial; Burnout; Escalas; Estudios de validación

Introduction

Burnout is a term that is historically used to refer to a syndrome characterised by emotional exhaustion and cynicism (development of indifference and/or aloofness towards other people), which often occurs in individuals who "work with people" in some way or another. One fundamental aspect of burnout syndrome (BS) is the increased perception of emotional exhaustion, and these two characteristics of BS are related to a certain extent. A third aspect of BS is the tendency to assess oneself in a negative manner, particularly with regard to working with others. Workers feel unhappy with themselves and their occupational achievements.1

Since healthcare professionals and dental students in particular are included in this category of human care, the intensity of these symptoms may be increased and can affect different aspects of their personal and professional lives. Mafla et al.2 determined in 2014 that 7% of dental students in Colombia meet the criteria for BS. Thus, measuring this construct becomes an important aspect for ensuring adequate psychological wellbeing.

The measurement of this construct was originally justified for professionals working in human care institutions. The instrument comprised 25 items and was structured around three main factors (emotional exhaustion, personal fulfilment and depersonalisation), with a fourth optional factor (participation).1 The fact that BS was initially only identified in human services workers is largely due to an artificial incident resulting from the universal use of the Maslach Burnout Inventory (MBI). Namely, this questionnaire can only be applied to the aforementioned population group due to the fact that it includes dimensions that are defined in terms of interactions with the survey respondents.

Even though the MBI was previously the most universally accepted instrument, there are currently other alternatives such as the Oldenburg Burnout Inventory (OLBI),3 the Burnout Measure (BM)4 and the Copenhagen Burnout Inventory (CBI).5 The OLBI has two subscales (exhaustion and disengagement) and is based on the argument that the MBI has psychometric limitations due to the fact that the items are all framed in the same direction (positive or negative). The BM was exclusively designed for measuring exhaustion. The items are worded in a general fashion and can thus be applied to any population group. However, studying the psychometric properties of this measure has revealed underlying theoretical issues.6 On the other hand, the CBI allows for context-free BS assessment (specific population group).

Due to growing interest in measuring this phenomenon and the early onset of common mental health disorders in young people, and taking into account that health sciences students dedicate at least 40-60% of their time and schedule to professional training and human care, respectively, which can lead to emotional exhaustion and thus a burnout, a version was designed for screening BS in students (MBI-Student Survey [MBI-SS]). This version contains 15 items with affirmative statements, the wording of which was modified in some cases to adapt it to the student context, and is available in various languages, including Spanish.7,8

In 2013, Rostami et al. assessed the psychometric properties of the MBI-SS among 238 female students from a university in Isfahan. Internal consistency was assessed with the Cronbach's alpha coefficient, according to the subscales, and revealed values between a = 0.84 and a = 0.90; factor structure determined through exploratory factor analysis (EFA) showed three factors: emotional exhaustion, cynicism and academic efficacy. The authors concluded that this instrument presented adequate validity and internal consistency for measuring the construct.9 Nevertheless, the study was only conducted on women, and its validity was determined using EFA. It is therefore necessary to report confirmatory factor analysis (CFA) in order to make suitable inferences regarding the content validity of this phenomenon.

Various research papers report the use of the MBI-SS for assessing BS in dental students in different languages. However, in Colombia there are currently no studies that describe psychometric properties including CFA of the MBI-SS in Spanish. As such, the objective of this study is to determine the content validity and internal consistency of the MBI-SS among dental students in Cartagena, Colombia.

Material and methods

Screening scale validation study with no reference criteria. Given that there are only three Dentistry faculties in Cartagena, which is considered a captive, geographic and spatially well-defined population, it was decided to perform a census. In this sense, the students of all three universities completed the instrument anonymously after familiarising themselves with the research objectives, taking into account the ethical considerations described in Resolution 008430 for health research in Colombia.

The sample of this study belongs to another broader study undertaken in Colombia in 2011: The STRESSCODE Study,2 led by the Cooperative University of Colombia (Pasto campus), whose protocol was approved by the university's Ethics Committee. However, the dental research committees of the three Dentistry faculties taking part in this study approved the protocol at a later time. Likewise, the participants agreed to participate by signing a written informed consent form.

Population characteristics

All undergraduate dental students from the three Dentistry faculties in Cartagena (Colombia) were invited to participate. The inclusion criteria were: students who agreed to participate voluntarily and who were registered and academically active in the second semester of 2011 and the first semester of 2012 during the conduct of the STRESSCODE Study.

The STRESSCODE Study is a multicentre study performed between 2011 and 2012 on 5647 dental students in Colombia. By applying a battery of tests, it revealed the prevalence of stress, psychological distress, burnout, overall self-regulation and related factors. However due to technical and research-related considerations, this study only describes the validity and internal consistency of the MBI-SS in Cartagena students who met the aforementioned inclusion criteria.

Instrument

The MBI is a scale comprising 15 items expressed in the form of statements that investigate the presence of BS-related elements, such as "I feel emotionally drained by my studies" or "I have become more cynical about the potential usefulness of my studies", measured on three factors: emotional exhaustion, cynicism and reduced academic efficacy. To apply the scale, the total time required is 5-8 min.

The frequency with which these aspects appear was determined by assessing the score of their Likert-type responses. They vary on a scale of 0 (never) to 6 (daily) and is considered dimensional given that there is no cut-off point. As such, scores may vary from 0 (lowest intensity) to 90 (highest construct intensity).

The Spanish language version used in this study derives from the version used in the STRESSCODE Study, to which a prior transcultural adaptation process was applied, comprising an English-Spanish translation followed by a Spanish-English translation by qualified scientific personnel.2

Statistical analysis

Descriptive statistics were initially performed. Proportions were calculated for the qualitative variables, while the mean ± standard deviation or median [interquartile range] were calculated for quantitative variables, depending on the distribution symmetry. Internal consistency was estimated with the Cronbach's alpha coefficient overall, as well as in each of the subscales (domains).

In order to assess content validity, the factorability of the matrix was determined with Bartlett's test of sphericity and the Kaiser-Meyer-Olkin measure of sampling adequacy. Subsequently, EFA was performed to reveal the factor structure. The number of factors to be extracted was determined using the values themselves and a Cattell sedimentation graph. To define factor conformity, factor loads with a value of >0.40 were taken into account. Oblique rotation (promax) was also used, given that it was thought that the potential factors would be correlated with each other.

To proceed with CFA, multivariate normality was verified using Mardia's test. Although multivariate normality was not detected, CFA is robust for these types of scenario when there are large sample sizes (>200) and no multivariate normality.10 CFA was performed to confirm the factor structure previously determined in the EFA, and thus the content validity through Partial Least Squares with Mean and Adjusted Variance (PLSMAV). To assess the model fit, the following statistical measures were determined: x 2, degrees of freedom (df), root mean square error of approximation (RMSEA) and confidence interval of 90% (90% CI), comparative fit index (CFI) and Tucker-Lewis index (TLI). These measures were assessed (model fit) using the criteria proposed by Hu and Bentler: p >0.05, RMSEA < 0.06, CFI > 0.95 and TLI > 0.95.11

The descriptive statistical analysis and EFA were performed on the Stata package, v.13.2, for Windows, (StataCorp., College Drive Station, Texas, USA); CFA was performed using Mplus v.7.31 for Windows (Muthén & Muthén, Los Angeles, California, USA).

Results

The MBI-SS was applied to a total of 886 subjects; of these, 59.3% were female, 44.1% were undertaking semesters in the clinical academic year, the majority of whom belonged to stratum 3, and 54.39% came from a private university. The mean age was 20.42 ±2.8 years. Fig. 1 depicts the study entry flow diagram. The study response rate varied from 88% to 100% (mean: 92.5%), while the MBI-SS scores were between 0 and 90. The highest scores on the scale corresponded to means of5.11 and 4.93, which were found under the "I have learned many interesting things during the course of my studies" and "I feel stimulated when I achieve my study goals" items. The lowest were found under the "I doubt the significance of my studies" and "I have become more cynical about the potential usefulness of my studies" items, with a mean of 1.23. The scores obtained in each domain of the scale are shown in Table 1 By domain, the lowest score corresponded to cynicism.

Fig. 1 Study flow chart. CFA: confirmatory factor analysis. 

Table 1 Descriptive statistics of each of the reactive components of the MBI-SS scale (Cartagena de Indias, February-August 2011, surveyed students). 

Content validity

Bartlett's test of sphericity, applied in the EFA, showed x 2 = 6924.232, df=105 and p < 0.001; the Kaiser-Meyer-Olkin measure of sampling adequacy was: 0.8529. Factor structure indicates three factors: factor I was named "emotional exhaustion" and showed an eigenvalue of 3.96, which explained the 26.41% total variance; factor II, named "cynicism", reached an eigenvalue of 3.45, accounting for the 23% total variance; and factor III, named "reduced academic efficacy", had an eigenvalue of 1.08, thus explaining the 7.26% total variance. Table 2 how is the factor solution obtained.

Table 2 Exploratory factor analysis (EFA) results. Item and uniqueness loads (Cartagena de Indias, February-August 2011, surveyed students). 

The CFA found the following fit indices: x 2 = 1137.38, df =87, RMSEA = 0.117 (90% CI, 0.111-0.123), CFI = 0.934 and TLI = 0.920. Nevertheless, after applying two modification indices, the following fit indices were obtained: x 2 = 926.036, df=85, RMSEA=0.106 (90% CI, 0.100-0.112), CFI=0.947 and TLI=0.934.

Internal consistency

Overall, the scale found a = 0.806 (95% CI, 0.787-0.819). By factors, the measures were represented as follows: factor I, 0.839 (95% CI, 0.825-0.854); factor II, 0.859 (95% CI, 0.839-0.866), and factor III, 0.871 (95% CI, 0.850-0.8744).

Discussion

Various studies have used the Spanish version of the MBI-SS to quantify BS, indicating a prevalence that ranges from 7% to 28%.2,12,13 However, few studies have explored its validity and internal consistency in Spanish and for populations in Colombia.2 In this sense, the results of this study provide the first ever approach to the content validity and internal consistency of this scale, in Spanish, among dental students from Cartagena, Colombia.

In validation studies, it is necessary to take population characteristics into account, especially when the test is applied to the general population; the age and gender of the participants could act as a barrier to generalising the results obtained,14 due to the fact that the risk of suffering from BS may vary according to these sociodemographic characteristics. This is one of the main reasons for developing the MBI-SS, given that the context to which the general version of the scale was traditionally applied was unsuited to students and implied measurement bias.

In this sense, the results of the study are only applicable to students registered at the three dentistry faculties of the city of Cartagena undertaking semesters in the clinical academic year, from stratum 3, for the most part, and private universities, which limits the application of the instrument to other health areas or to populations with characteristics that differ from those of the sample included here.

The MBI-SS is easy to apply due to the lower number of questions, and the scale used for construct quantitation facilitates quick and reliable measurement. These questions explore the major elements that may define the presence of BS, which is reflected by an adequate internal consistency according to Cronbach's alpha coefficient, calculated overall as well as for the three domains proposed, thus also agreeing with other studies that have validated the MBI-SS in other languages.10 It is important to take into account that reliability is a necessary step for initiating the validity estimation process, but a reliable instrument is not necessarily valid and, in turn, a valid instrument is not useful if it is not sufficiently reliable, thereby reflecting the need to estimate internal consistency in validation studies in order to express interrelation between construct items.15-17

The results of this study show that the MBI-SS on undergraduate dental students in Cartagena offers a duplicable factor structure compared to the version that was originally proposed1,11,18 and studies on the content validity of this scale in other languages, such as Portuguese.19 The three factors found explain more than half of the variance; this may assert that the construct is present in the items included in the scale. This process is important for gathering data with adequate validity and reliability, and must be performed before beginning any study. However, on evaluating the fit indices provided, the CFA revealed that the model presents poor fit. In this sense, it is necessary to continue investigating some items that in this population do not completely reflect the nature of the construct despite the fact that its factor structure was successfully duplicated in the exploratory factor analysis.

A research study in Isfahan to determine the validity of the Persian version of the scale showed a three-factor structure, but with two fewer items in the cynicism factor, which were included under the emotional exhaustion factor. However, these authors did not perform CFA, which is an essential approach to determine the validity of a scale.9,20 This discrepancy between the findings can also be explained by the fact that this study only included female students with different academic levels (masters and doctorates), which may signify greater sources of variance in construct measurement.

The justification behind the inclusion of modification indices (MI) in the CFA models is that, by identifying "areas of tension" in the model, statistical strategies may be applied to improve the content validity of the instrument, which is reflected in the improvement of the model fit indices. The modification made to the model is based on the similar wording of the "I feel emotionally drained by my studies" and "I feel exhausted at the end of a day at university" items; moreover, due to belonging to the same domain, these items may share the error covariance. Moreover, this phenomenon could also be observed in the "I have become more cynical about the potential usefulness of my studies" and "I doubt the significance of my studies" items. The usefulness of including the MI may thus be reflected in the statistically significant improvement obtained for the model on comparing the two models in CFA.10,20

In this sense, in the case of the MBI-SS, two MIs were applied, resulting in a statistically significant improvement in the model fit; however, the scale still presents a poor fit with respect to its content validity. These findings do not match those reported in other studies, which indicate a good fit on employing CFA techniques.9 The differences lie fundamentally in the criteria used to assess the model fit obtained, which note a CFI value of >0.90 and a RMSEA of <0.10.21,22 Nevertheless, in this research study, stricter assessment criteria were used that were universally consistent with statistical scientific literature on CFA and structural equation modelling, which consider model fit "acceptable" if the RMSEA is <0.06, x 2 with p>0.05 and CFI and TLI >0.95. Although the model had poor fit, the results of the x2 test are expected to be significant due to the fact that its use in CFA techniques is sensitive to sample size; on increasing the sample size, this will tend to be significant. In light of this, other measures of fit, such as the CFI, RMSEA and TLI, must be taken into account for assessing the model.6

Measuring BS is important as it permits the early detection of potential cases, thereby attenuating its serious consequences on physical and mental health. In this sense, historically, few instruments have been proposed for measuring this construct, so the MBI is the most used model; more than 90% of the world's research on burnout has been performed with this instrument.1 Although various versions have been proposed, including the MBI-SS, with the purpose of improving the detection of the construct in different population groups, numerous authors have noted the marked shortcomings of the instrument's design, which may compromise its results when applied to populations other than the one for which it was originally developed.2 Exhaustion is the central aspect of this syndrome and the MBI, on considering the other two factors, erroneously characterises the concept of burnout given that these dimensions represent a coping strategy and consequence of the syndrome, respectively. This may partially explain the poor fit observed on assessing content validity. Although these hypotheses stem from research studies with the original instrument for professionals, it must be taken into account that the student version is an adaptation.

This created the need to obtain new instruments for measuring BS, such as the Copenhagen Burnout Inventory3 and the one-dimensional student burnout scale.23 The former is intended for measuring the construct among professionals, while the latter is for students (originally designed in Spanish). The one-dimensional student burnout scale still has no scientific evidence on its content validity, but EFA results show that there are two diverging dimensions with respect to the MBI-SS: behavioural indicators and attitude indicators.

These findings highlight the importance of performing more in-depth studies on the content validity of the MBI-SS and of exploring other instruments for this population that may reflect adequate content validity. Future efforts will have to be made to improve this aspect of its psychometric performance, and to thus improve the quality of the measurements performed with the instrument and, as a result, the possibility of generalising the results derived from such research.

Acknowledgements

To the undergraduate students belonging to the research groups, who generously provided their assistance in the data collection process. To the management boards of the University of Cartagena for the academic and administrative support provided during collection of data for this research.

References

1. Maslach C, Jackson SE. The measurement of experienced burnout. J Occup Behav. 1981;2:99-113. [ Links ]

2. Mafla AC, Villa-Torres L, Polychronopoulou A, Polanco H, Moreno-Juvinao V, Parra-Galvis D, et al. Burnout prevalence and correlates amongst Colombian dental students: the STRESSCODE study. Eur J Dental Educ. 2015;19:242-50. [ Links ]

3. Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. The job demands-resources model of burnout. J Appl Psychol. 2001;86:499-512. [ Links ]

4. Pines A, Aronson E. Career burnout: causes and cures. 1st ed. New York: Free Press; 1988. [ Links ]

5. Kristensen TS, Borritz M, Villadsen E, Christensen KB. The Copenhagen Burnout Inventory: a new tool for the assessment of burnout. Work Stress. 2005;19:192-207. [ Links ]

6. Schaufeli WB, Van Dierendonck D. The construct validity of two burnout measures. J Organ Behav. 1993;14:631-47. [ Links ]

7. Salanova M, Schaufeli W. Exposure to burnout and its relationship to burnout. Behav Inform Technol. 2000;19:385-92. [ Links ]

8. Schaufeli WB, Salanova M, González-Romá V, Bakker AB. The measurement of engagement and burnout: a two sample confirmatory factor analytic approach. J Happiness Stud. 2002;3:71-92. [ Links ]

9. Rostami Z, Reza Abedi M, Schaufeli W,Ahmadi A, Sadeghi AH. The psychometric characteristics of Maslach Burnout Inventory Student Survey: among students of Isfahan University. Zahedan J Res Med Sci. 2013;15:29-32. [ Links ]

10. Brown T. Confirmatory factor analysis for applied research. 2nd ed. New York: Guilford Press; 2015. [ Links ]

11. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Eq Model: Multidisc J. 1999;6:1-55. [ Links ]

12. Bastidas CdRC, Ceballos OOD, Delgado LO. Síndrome de burnout en estudiantes de pregrado de la Universidad de Nariño. Rev Electr Psicol Iztacala. 2011;14:223. [ Links ]

13. Rísquez MIR, Garcia CC, Tebar EDLAS. Resilience and burnout syndrome in nursing students and its relationship with sociodemographic variables and interpersonal relationship. Int J Psychol Res. 2012;5:86. [ Links ]

14. Campo-Arias A, Miranda-Tapia GA, Cogollo Z, Herazo E. Reproducibilidad del Índice de Bienestar General (WHO-5 WBI) en adolescentes estudiantes. Rev Cient Salud Uninorte. 2015:31. [ Links ]

15. Streiner DL. A checklist for evaluating the usefulness of rating scales. Can J Psychiatry. 1993;38:140-8. [ Links ]

16. Campo-Arias A, Herazo E, Oviedo HC. Análisis de factores: fundamentos para la evaluación de instrumentos de medición en salud mental. Rev Colomb Psiquiatr. 2012;41:659-71. [ Links ]

17. Oviedo HC, Campo-Arias A. Aproximación al uso del coeficiente alfa de Cronbach. Rev Colomb Psiquiatr. 2005;34:572-80. [ Links ]

18. Schaufeli WB, Martinez IM, Pinto AM, Salanova M, Bakker AB. Burnout and engagement in university students. A cross-national study. J Cross-Cult Psychol. 2002;33:464-81. [ Links ]

19. Carlotto MS, Câmara SG. Características psicométricas do Maslach Burnout Inventory-Student Survey (MBI-SS) em estudantes universitários brasileiros. Psico-USF. 2006;11:167-73. [ Links ]

20. Harrington D. Confirmatory factor analysis. Oxford: Oxford University Press; 2008. [ Links ]

21. Byrne BM. Structural equation modeling with AMOS: basic concepts, applications, and programming. New York: Routledge; 2013. [ Links ]

22. Marôco J. Análise de equacões estruturais: fundamentos teóricos, software & aplicacões. Sao Paulo: Report Number; 2010. [ Links ]

23. Macías AB. Validación psicométrica de la escala unidimensional del burnout estudiantil. Rev Intercont Psicol Educ. 2011;13:51-74. [ Links ]

* Please cite this article as: Simancas-Pallares MA, Fortich Mesa N, González Martínez FD. Validez y consistencia interna del Inventario Maslach para burnout en estudiantes de Odontología de Cartagena, Colombia. Rev Colomb Psiquiat. 2017;46:103-109

Ethical disclosures

Protection of human and animal subjects. The authors declare that the procedures followed were in accordance with the regulations of the relevant clinical research ethics committee and with those of the Code of Ethics of the World Medical Association (Declaration of Helsinki).

Confidentiality of data. The authors declare that they have followed the protocols of their work centre on the publication of patient data.

Right to privacy and informed consent. The authors have obtained the written informed consent of the patients or subjects mentioned in the article. The corresponding author is in possession of this document.

Conflicts of interest The authors have no conflicts of interest to declare.

1 Kristensen TS, Borritz M, Villadse E, Christensen KB. The Copenhagen Burnout Inventory: a new tool for the assessment of burnout. Work Stress. 2005;19:192-207.

2 Kristensen TS, Borritz M, Villadse E, Christensen KB. The Copenhagen Burnout Inventory: a new tool for the assessment of burnout. Work Stress. 2005;19:192-207.

3 Kristensen TS, Borritz M, Villadse E, Christensen KB. The Copenhagen Burnout Inventory: a new tool for the assessment of burnout. Work Stress. 2005;19:192-207.

Received: July 10, 2015; Accepted: February 28, 2016

** Corresponding author. E-mail address: msimancasp@unicartagena.edu.co (M.A. Simancas-Pallares). http://dx.doi.org/10.10167j.rcpeng.2017.05.009

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License