What do we know about this topic?

The cognitive load theory states that working memory has limited capacity and hence the need to minimize the load received by learners during the learning process, in order to render the process more effective. Cognitive load can be intrinsic if associated with the design of the task performed, or extrinsic, if related to external factors pertaining to individual learner context. Germinal load is the work required exclusively for learning. When both intrinsic and extrinsic loads are low, more working memory is available for germinal load, optimizing the learning process.

What is new about this study?

It allows to improve learning activities in order to achieve better alignment between objectives, activity performance and medical competency evaluation. This is so because intrinsic load contribution in simulation workshops is unknown. Many factors are involved in medical skill learning. Some are directly related to the methodology used (intrinsic load), while others depend on each learner's individual context (extrinsic load).

INTRODUCTION

The cognitive load theory states that working memory has limited capacity, hence the need to minimize the load received by learners during the learning process, in order to render the process more effective. Cognitive load may be intrinsic if associated with the design of the task at hand, or extrinsic if related to external factors associated with each learner's individual context. Germinal load is the work required exclusively for learning. When both intrinsic and extrinsic loads are low, more working memory is available for germinal load, optimizing the learning process ^1-3 (Figure 1).

Source: Adapted with permission from Van Merrienboer and Sweller ⁴.

Figure 1 Cognitive load composition. 

Several tools are currently available to improve learning in medicine, simulation workshops being among the most widely used. Simulation workshops are used in different clinical subject areas offered by the Universidad de los Andes School of Medicine as a method for teaching basic clinical skills.

Two main tools are used in the context of cognitive load evaluation: the first is the National Aeronautics and Space Administration - Task Load Index (NASA-TLX) ⁵ and the second is the Paas Scale ⁶. The NASA-TLX is more useful for tasks with psychomotor characteristics. It consists of six items which explore mental, physical and temporal demands, performance, effort and fatigue. Each item is assessed using a 12 cm visual scale to obtain a score between zero and twenty. That score is then multiplied by five to obtain a final score between zero and one-hundred for each item ⁷. The student is asked to rate each of the items using a visual analogue scale. on the other hand, the Paas scale is preferred for predominantly cognitive tasks and it consists of a single item that rates the mental effort invested in the task, using a score from one to nine ⁸.

Measuring cognitive loads associated with simulation activities allows to evaluate the types of activities students find difficult to learn in order to modiffy the design of those activities and facilitate learning in the future. The objective of this study was to compare cognitive loads between the different simulation activities, including anesthesiology and surgery simulation workshops in medical students.

METHODS

This cross-sectional analytical observational study used the original NASA Task Load Index (TLX), developed and validated in 1988 by Hart and Staveland ⁵ and the Paas scale developed and validated in 1992 ⁹ as tools to measure cognitive load (annex 1.0). Additional questions of relevant demographic interest (sex, age and rotation order) were included. This tool was submitted in writing and completed by each student individually, anonymously and voluntarily after having read and signed the informed consent that accompanied the scale. The scales were self-administered under the supervision of any of the project researchers and documented in a hard printed form. This research protocol was submitted to the Research Ethics Committee of Universidad de los Andes and approved on May 27, 2019 (Minutes 1009-2019).

The tools were administered immediately after each of the four simulation academic activities (2 in anesthesia and 2 in surgery), allowing no less than 15 minutes for reading and completion. One of the researchers was always present to clear doubts or answer questions from the participants. The scales were administered in English (with the approval of the Ethics Committee) considering that they are not currently validated in Spanish. Despite the fact that Spanish is the native language of the study population, all of the students met the English requirement of Universidad de los Andes and had no problem interpreting or understanding the tools. Each participant was assigned a code in order to protect them from risk of disclosure of their personal information.

The workshops designed for the surgery and anesthesia specialties were selected for this study because these subject areas are attended in the same academic term by Universidad de los Andes medical students (eighth semester). The workshops attended by the students were the following: venoclysis (practice of venoclysis in human subjects), airway (teaching of airway devices and orotracheal intubation in simulator), sutures (different suturing techniques in skin simulators) and urinary catheters (types of catheters, indications and placement of bladder catheter in a simulator). All of the workshops were the same for all subjects in terms of content, methodology and instructors. Figure 2 illustrates the way in which the tool was administered to the students, and the two groups were divided according to the timing of each workshop.

Source: Authors.

Figure 2 Flow diagram illustrating the administration of the scales according to student order of rotation (route 1 and route 2). 

Sample size was not calculated because there are no studies in the scientific literature exploring this topic in similar populations in Bogotá or in Colombia, resulting in lack of knowledge regarding cognitive load levels in simulation activities performed by medical students. Given the total absence of prevalence descriptions of the study condition, this work was considered to be exploratory. For this reason, convenience sample size was used with all the students enrolled for the eighth semester during the second term of 2020, for a total of 66 participants.

Potential biases were addressed during the design stage of this study. Courtesy bias was controlled by ensuring anonymity and confidentiality of the responses. Information bias was mitigated through the inclusion of participants who volunteered freely under no form of coercion. Selection bias was controlled by evaluation of the same students attending the same four workshops during the same period. Moreover, in order to diminish recall bias, the scales were applied within the 15 minutes following the end of each workshop. Finally, in order to control for confounding factors, the workshops were taught by the same instructor and included the same content and methodology.

After they were collected, data were tabulated in Microsoft Excel. Data analysis was carried out using the SPSS Statistics software package, version 24. The answers to the questionnaire were entered in a database using a double entry data monitoring system. Initially, one researcher would enter the data and would identify erroneous or outlying values. Another researcher would enter the same data, identifying errors and outliers. Both entries would be compared by a third researcher in order to identify and confirm anomalous data. In the event such anomalous data were found, the physical source of information would be reviewed (scale completed by the participant) in order to determine the correct values.

Categorical variables were described as frequencies with percentages for the total population and subgroups, while continuous variables were described as arithmetic means and standard deviation. Comparison of the scores obtained in each of the scales between the study groups was carried out using the non-parametric student T test, the Mann-Whitney test and Fisher's exact test. The Kolmogorov-Smirnov test was used to check for normality of each of the variables.

RESULTS

The study population consisted of 66 eighth-semester students, 51.5% females and 48.5% males, with a mean age of 21.6 years. Thirty-two students followed route 1 and 34 students followed route 2 (Figure 2). Of this population, 52 students were included in the final analysis, because of missing data in the remaining students (non-attendance to the workshop or failure to complete the collection tool because of unwillingness to be part of the study).

In terms of the Paas scale, differences were found between the urinary catheter and sutures workshops (p = 0.04), between the urinary catheter and airway management workshops (p = 0.002) and between airway management and venoclysis workshops (p = 0.011). In these cases, the workshops that required the greatest mental effort were airway management (mean 5.83) and sutures (mean 5.5), as compared to venoclysis (mean 5.25) and urinary catheter insertion (mean 5.02).

There were no differences on the Paas scale in terms of mental effort for the rotation order at the time of attending the workshops, except for the airway management workshop, in which the students who rotated first in surgery and then in anesthesia reported a higher cognitive load than those who rotated first in anesthesia and then in surgery (6.19 vs. 5.53, p = 0.029). Differences in the Paas scores for the 4 workshops broken down by rotation order are shown in Table 1.

The Nasa-TLX scale assessed physical, mental and time demands on the students, as well as effort, frustration caused by the activity and self-perceived performance. The workshop that elicited less frustration among the students was urinary catheter insertion (mean 28.37), while the highest level of frustration was experienced in the airway management workshop (mean 43.75). Significant differences were identified in student-perceived physical effort in the venoclysis workshop - when comparing the two routes - given that physical effort was greater among the students who rotated in anesthesia before doing so in surgery (59.38 vs. 42.68, p = 0.016).

In the airway management workshop in particular, significant differences were found between the NASA-TLX items relating to mental demand and frustration (p = 0.022), time demand and effort (p = 0.004), frustration and performance (p = 0.026), and effort and frustration (p = 0.000). In the venoclysis workshop, the other workshop included in anesthesia rotation, significant differences were found in mental demand and performance (p = 0.012), mental demand and frustration (p = 0.013), physical demand and performance (p = 0.037), physical demand and frustration (p = 0.004), time demand and performance (p = 0.000), time demand and effort (p = 0,015), performance and frustration (p = 0.000), and effort and frustration (p = 0.000).

When comparing the catheter insertion and airway management workshops, significant differences were found in all the items of the NASA-TLX scale, except for time demand of each of the workshops (p = 0.168). Scores were higher in all items for the airway workshop, except for performance, which was perceived to be higher in the catheter insertion workshop.

Venoclysis and suture workshops were also compared. In this case, the highest scores were found for the suture workshop, reflecting greater cognitive load. In terms of performance during the workshop, this relation is inverted, with a higher score obtained for the venoclysis workshop. Physical demand was the only item with no significant differences in this case (p = 0.061).

DISCUSSION

This study documents factors potentially implicated in intrinsic (modifiable and related to workshop structure and content) and extrinsic (non-modifiable and related to the personal context) cognitive loads, which could have influenced student performance during the workshops.

Regarding the order of rotation at the time of attending the workshops, it was found that the students who followed route 1 had a higher cognitive load (Paas score) than the students who followed route 2 in the airway workshop. It is tempting to think that increased cognitive load and mental effort measured by the Paas scale at the end of the term in the airway workshop could be due to greater academic pressure derived from the need to obtain certain grades to pass the semester, or to student burnout at the end of the study cycle ¹⁰. In contrast, it was found that the students who followed route 2 perceived that physical effort (NASA-TLX) was greater in the venoclysis workshop, compared to those who followed route 1. During surgery rotation, students acquire different technical and manual skills that can make it easier to acquire the skills needed to perform venoclysis later on, perhaps explaining this result. This is similar to the finding by Aldekhyl et al. ¹¹ and Haji et al. ¹², who determined that expertise reduces cognitive load in simulation settings.

Differences related to rotation order could be explained on the basis of the intrinsic load, given that the workshops had the same design in terms of content and methodology, maintaining a stable extrinsic cognitive load. Academic pressure at the end of the term could increase the intrinsic load, while prior surgical experience could reduce it, meaning that conditions were not equal for the groups at the time of performing the academic simulation tasks, as assumed by the curriculum design. This is to say that prior experience, either negative or positive, as well as the timing of the workshops, could play an important role in learning in simulation settings ¹³.

In terms of cognitive load differences among workshops (NASA-TLX scale), the workshop associated with the highest frustration rate was the airway workshop, while the one associated with the lowest frustration was catheter placement. Additionally, a higher cognitive burden was observed for the physical and mental demand and effort items in the airway workshop compared to the urinary catheter workshop, with higher performance being perceived in catheter insertion. This is comparable with the study by Ambardekar et al., where high levels of cognitive load were found in difficult airway simulation settings of the American Society of Anesthesiologists (ASA) ¹⁴. These differences can be attributed to higher levels of intrinsic load in orotracheal intubation and other tasks associated with airway management when compared to catheter insertion ¹⁵. They can also be attributed to the fact that students are under higher pressure when performing tasks related to airway management because the implications of acquiring or failure to acquire the necessary knowledge in this area can be critical for prognosis and morbidity and mortality in real patients later on in their professional lives.

Similarly, Fraser et al. found that increased arousal and reduced relaxation increased cognitive loads in cardiac murmur simulation settings ¹⁶. Also, airway management occurs in situations of care for critically ill patients and has a component of diligence and speed that has a direct influence on outcomes, something that does not happen with urinary catheter insertion ¹⁵.

Finally, in the comparison between the suture and venoclysis workshops, no differences were found in terms of physical effort reported on the NASA-TLX scale. Scores on mental effort, time demand and frustration items were significantly higher in the suture workshop, with lower scores for performance perception being found also in the suture workshop. This might be explained by the multiple components of this workshop which involves making different types of sutures of variable degrees of difficulty in varying materials, increasing the intrinsic load of the workshop. This explanation is consistent with the finding by Haji et al., who showed that simulation groups with simple tasks carried out in stages had a higher performance than those involving complex tasks ¹⁷.

A strength of this study was the use of both the Paas and the NASA-TLX scales, considering that they each assess different components of the construct. The NASA-TLX assesses tasks with a major psychomotor component ⁷, while the Paas assesses predominantly cognitive tasks ⁸. Another strength of this study was the comparison of the different workshops as well as the determination of differences in cognitive load according to the route, considering that it is well known that previously acquired knowledge is a fundamental component of the intrinsic load ^18,19.

In terms of limitations, the scales used for measuring cognitive load in the various learning activities do not distinguish between the intrinsic and extrinsic load component, limiting the analysis of the reasons why load can be greater in one group as compared to the other. On the other hand, the questionnaires used have not been validated in Spanish and were given in English, their original language, limiting understanding of the tool; however, the entire study population met the English knowledge requirements of Universidad de los Andes, and the risk was considered low given that the questionnaires had simple wording and a numerical component. Finally, unlike what had been determined at the start of the study, data were entered by a single researcher who then verified by comparing with the paper questionnaires to ensure that the data were correct. It was done in this way because the researcher new the database better, reducing the possibility of errors derived from editing by several researchers.

This study found that the order in which a student participates in the different rotations has an effect on cognitive load and, therefore, the way the student learns. Consequently, it could be important to assess cognitive load in advance and build the curriculum with a rotation order aimed at facilitating student learning. It was also found that the airway workshop had the highest cognitive load, which might be explained by the multiple technical tasks involved in airway management, apart from previous knowledge of the anatomy and of the various devices to be used. Strategies should be created to reduce the cognitive load associated with this workshop.

Evaluating academic activities on the basis of cognitive load is a complex procedure and is part of one of the areas of research into modern medical education. It requires analysis and evaluation of learning tasks in clinical areas of medicine and of the mechanisms to attain curricular goals through the approach to different domains that lend themselves to research ²⁰. Academic activities are not formally evaluated on a routine basis, and feedback is not gathered from the students. Learning goals established in the curriculum must be attained by the students and, at the same time, students must be allowed to acquire knowledge and integrate that knowledge for future performance in their professional lives ²¹. Hence the need for institutions and faculty to conduct a quantitative and qualitative assessment of the learning process, in such a way that attainment of academic goals established in the curriculum can be verified.

Future studies could interview students about their qualitative perception of the workshops they attend. This would shed light on the most salient hurdles to learning and on the way to mitigate them, which has been shown to contribute to curricular improvement ²². Also, questionnaires or focus groups in which students can actually participate in the improvement of academic learning activities are a means to enhance the relationship between the academic objectives and the learning process that really occurs during these types of activities ²³. Finally, it is of crucial importance to approach the factors that can influence intrinsic and extrinsic cognitive loads associated with simulation workshops, in order to modify those factors that influence learning ⁸.

ETHICAL RESPONSIBILITIES

ACKNOWLEDGEMENT