English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Cited by Google 
Similars in
SciELO 
Similars in Google 
Permalink
I. Introduction
The analysis and study of human emotions is a complex field of research that can be approached from different points of view depending on the mechanisms employed for the stimulation, measurement, and analysis of emotional reactions in the human body. Currently, video games are among the tools with the greatest potential to stimulate the emotions of an individual; their strength lies in immersing the player in a virtual world, where they can experience different emotional reactions with no need to move large distances, engaging their attention, concentration, and time, in order to achieve the goals established by the immersive story of the game. Video games use stories, challenges, objectives, characters, scenarios, music, animation, and other audio-visual multimedia tools to capture the players' attention, and generate emotions that allow the individual to feel identified with the game, and thus, promoting a more outstanding game experience for those who would like to take up the challenge and use their time for this kind of activities. Along with books, music, images, and movies, videogames constitute a group of audio-visual tools that generate emotional reactions in individuals, according to the level of empathy they generate in the person. However, in contrast with the previous examples, video games represent a greater source of information on reaction, given that they are essentially interactive 1.
Apart from being one of the largest industries of technological development nowadays, video games have become a great source of research in applications oriented toward entertainment, as well as in applications known as "Serious Games," which try to solve problems in fields such as medicine [2the new Medicine degrees of the European Higher Education Area (EHEA, 3], the military 4, education 5,6, business 7, politics, citizenship [8], etc. Throughout the last decade, the convenience and importance of using video games or "Serious Games" have been demonstrated by analyzing the response of the players and potentiating some of their skills, simulating different virtual environments, and objectives that involve the direct participation of the people with the digital world. This study was carried out starting from the premise that a videogame can generate empathy in the player, producing different emotional reactions during gameplay, and that these reactions can be captured, measured, and analyzed through emotion recognition techniques and affective computing 9, such as face recognition 10, speech recognition 11, and physiological signals 12,13. The objective was to identify evidence and current research that integrate techniques of emotion recognition, involving physiological signals, such as electrocardiography (ECG), Galvanic skin response (GSR), and electromyography (EMG), with the use of video games for stimulating emotions; for this, we conducted a systematic review between 2010 and 2016, in which we identified a growth in the number of articles focused on video games and emotion recognition systems, demonstrating the work that is developed in the field.
II. Materials and methods
A. Selection criteria
The articles were selected from eight different bibliographic and academic databases related to engineering and medicine: ScienceDirect, IEEEXplorer, SCOPUS, EngineeringVillage, ISI Web of knowledge, ACM, EMBASE, and PubMed. The articles corresponded to studies focused on emotion recognition from the measurement of physiological signals (ECG, GSR, or EMG), and the use of video games as tools for emotion stimulation, published between 2010 and 2016, due to the growth of these approaches employing these techniques.
B. Search strategy
The search and selection of articles that deal with developments in the mentioned field was based on two searches using different terms that refer to the field of study.
In the first search, the selected works contained in their title, abstract, or key words, the terms "emotion recognition" and "physiological signals", and were published between January 2010 and April 2016. The search was carried out in the databases previously mentioned and, as a result, the following number of papers was obtained: ScienceDirect (14), IEEEXplorer (162), SCOPUS (144), EngineeringVillage (125), ISI Web of knowledge (291), ACM (7), EMBASE (16), and PubMed (13). Given that we were looking for studies that used video games as emotion stimulation tools, the previously search was complemented including the word "games" in any section of the article. The new search was conducted in the same eight academic databases and produced the following results: ScienceDirect (4), IEEEXplorer (7), SCOPUS (19), EngineeringVillage (8), ISI Web of knowledge (7), ACM (4), EMBASE (0), and PubMed (0), obtaining 49 articles in total.
A second search was carried out to complement the results obtained from the first search. This time, the objective was to identify works published between January 2010 and April 2016 that contained in their title, abstract, or key words the term "affective gaming." This term refers to video games that interact with the player through commands or actions using specialized tools, and respond or modify according to the emotions experienced by the player 14. The consulted databases produced the following results: ScienceDirect (121), IEEEXplorer (196), SCOPUS (27), EngineeringVillage (27), ISI Web of knowledge (16), ACM (451), EMBASE (1), and PubMed (15). Then, along with the previous search, the term "physiological signals" was included to be found in any section of the article; this search produced the following results: ScienceDirect (23), IEEEXplorer (14), SCOPUS (10), EngineeringVillage (5), ISI Web of knowledge (3), ACM (7), EMBASE (1), and PubMed (1), obtaining 64 articles in total.
C. Selection study
In total, 113 articles were found in the consulted databases (49 in the first search and 64 in the second search) containing the stated selection criteria. After excluding the duplicated papers, the number of articles was reduced to 74. Out of them, 46 were available in full. Finally, the accepted articles were classified according to the type of physiological signals measured, and the tool used to stimulate the individuals' emotions. 15 articles reported to have measured physiological signals such as ECG, EMG, or GSR, along with video games as a stimulation tool for emotion recognition, and described the results of this process.
D. Data analysis
The search was conducted to determine state-of-the-art techniques of emotion recognition based on physiological signs and video games. Of the 46 articles, 50 % (23) used video games as emotion stimulation tools, 10.87 % (5) used videos or movies as emotion stimulation tools, 4.35 % (2) used official images from the International Affective Picture System (IAPS), and 4.35 % (2) used some kind of mechanical device to stimulate emotions (vibrations 15 yet it is largely unmeasured, modeled or characterized. A better understanding of user perception will aid the design of tactile behavior that engages touch, with an experience that satisfies rather than intrudes. We measured 30 subjects\u2019 affec- tive response to vibrations varying in rhythm and frequency, then examined how differences in demographic, everyday use of touch, and tactile processing abilities contribute to variations in affective response. To this end, we developed five affective and sensory rat- ing scales and two tactile performance tasks, and also employed a published \ u2018Need for Touch\u2019 (NFT, LEGOS 16). The remaining 30.43 % (14) corresponded to studies in the field of emotion simulation 17-20, the development and adaptation of tools for measuring emotions 21, literature revisions and editorials 6,14,22-24, and the interaction of the individual with their environment and other individuals 25-282009. ACII 2009: IEEE.
III. Results
Out of the 46 publications found in this work, 14 reported developments using physiological signals different from ECG, GSR, and EMG (Table 1). Of the other 32, seven articles reported having worked with emotion stimulation tools other than video games (Table 2). Finally, of the last 25, 10 corresponded to articles on developments in emotion simulation, revision of literature regarding machine learning algorithm techniques and serious games, and development of tools such as toolkits and editorials for technical events (Table 3).
Table 1 Articles excluded for not using physiological signal measurements, such as ecg, gsr or emg
| Measurement signal | Number of articles | Articles |
|---|---|---|
| EEG | 4 | [29-32] |
| Magnetic Resonance Images of the brain | 2 | [27, 33] |
| Interface/ Questionnaire | 3 | [15, 25, 34] |
| Behavior | 3 | [26, 28, 35] |
| Body posture/ Facial -speech recognition | 2 | [36, 37] |
Table 2 Article excluded for not using video games as an emotion stimulation tool
| Stimulation tool | Number of articles | Articles |
|---|---|---|
| Movies/videos | 5 | [38-42] |
| Images | 1 | [43] |
| LEGO | 1 | [16] |
Table 3 Articles excluded for not presenting the results of the measurements or the research in the delimited area
| Main topic | Number of articles | Articles |
|---|---|---|
| Systematic reviews and studies (Machine learning, Serious games) | 4 | [6, 14, 23, 24] |
| Development of computational tools | 1 | [21] |
| Editorials | 1 | [22] |
| Digital simulation of emotions | 4 | [17]-[20] |
The 15 articles selected in the study are shown in Table 4, which briefly summarizes the most important aspects taken from each study. We can identify that these studies, mostly, carried out multimodal emotion recognition, in which they integrated signals from different sensors to measure the physiological responses of the individual interacting with the video game, and thus, identified the emotion that was being experienced during the game. GSR, along with heart rate (HR) from ECG or blood volume pulse (BVP), are the most used signals to identify emotions, due to the high efficiency that can be achieved in their detection and measurement, as well as the ease of access to the measurement equipment. We can also observe that 17.4 % of the referenced articles involved the measurement of signals through electroencephalograms (EEG) as the main measured signal, or as a complement, given that the information extracted from the behavior of the brain waves is relevant for detecting emotions. Including sensors to measure EMG signals allow to identify movements in the corrugator and zygomaticus major muscles, which are in charge of controlling the movement of the eyebrows and cheeks in the face, and therefore are directly related with the recognition of the facial expression.
It is important to mention that a large amount of the literature reported in academic databases does not clearly identify the key words or the indexed terms; therefore, the search protocol and the subsequent meta-analysis did not include all the studies that report innovative and important developments in the field of study.
Table 4 Articles selected for the systematic review
| Article | Physiological signals | Video game type | Measured emotions | Analysis technique | Population |
|---|---|---|---|---|---|
| [44] | EDA (SCL, SCR) | Racing | Arousal levels | Open and closed loops control | 20 students (between 18-33 years) |
| [45] | Respiration, SC, EMG shoulder, TEMP, BVP | Tetris | 12 discrete positive and negative emotions | Questionnaire and Visual analysis-correlation | 32 subjects (11 female and 21 males) ages between 20-61 |
| [46] | EEG, GSR, BVP, Respiration belt, temp | Tetris | arousal and pleasure levels | Questionnaire and ANOVA analysis | 20 participants (mean ages 27; 13 males; all right handed) |
| [47] | EMG to Eye-blink | Fire emergency simulator VR | Stress | Manual analysis | 36 participants (21 M, 15 F ages between 16 and 35 |
| [48] | GSR and blood volume pulse (BVP) | Super Mario 64 (NINTENDO64) | Stress | ANOVA analysis | 14 males and 4 females university students aged 19 to 31 |
| [49] | Heart rate variability (HRV) and low skin conductance level (SCL) | Medical videogame: 'Virtual Remote Dome Controller' | Positive and Negative Affect Scale (PANAS) | Linear correlation analyses | 28 male volunteers aged between 19 and 27 (mean age of 22) |
| [50] | Skin conductance | StartleMart | Stress | Event-based stress | 13 veterans suffering from posttraumatic stress disorder |
| [51] | EEG, ECG and keyboard-mouse activity | Horror game Slender: The Eight Pages (STEP) | Fear | Machine learning algorithms and questionnaire | 11 participants (6 Males, 5 Females) aged between 21 and 32 years old |
| [52] | EDA (SCL, SCR), HR, HVR | Three emotionally laden VR scenarios with Oculus rift | Russells model: Pleasure, Arousal, and Dominance(PAD) | Matlab and SPSS, oneway ANOVA, paired t-test | 6 high functioning children and adolescents with ASD between the ages from 9 to 14 |
| [53] | EDA, BVP, Facial EMG | Levels in which virtual character carrying out a task inside a realistic VE | Stress | EDA-only algorithm Multi-sensor algorithm | 35 participants (26 Males, 9 Females) |
| [54] | SC, HR and facial EMG | Horror game | Russell's AV dimensional theory of emotions | Triangulation process between events and arousal space. Simple local maxima/ minima (LMM) detection algorithm | 6 volunteers |
| [55] | ECG, facial EMG, HR, SBP, and DBP | MadWorld Wii (Nintendo wii) | Stress, boredom, and relaxing | Questionnaire and correlation signals | Female (n= 62) and male (n=74), (M age = 20.00, SD = 2.25) |
| [56] | SCL | Ultimatum Game | Sensitivity to offers | ANOVA analysis | Two groups: 12 students (5 females) The mean ages were 19.1 (SD = 0.49) years for the psychopathic group, 16 students (8 females). and 19.1 (SD = 1.27) years for the low psychopathic group |
| [57] | EEG, Eye tracking, facial expressions, skin conductivity, Posture, finger pressure | Guitar Hero | engagement/boredom, frustration | Agents | 21 students |
| [58] | EEG and ECG | Survival horror | anxiety, suspense, and fear | Affect annotation tool (AAT) | 12 participants (11 males and 1 female) aged between 22 and 36 years old (mean = 25.42, SD =3.75) |
IV. Conclusions
From the systematic revision, we can conclude that video games have become an innovative approach as emotion stimulation tools, due to their potential to involve stories and multimedia tools that can interact directly with the person. However, video games still lack a classification that allows them to be linked with the emotions they may evoke. When using video games in studies of affective recognition, their selection is carried out according to the situation that will be stimulated, in order to obtain the same reaction from the player to the virtual environment and, as we suppose, to a real one. The use of commercial video games, paid or free (the most popular ones are riddles, jigsaw puzzles, simulators, and survival/horror, among others), is a common denominator among the studies referenced in this article. However, a selection methodology that allows for an initial classification of different video games that might be used in the same study is not reported in the different works, nor which game is more suitable to the requirements of the research being carried out.
Signals like GSR and ECG are widely used in this type of studies for detecting and measuring parameters related to emotions, such as HR, SCL and SCR, due to their high efficiency showing the physiological change connected to the emotions experienced by the individual. The EMG signals are used to recognize facial expressions and their changes; however, when considering the use of virtual reality tools, such as immersive glasses, this type of measurement is compromised due to the necessity of putting sensors on the face. For this reason, it is necessary to identify areas of the anatomy where it is possible to put sensors to measure the muscular activity when some kind of emotion is experienced, mainly stress, fear, and anxiety.
Finally, the field of affective video games is growing, with different possible action fronts; from the inclusion of avatars, which can demonstrate emotion through biofeedback, to the inclusion of emotion detection from players to improve the video game development process, as well as other fields of social, educational, and health. To obtain an accurate and robust technological system for the health and treatment field using videogames, it is necessary to develop new hardware and software tools that allow to measure and analyze signals in real time, without interfering with the normal behavior of the individual when is interacting with the video game. For example, biofeedback is a novel tool developed in the field of game design, with the potential to not only affect the players' game experience, but also create new branches in applied fields such the treatment of mental disorders. This kind of works are still in an experimental stage, focusing on the implementation of tools and protocols, validating not only the sensors and the acquisition system, but also, the process to acquire the physiological data to eventually apply them in the rehabilitation field.
