Introduction

Eye tracking, the concurrent registration of eye movements, is widely employed in psychological research, where it has been said to provide the ‘gold criterion’ in psycholinguistic research (^{Rayner, 2009}; ^{Godfroid et al., 2020}). Eye tracking technology has been utilized in language processing research due to its controlled nature, offering insights into attention and various cognitive behaviors of learners. From 2013 to 2022, there has been a rise in studies exploring the use of eye-tracking technology in L2 assessments, including cognitive validity, correlation between viewing patterns and language performance, text readability, attention to visual cues in listening tests, and reading behaviors of students with varying comprehension levels. These studies, situated at the intersection of psycholinguistics, second language acquisition, and cognitive sciences, demonstrate the diverse and rich landscape of contemporary eye-tracking research in language testing and assessment, engaging with related disciplines.

Researchers employed eye-tracking technology to explore the depths of processing patterns of test takers and the cognitive validity of testing because eye-tracking offers millisecond-precise information on what a participant is visually attending to and most likely processing at any given time (^{Godfroid et al., 2020}). In doing so, eye-movement data can advance researchers’ understandings of the performance of test takers, as well as how different types of test items—such as multiple-choice questions and the argumentative writing—can influence the cognitive behaviors of test takers. Eye-movement data can assist researchers in comprehending how test takers interact with testing materials, and how this interaction may vary based on learners’ backgrounds, task instructions, item types, and test delivery methods. Findings from eye-tracking studies can provide valuable insights for teachers and test developers in enhancing the design and planning of testing materials to optimize learning outcomes.

In this paper, we review studies on the use of eye-tracking technologies in L2 assessments, as well as their applications in test design and pedagogical practices. Eye-tracking researchers have expanded the scope of language testing by incorporating other data-collection methods such as recall interviews, keystroke logging, and speak-aloud prototypes for data triangulation. We hope that this review will offer valuable insights for future research in this area.

Eye-tracking Technology in Listening Tests

Eye-tracking technology has also been applied to uncover the cognitive processes, including viewing behaviors, levels of attention in real-time, the effect of the spatial location of options on the processing order, and the correlation between gaze bias and domain knowledge of test takers. In these studies, researchers can infer from gaze behaviors the allocation of attention of examinees when they are completing test items, such as multiple-choice questions and short-video prompts. It has been concluded that listeners tended to focus their attention on objects that are anticipated to appear in the video-based listening tests. (^{Altmann, 2011}) even when they are told to ignore the auditory stream and instead look elsewhere (^{Salverda & Altmann, 2011}).

Eye-tracking has also been used to capture psycholinguistic processing in listening comprehension tests, such as in the studies of the effects of test methods on listening performance and cognitive load and attention to visual cues in L2 listening tests (^{Batty, 2021}). For instance, recent studies have focused on the influence of test methods on cognitive load and listening performance by examining test takers’ brain activity patterns (measured by near-infrared spectroscopy) and gaze behaviors (measured by eye- tracking) in while-listening performance (WLP) and post-listening performance (PLP) formats. As indicated by data on fixation rate, fixation duration, and neuroimaging, the research found that, consistent with both the cognitive load theory (^{Sweller, 2011}) and the study by ^{Aryadoust et al. (2020)}, WLP tests imposed a lighter cognitive load upon test takers than notetaking-while-listening in the PLA-Audio. Additionally, listening-while-answering test items during the WLP tests require less bottom-up and top-down processing than answering test items in the PLP-Question phase. Being the first study that integrated the data of eye-tracking and neuroimaging, this study was able to explain the observed test method’s effect on gaze behavioral measures and brain activity patterns.

Research of visual effects on the interpretation of test takers’ performance on video-based L2 listening tests has been based on test scores and self-reported verbal data (^{Wagner, 2010}; ^{Ockey, 2007}). However, few studies have employed eye-tracking technology to investigate examinees’ gaze patterns. To address this gap, ^{Suvorov (2015)} conducted a study on eye-movement measures—including fixation rate, dwell rate, and total dwell time—for context and content videos. The data revealed significant differences in fixation rates and total dwell time values, although no significant correlation was found between these eye-tracking measures and test scores. The findings nonetheless demonstrated that test takers utilized both visual and auditory information while listening. With the increasing shift towards online education, there has been an exploration into whether video instruction effectiveness can be evaluated remotely using standard web cameras. By analyzing the synchronized eye movements of 1000 participants through standard web cameras, the study found that students’ eye-movement patterns in instructional videos were similar. Furthermore, the study established a correlational relationship between eye movements and test performance, suggesting that stronger students were able to both comprehend the video content better and perform better on tests.

Response formats have been shown to be an influential factor in test takers’ performance in listening assessments (^{Brunfaut, 2016}), yet few studies have examined the impact of spatial effects on multiple-choice spatial relations. To address this gap, an eye-tracking study was conducted to investigate how the response order in multiple-choice questions influenced item difficulty. In the first part of the study, it was found that participants focused significantly longer on responses placed higher on the screen, marking the first time such results have been observed. The second part of the study tested whether items are more difficult when the key is placed in later positions in the Aptis Listening Test items. Consistent with previous research (^{Hohensinn & Baghaei, 2017}), the results indicated a direct effect of key position on item difficulty in a sample of 200 live Aptis items. The findings further suggested that the spatial location of the key in multiple-choice listening tests influenced the amount of processing it receives and the item’s difficulty. Given the widespread use of multiple-choice tasks in language assessments, these findings are crucial, particularly for tests that randomize response order, and for candidates who, by chance, encounter many keys in later positions.

Eye-tracking Technology in Writing Tests

It has also been shown that studies on L2 reading and listening have steadily increased in recent years. In contrast, eye-tracking has not seen widespread adoption in the study of L2 writing. This may be due to ongoing exploration by writing researchers regarding the applicability of eye-tracking in studying the writing process, with recent studies beginning to demonstrate its affordance in this area (e.g., ^{Ranalli et al., 2019}). The analysis of participants’ eye movements during composition often combines digital screen recordings with visualizations of eye movements (Rvsz et al., 2019). Under these conditions, the resulting video streams must be manually annotated by creating moment-by- moment segments. This is a time-consuming procedure, and such methodological complexity can hinder the development of L2 eye-tracking writing research. One potential solution is to fix the screen areas where participants can type their texts, but this may diminish the authenticity of experiments. This is particularly true in source-based academic writing—where authors frequently alternate between texts or scroll up and down—engaging in various cognitive processes such as reading, viewing, skimming, scanning, confusion, confirmation, rebuttal, and more, which would be extremely difficult, if not impossible, to disentangle in gaze behavior data. In such cases, writing researchers need to employ multiple methods to investigate these processes; for example, Rvsz et al. (2019) combined keystroke logs, eye-tracking, and stimulated recall comments, demonstrating the methodological complexity required in this field of study

Research on second-language (L2) writing encounters challenges as feedback from instructors often focuses on the final written products rather than the various processes involved in their creation, including planning, formulation, and evaluation. Utilizing eye-tracking technology enables researchers to observe writing processes in a naturalistic setting, avoiding additional cognitive burden on participants compared to methods like concurrent think-aloud. By examining students’ source use, valuable insights can be gained to explain differences in integrated writing performance, as indicated in previous studies. Therefore, the application of eye-tracking technology has also been used to uncover cognitive processes in writing tasks, revealing underlying motivational issues that may have limited participants’ learning.

^{Zhu et al. (2021)} investigated the correlation between discourse synthesis skills and overall integrated writing performance in both Chinese (students’ L1) and English (their L2). The results of integrated writing tests, eye-movement data, and stimulated interviews revealed that discourse synthesis skills are important indicators of integrated writing performance, supporting the validity of the two writing tasks that assessed underlying discourse synthesis skills. Researchers in the field of writing processes have utilized keystroke logging (KL) and eye-tracking (ET) to analyze and visualize process engagement, aiming to enhance the assessment of L2 writing. In their study, two Chinese L1 students enrolled at a U.S. university were selected as case studies. They completed argumentative writing tasks while a KL-ET system tracked their processes and generated visualizations for personalized tutoring. The findings indicated that eye-tracking technology enabled a deeper understanding of the participants’ writing development and identified motivational issues that hindered their learning.

Limitations and Directions for Future Research

This review aimed to investigate research on eye movement in L2 assessments, including listening, reading, and writing tests. By synthesizing this research, we hope that it will provide implications for further research. However, several limitations should be acknowledged. First, this research investigated research on eye-tracking technology in L2 assessments by synthesizing conclusions, methods as well as implications of relevant findings, however, this review only examined L2 assessments with eye-tracking technology, future investigations are suggested to provide an in-depth review on how eye-tracking technology, combined with other neuro-related data, such as neuroimaging data to give a full picture of the cognitive processes of test takers.

Despite significant advancements, several limitations remain prevalent in current eye-tracking research within second language (L2) assessment, highlighting the necessity for further exploration and methodological refinement.

First, the predominance of fixation-based measures in existing studies (e.g., fixation counts and durations) indicates an imbalance, with other potentially insightful metrics, such as saccades, regressions, pupil dilation, and blink rates, underutilized. Such measures can offer deeper insights into cognitive load, emotional arousal, and attentional shifts. Future research should diversify eye-tracking measures, exploring their unique contributions to understanding cognitive processes in L2 assessments.

Second, a substantial proportion of studies have concentrated on reading assessments, particularly high-stakes tests like TOEFL and IELTS. However, eye-tracking studies in listening, writing, and speaking assessments remain limited. Particularly, eye-tracking research in oral proficiency tests and interpreting and translation tasks is virtually non-existent. Future studies should address these gaps, examining whether eye-tracking can effectively capture cognitive processes in these modalities, potentially enriching construct validity evidence.

Third, methodological challenges persist regarding the integration and synchronization of eye- tracking with multimodal data. Accurate synchronization between auditory and visual stimuli in Reading-While-Listening (RWL) tasks, for example, remains challenging yet crucial. Future research should employ advanced synchronization techniques and software to minimize timing discrepancies and enhance the validity of multimodal studies. Moreover, current eye-tracking research predominantly involves controlled laboratory settings, which may limit ecological validity. There is a need for increased research in authentic, real-world contexts, including online and classroom-based assessments, to better reflect naturalistic L2 processing and behavior. The adaptation of remote eye- tracking technology, which leverages standard web cameras, offers promising opportunities for increased ecological validity and large-scale data collection. In addition, there is a growing—yet still insufficient—integration of eye-tracking data with other neurocognitive methodologies, such as neuroimaging techniques (fMRI, EEG) or physiological measures (heart rate variability, galvanic skin response). Future studies should pursue multimethod triangulation to create comprehensive cognitive profiles of L2 learners, thereby enhancing the robustness of cognitive validity evidence.

Integration of eye-tracking technology with artificial intelligence (AI) presents promising avenues for future research. AI-driven analytical methods, such as machine learning algorithms, can be employed to analyze complex and extensive eye-tracking datasets more efficiently, identifying subtle cognitive patterns and behaviors that traditional analyses might overlook. AI technology could also enhance adaptive testing systems, dynamically adjusting test difficulty based on real-time analysis of learners’ visual engagement and cognitive load. Furthermore, AI integration facilitates automated analysis of eye-tracking data for immediate feedback, providing learners and educators with actionable insights to tailor instructional strategies effectively.

Finally, participant diversity remains relatively limited in eye-tracking studies, with research primarily involving university students from specific language backgrounds. Future studies should broaden participant demographics to include varied proficiency levels, age groups, linguistic backgrounds, and learning contexts. This diversity would help generalize findings and provide deeper insights into differential cognitive processes across populations.

To sum up, while the integration of eye-tracking technology has profoundly enriched L2 assessment research, addressing these outlined limitations can significantly advance our understanding of cognitive processes underlying language assessment tasks. Future research efforts are encouraged to diversify methodologies, expand assessment modalities, increase ecological validity, integrate complementary neurocognitive measures, and enhance participant diversity, contributing to the robust evolution of eye-tracking studies in applied linguistics.

Conclusion

The integration of eye-tracking technology within applied linguistics has significantly reshaped research paradigms, opening pathways to more nuanced understanding of language processing and learner cognition. Eye-tracking, which records ocular movements, fixations, and saccades, has emerged as an invaluable methodological tool in applied linguistics research, particularly due to its capacity to yield direct, fine-grained insights into cognitive processes that underlie language comprehension and production (^{Godfroid, 2019}; ^{Conklin et al., 2018}; ^{Rayner & Clifton, 2009}).

Eye-tracking technology serves as a valuable bridge between theoretical constructs in applied linguistics and empirical cognitive evidence. Recent research emphasizes its effectiveness in capturing subconscious cognitive processes, including attentional allocation, lexical access, and syntactic parsing, thus offering concrete validation to theoretical frameworks in second language acquisition (SLA) and language pedagogy (^{Godfroid et al., 2020}; ^{Michel et al., 2021}). For instance, eye-tracking provides empirical substantiation to ^{Schmidt’s (1990)} Noticing Hypothesis, elucidating how learners consciously or subconsciously notice linguistic forms during exposure (Godfroid & Uggen, 2013). By measuring fixation durations and gaze sequences, researchers gain critical insights into learner engagement and cognitive load, bridging qualitative interpretations of learner behavior with quantitative cognitive metrics (^{Winke et al., 2013}).

Crucially, the relationship between eye-tracking and applied linguistics is not merely methodological but epistemological, reshaping the kinds of questions researchers can ask. Applied linguistics research has increasingly leveraged eye-tracking to explore nuanced aspects of language processing such as metaphor comprehension (^{Holmqvist & Andersson, 2017}), pragmatic inference- making (^{Fukuta & Yamashita, 2021}), and multimodal literacy development (^{Pellicer-Sánchez, 2020}). Eye-tracking’s capacity to precisely locate where and how long learners visually engage with textual or visual stimuli provides invaluable qualitative insights. This aligns closely with the editorial orientation of the Colombian Applied Linguistics Journal (CALJ), which traditionally prioritizes non-experimental, qualitative approaches that foreground learners’ subjective experiences and interpretative processes.

Despite its quantitative roots, eye-tracking data can indeed be effectively integrated into qualitative research designs. Gaze plot analyses, for example, have been qualitatively interpreted to illustrate learners’ strategic reading behaviors, decision-making processes, and metacognitive strategies (^{Conklin et al., 2018}). Complementing qualitative methods such as stimulated recall or think- aloud protocols, eye-tracking data enrich the interpretative depth, ensuring triangulation of data sources and enhancing credibility in qualitative studies (^{Fukuta & Yamashita, 2021}; ^{Pellicer-Sánchez, 2020}). Thus, rather than contradicting CALJ’s qualitative methodological preferences, eye-tracking serves to extend and deepen qualitative insights, providing explicit, real-time evidence of cognitive phenomena previously accessible only through retrospective self-report.

Recent studies employing eye-tracking have successfully illustrated qualitative processes in vocabulary acquisition and reading comprehension. ^{Godfroid et al. (2020)} qualitatively analyzed gaze sequences to explore second language vocabulary retention, highlighting nuanced individual differences in learning trajectories and strategies. Similarly, ^{Michel et al. (2021)} interpreted fixation patterns to elucidate the cognitive demands of syntactic complexity in second language reading, offering insights that purely qualitative methodologies might overlook.

Another significant implication is methodological transparency and reproducibility. Eye-tracking enhances methodological rigor in qualitative applied linguistics research by providing explicit visual representations of learner cognition, significantly reducing the interpretative ambiguity typically associated with qualitative data analysis. This technology allows applied linguists to substantiate qualitative claims with robust visual evidence, strengthening interpretative arguments (^{Conklin et al., 2018}).

Furthermore, eye-tracking’s inherent multimodality aligns well with contemporary qualitative approaches within applied linguistics, which emphasize learners’ interactions across different modalities (^{Pellicer-Sánchez, 2020}). By simultaneously tracking visual engagement with textual, graphical, and multimodal input, researchers can offer richly contextualized interpretations of learner behaviors, thereby resonating strongly with the qualitative ethos favored by CALJ.

From a pedagogical standpoint, eye-tracking research generates qualitative insights with direct implications for instructional practices. For example, qualitative analyses of gaze data have informed classroom practices by elucidating effective reading strategies, guiding the design of learner-friendly texts, and identifying specific instructional interventions for learners experiencing cognitive overload or attentional difficulties (^{Winke et al., 2013}; ^{Godfroid et al., 2020}).

Given the substantial qualitative interpretative potential of eye-tracking data, applied linguists adopting eye-tracking methodologies are advised to articulate explicitly how these data align with and enrich qualitative analytic frameworks. Such explicit articulation would facilitate acceptance of eye- tracking studies in CALJ, given the journal’s preference for qualitative and interpretative methodologies.

In conclusion, eye-tracking technology makes a significant contribution to applied linguistics research by providing nuanced, real-time insights into cognitive processes underlying language acquisition and processing. Although traditionally quantitative, eye-tracking data are readily amenable to qualitative interpretation and integration into qualitative research frameworks. By bridging empirical cognitive insights and qualitative interpretative analyses, eye-tracking enables applied linguistics researchers to engage deeply with learner cognition and interaction, advancing the field’s theoretical, methodological, and pedagogical frontiers. Consequently, eye-tracking research, when explicitly positioned within qualitative methodological paradigms, aligns seamlessly with the scholarly ethos and methodological preferences of CALJ.