(pdf)
SciELO 
Google 
SciELO 
Google 
Permalink
The new conception of reading implies considering at least two psychological dimensions: comprehension and self-regulation of the reading process. The self-regulation dimension presupposes a qualitative leap with respect to the traditional conception. Currently, the definition is more extensive and naturalistic, one that includes the characteristics of the reader, those of the text, those of the reader’s sociocultural context and the comprehension activities used that will interact to determine the comprehension of the texts (Gutierrez & Schraw, 2015; Schraw, 2009; Schraw & Dennison, 1994; Soto et al., 2018). It also implies the need for new individual diagnostic instruments that capture the capabilities that define the competent reader. If students understand only at the level of the propositional content of the text, it is extremely difficult to achieve deep learning because the mode of reading focused almost exclusively on the content does not favour reflective reading, but, on the contrary, tends to mechanise it, promoting superficial learning (González & Romero, 2000; Narvaja et al., 2003).
Any analysis of academic literacy for university students must presuppose that most of them are inexperienced readers, with few strategic cognitive resources, having mainly metacognitive ones. This will pose difficulties for them when dealing with texts, the requirements of teachers, the meanings of the problems, and the social and human implications of the problems. Reading expository texts, for instance, is not about rote learning, but rather learning by constructing the meaning of what is read. Academic literacy is an essential skill, not only a technical profile but the development of adaptive communication skills is required. In Chilean Universities results showed that the development of these soft skills among students is deficient. For this, a digital tool is proposed that allows a diagnosis of the reading level and its metacognitive strategies to be obtained (Cáceres-Serrano & Alvarado-Izquierdo, 2019; Calderón et al., 2020; McNamara, 2004). Analysing the reading profile in the aspects of reading comprehension implies that the instrument is sensitive in capturing the strengths, weaknesses and potential of the candidates or applicants as regards entering university. The instrument should emphasise the strategic aspects of comprehension and regulation, such as basic competences.
Theoretical framework
The objective of the study is to evaluate, in the new admissions of university students, four fundamental reading skills, for which we have not found an adequate instrument in the Spanish-speaking world. To remedy this deficiency, a group of experts in education and cognitive sciences selected and adapted four “ad hoc” texts in order to measure the reading profile of university students based on the following strategies and considerations:
(a) Recognise the potential of titles and summaries as activators of prior knowledge and organisers of mental schemes (Almanza et al., 2022; Berrocal & Ramírez, 2019; Castrillón Rivera et al., 2020).
(b) Train critical reading in areas of knowledge (for example, texts on humanities, social and political sciences) (Ali & Razali, 2019; Miyamoto et al., 2019).
(c) Investigate the importance of metacognitive reading and the high-level processes involved in critical and reflective reading. Many researchers conduct studies on the metacognitive reading strategies readers use. Studies relating to metacognitive reading strategies awareness and metacognitive reading strategies use have been conducted in different contexts and samples (Abu-Snoubar, 2017; Aktar & Ahmed, 2018; Alami, 2016; Dardjito, 2019; Rastegar et al., 2017; Sheikh et al., 2019; Teng, 2020; Wudeneh, 2018).
(d) Examine the role of inferences in reaching meanings beyond words and skimming, examining context, situation, and mindsets such as search engines, taking into account Concept Mapping (CM) as a powerful tool for discovering the meanings and relationships between concepts (Ali & Razali, 2019; Barreyro et al., 2017; Ferrara & Panlilio, 2020; Soto et al., 2019)
Measurement of reading comprehension at university
Vega et al. (2011) analysed the behaviour of university students in relation to self-regulated comprehension processes while reading multiple texts. They were to read three scientific texts on bacterial resistance, with the aim of responding to an inferential task that required the study of multiple documents. The self-regulation processes were captured through the methodology of thinking aloud (think-aloud)., In addition, different measures of comprehension were obtained. The results indicate that, in the first place, university students showed little evidence of planning processes for the reading activity. Second, regarding the monitoring process, they carried out both content evaluation and learning judgments; however, they were poorly adjusted in terms of the achievement of objectives for carrying out the task. Third, regarding the use of strategies, it was found that students make greater use of strategies aimed at the superficial understanding of irrelevant information. Likewise, deep understanding strategies were used indiscriminately with respect to relevant and irrelevant information, only at the intratextuality level. Finally, regarding performance in comprehension measurement, the results show students tend to score better on the superficial comprehension measure and score low on the learning transfer measure. These results are discussed in terms of the theory of comprehension of the text (s).
The present study
Based on the literature surveyed, this study aims at assessing reading metacomprehension in university students. The strategies evaluated with the instrument are: cognitive, metacognitive and linguistic. All of them are distributed in the four texts, although not uniformly. For example, in T1 more emphasis is placed on the use of cognitive strategies related to the mental scheme of representation of the text generated by the title and the abstract. In T2 and T3, the strategies are linked to the construction of cognitive maps and critical reading. Linguistic strategies are manifested in all the texts, especially with regard to the inferential process. T4 emphasises metacognitive strategies, but these strategies are also manifested in the other texts.
Method
Participants
Participants were students (N = 579) selected via convenience sampling and admitted to the degree in education in their various specialties: Basic General Education, Physical Education, Music Education, Kindergarten Education, Bachelor of History, Bachelor of Literature, Pedagogy in English and Psychopedagogy. Participants completed the PSU test, a standardised university selection test, with multiple-choice questions and closed answers, which includes the minimum mandatory contents of the curricular fra-mework. It does not include aspects related to opinions, field work, written reports, debates, and presentations. The mean age was 19 years and 2 months. All participating students were in their first year of studies, and 70% identified as female and 30% identified as male. Interestingly, only in the Degree in Physical Education were the participants distributed in a homogeneous way between gender identity. The PSU results range from 384 to 646 points. Values below 500 points are not usually accepted in most public universities and in scientific degrees, and only infrequently is it accepted in some private universities.
The investigation was approved by the ethics commission of the Faculty of Education and Social Sciences, of the Andrés Bello University, Chile, April 2018. The participants gave informed consent.
Instruments and materials
The reading comprehension test is comprised of four texts.
Text 1. Title: “Relationship of lifestyle and nutritional status in university students”
Text of scientific structure (of medium difficulty and short in length) where the abstract stands out as an activating strategy for reading comprehension. Readers should “pay attention” to the title and “analyse” the abstract (objective, method, results, and conclusion).
Text 2. Title: “Reggaeton, the music of reality”
Journalistic opinion piece that examines the reggaeton movement as music for young people. Efficient reading requires mastery of medium-high vocabulary and critical rea-ding to capture musical forms, characteristics, literary metaphors, the meanings of unknown words, and the construction of a concept map to represent content.
Text 3. Title: “Communication of bees” (Berko)
Entomology text of medium difficulty. The main challenge is to fill gaps in a hierarchical concept map, with two main square elements (e.g., nouns) and arrows (e.g., relationships). Other questions ask to retrieve meanings and employ frequently used generic reading strategies (before, during and after reading)
Text 4. Metacognitive Behaviour Subtest
Resolution of situations using “metacognitive judgments” and “self-regulation”. Readers assess four reading situations that appear on the test to assign a number to each of the strategies. The task requires the reader to have an appropriate level of awareness of the role of each of the rea-ding strategies. The way in which the readers resolve the situations reflects the level of strategic awareness of the subjects.
The ELCU was answered online through the SIGECOL platform, requiring between 30 and 40 minutes to complete. For more information on the characteristics of the test, any interested researcher can request access to the ins-trument by writing an email to the authors.
Cognitive Diagnostic Assessment (CDA)
There are very few tests to assess reading comprehension that have been developed through CDA (Tropak & Cakir, 2021), usually the Cognitive Diagnostic Models (CDM) have been applied a posteriori, to try to discover the strategies or skills that are re-quired by reading comprehension tasks (in Spanish, see for example, Alvarado et al., 2015; Cáceres-Serrano & Alvarado-Izquierdo, 2019).
The present investigation has the novelty of following a CDA design in which adequate texts were developed to evaluate the different abilities in order to later evaluate the goodness of fit to the componential IRT models such as the LLTM or to the CDMs such as the DINA. Both the LLTM and the DINA are confirmatory models that require previously defining the Q matrix of skills necessary to perform each task (or respond correctly to each task item). In our case, the Q matrix is very simple since the three questions of each text were simply specified with a one and zero in the others; Thus, for the first text the row of the Q matrix is 111,000,000,000, for the second text 000,111,000,000, for the third 000,000,111,000 and for the fourth 000,000,000,111.
Although the Q matrix is common to the LLTM and the DINA model, the psychometric tradition and interpretation of each model is very different (for a discussion of the differences between the LLTM and the DINA model see De la Torre and Douglas (2004). The LLTM is an extension of the Rasch model that allows grouping the different competencies or skills according to their difficulty, so once the tasks are ordered by difficulty, in order to obtain a good fit to the LLTM, it should be observed that both the difficulty parameters obtained through the Rasch model (without task groupings) correspond to the difficulty parameters grouped by tasks in the LLTM. Thus, both the Rasch model and its LLTM extension conceive of the trait as a continuous or metric latent variable, making it possible to order subjects by their highest or lowest level in the trait.
The Rasch model combines its simplicity, as it only considers one ability parameter bv with its mathematical strength, since when the model presents a good fit, the parameters are sufficient, consistent, efficient and unbiased. These properties make this model the only one proceeding from the Item Response Theory (IRT) in which the use of the sum of hits as a parameter is a justified aptitude (Wright, 1977). Determining that a scale fits the Rasch model does not imply that it must necessarily be homogeneous; in fact, an extension of the Rasch model is the LLTM, in which items are grouped by tasks that differ in difficulty and has been used in the design of reading comprehension instruments (see, for example, Alvarado et al., 2015). Now, the LLTM is a compensatory model in which the deficiency in aptitudes or abilities to perform one task may be offset by skills in performing another.
An alternative conceptualisation comes from the CDM (for a review see Rupp & Templin, 2008) that shifts the focus from items to individuals; considers that the latent variable is discrete, that is to say, there are multiple distributions, mixtures or classes of subjects, what is of interest is not so much the global score that can be obtained but to be able to characterise the subjects by the response pattern or class to which they belong. In the CDM, two subjects can have the same overall score but come from very different response patterns. In our case, two subjects can get three points if they master one of the four tasks (correct the three items of the task that are defined in the Q matrix) however, that score is not comparable unless the same task is mastered. Within the CDMs, the most parsimonious model is the DINA model, in which each task requires specific skills that cannot be supplemented by other skills. The most important advantage when a good adjustment to the DINA model is achieved is that it is possible to separate the subjects into classes, depending on the skills they may or may not have.
Data analysis
The analyses were carried out in R (R Core Team, 2017), using the eRm (Mair & Hatzinger, 2007), CDM (George et al., 2016) and G-DINA (Ma & de la Torre, 2018).
Results
Rasch and LLTM models
The Rasch model requires showing that there is a variance common to all the items, forming a one-dimensional structure. To test this assumption, a parallel analysis was performed verifying that only the eigenvalue of the first component = 1.87 exceeded the simulated cut-off value, its eigenvalue = 1.25. Secondly, once the assumption of unidimensionality has been accepted, it is necessary to contract the goodness of fit to the Rasch model, which was carried out using Andersen’s LR test (1973), showing an acceptable goodness of fit Chi-square (11) = 17.038, p =.107.
On Table 1 we can see the difficulty parameters of the items, showing great variability, from T3.P2, which was only correct by 40% (average hits or difficulty indices (id) = 0.40, parameter B = -1.29), to T2. P3 correct for 9 2% (id = 0.92 and parameter B = 1.70). The fact that there is variability in the Rasch parameters is necessary to propose the LLTM, since otherwise it is not possible for there to be groupings in the tasks in relation to the difficulties of the items that comprise them.
Table 1 Mean, SD, B parameters (item difficulty) for the Rasch Model (B) and η parameters (task difficulty) for the LLTM
In relation to the tasks, the task that was most difficult for the sample was T4, with its average difficulty index (idM) = 0.54 and a parameter η = -0.64 being estimated for the LLTM, followed by T3 (idM = 0.58 η = -0.43), T1 (idM = 0.73, η = 0.27) and T2 (idM = 0.82 η = 0.80). The fact that standard errors are very high in relation to the Rasch model indicates that there is no consistent grouping of the items based on their difficulty indices in the different tasks. In fact, the LLTM should be rejected since the difference between the goodness of fit between the Rasch model and the LLTM is statistically significant (Log-likelihood Rasch Model = -2760.104, Log-Likelihood LLTM = -2942.188, chi-square (7) = 364.16, p < 0.001). In addition, the correlation between the Rasch parameters and the LLTM was only 0.74, which leads to rejecting the LLTM and consequently to ruling out the possibility of defining the trait quantitatively (with different tasks that require different levels of cognitive processing with different levels of difficulty) leaving open the possibility that instead of a single class of individuals there are different classes depending on the strategies they use to solve the tasks and that will be evaluated below.
DINA model
The DINA model is a non-compensatory model and evaluates the skills required for the correct execution of each task independently. When an adequate goodness of fit to the DINA model is observed, it is possible to classify the examinees into classes, based on their abilities to perform each task correctly. So, if we also observe a fit to the DINA model, we have evidence that the skills (all of them related to the level of reading comprehension since they form a scale) required for each task are qualitatively different, unlike what would happen in a LLTM (which was tested and did not adjust) in which the differences are merely quantitative, in terms of difficulty.
The goodness of fit of the DINA model was assessed implementing the probability of the maximum chi-square value (max X2), the mean of the absolute deviations between the observed and predicted correlations for each pair of items (MADcor), the square root of the mean standardised residuals (SRMS) and the root mean square error of approximation (RMSEA). For an acceptable goodness of fit, p of max X2 and MADcor must be greater than 0.05 while SRMS and RMSEA < 0.05 (Cáceres-Serrano & Alvarado-Izquierdo, 2019).
After evaluating the goodness of fit to the DINA model, it was observed that the description in the four skills in a DINA model shows an adequate goodness of fit: max X2 = 2.382, p = 1, MADcor = 0.023, RMSEA = 0.021, SRMSR = 0.029 (see Table 2 for correlations between skills). Additionally, the adequacy of the Q matrix was evaluated using the CDM command din.validate.qmatrix() which reported the correct specification of the matrix.
Figure 1 shows the estimated parameters for the DINA model, since the difference between Guessing and Non-Slipping indicates the discrimination of the item, the most discriminative are those of the T4 test, which are also the least likely to have guesswork or chance, a result that is consistent with the assumption that miscomprehension is the superior skill.
The DINA model allows the sample to be separated into classes based on of the skills of the participants. On Table 2 we can see the different classes for the different possible skill patterns: the values 0 (no domain) and 1 (domain) are shown for the skills required by the four tasks examined (T1, T2, T3 and T4). The different classes range from “0000” for not mastering any skill, to “1111” for mastering the four skills considered. It is necessary to clarify that a participant can be assigned to class 0000 even if he/she gets some items right, since the model takes into account success by chance; in the same way a participant can be assigned to a pattern 1111 even if he/she fails some item, since the possibility of a “slip” is also taken into account, that is, a participant fails an item due to an oversight or slips up on an item that should be solved correctly.
If we observe the distribution of classes on Table 3, we can see that more than half of the participants master the four skills evaluated, although it is striking that 15.07% do not master any of them. It is also noteworthy that 7.59% master the first three skills, but not the last in T4, and in comparable terms, 6.67% master three of the four skills except for T2.
The rest of the classes show very low percentages, so it would be necessary to assess whether they can be attributed to an excess of correct answers by chance or to the difficulty in understanding any of the tests applied. Thus, for class 0111, which appears in 2.92%, it would have to be investigated whether they also mastered the skill referred to in task 1, and whether there was some comprehension problem regarding the task to be performed. Likewise, the examinees who present pattern 0110, who account for 2.54%, probably form part of 1110 and, finally, those assigned to pattern 0001, who curiously only do the most difficult task well, could have had an over-correction phenomenon by chance, and could actually form part of class 0000.
Discussion
Reading comprehension for university students is crucial for academic success. This cognitive activity enables countless learning experiences that have to do with both the acquisition of disciplinary content and the development of cognitive skills (Carlino, 2009). This assumption has generated numerous studies that confirm a widely accepted conclusion: there is a positive correlation between the level of reading comprehension and academic performance (Bashir & Mattoo, 2012; Cimmiyotti, 2013; Oliveira & Santos, 2006). College reading studies acknowledge some trends. The first is oriented towards the study of practical didactics. The second focuses on the evaluation of student performance in reading comprehension, with the purpose of making diagnoses and then implementing intervention programmes and improving performance. The third, which complements the second trend, studies the reader’s symptomatology, habits and reading skills. The objective of this study is to frame, in a particular way, the third trend as an instrument of diagnosis and psychoeducational intervention.
In order to ascertain the fundamentals that inspire the construction of the ELCU, the literature on reading comprehension difficulties focusing on the inferential and the contextual levels was reviewed. On the inferential level, the student builds the meaning and establishes the relationship of the ideas including implicit information, by carrying out processes of analysis, synthesis, induction, deduction, etc. On this level, greater difficulty is presented with problems regarding deducing, analysing and concluding based on new information found in the text. This weakness offers a key opportunity to work on inference processes and reading comprehension strategies (Puente et al., 2019). The contextual level of reading seeks to measure the ability to place the information in the text in an environment of meaning, which in practical terms means knowing the communicative intention of the author and his/her statements, assessing the usefulness of the content, the validity of the arguments, the relevance of the chosen superstructure and its coherence with the communicative intention, as well as critically analysing the content. The development of critical reading and the work with textual structures is evident, then, as a possibility of improvement, in order to develop pertinent reading comprehension strategies that do not exclude the re-flexive, evaluative, critical and scriptural processes.
Reading strategy plays a prominent role in comprehension because readers use them to construct the coherent mental representation and explanation of situations described in the text (Graesser, 2007). Comprehension strategies are also regarded as deliberate and goal-oriented processes used to construct meaning from text (Afflerbach et al., 2008). In particular, the use of deeper levels of strategies such as predicting upcoming text content, generating and answering questions, constructing self-explanation and clarification, capturing the gist of the text, and monitoring comprehension seems to promote good reading comprehension (McNamara, 2007). Basic research yields important information on reading comprehension strategies; therefore, it should not surprise that teachers and researchers of reading emphasize its importance in educational practice (Murphy et al., 2009).
The most efficient strategies of the ELCU test of academic literacy are observed in the good handling of texts: T1 (title and summary) is an excellent strategy that facilitates the identification of the main idea and discriminates secondary or filler ideas. Analysing the title and, above all, building a good summary from the data of the source text, facilitates the organisation and hierarchy of the text, according to the purposes of the reader and the type of text in question. T2 is a journalistic text whose reading requires critical strategies that reveal the argumentative value and the author’s beliefs. To read the text, it is necessary to recognise the meaning of all or most of the words in the text, through inferential mechanisms based on the text and the context. T3 is a mid-level entomology text. The most evident expression of comprehension is the elaboration of a concept map, as a valuable tool to measure reading comprehension. T4 is a different task from the previous ones. Students must make metacognitive judgments in reading situations. Skilled and unskilled readers can be differentiated by metacognition in reading. Such readers are distinguished based on their ability to comprehend a text relying on their overall world information and to being able to recognise various ideas given in the manuscripts (Mokhtari et al., 2018, Mokhtari & Reichard, 2002; Sheikh et al., 2019). Thus, those who know what they read as well as understand why they read and form strategies to handle the related difficulties and monitor their comprehension of information are called skilled readers. Similarly, those who do not possess all these characteristics to comprehend the meaning of any given text are unskilled readers. Such readers concentrate on a decoding process instead of producing its meaning. Most significantly, all of the components of metacognition support each other to accomplish comprehension and affect a reader’s performance. Thus, in order to enhance the students’ capability in reading, they should be taught reading strategies at school (Rabia et al., 2017).
Although the ELCU was designed following relatively objective criteria, some researchers of complex reading processes shy away from these criteria because objective tests do not clearly reflect the complexity of the cognitive operations of reading. This dissatisfaction was partially remedied by including texts that truly reflect the complex operations of reading, departing from the traditional and standardised path of objective tests. The new reading models of the Organisation for Economic Cooperation and Development (OECD) and other international organisations that measure reading performance are increasingly welcoming new innovative procedures. For this reason, although these texts have a low predictive level, their inclusion in the ELCU test is suggested as an exploration model that allows further investigation and refinement of the test.
Reading competence is a much broader concept than the traditional concept of the ability to read and write (literacy); in this sense, the OECD points out that the reading training of individuals for effective participation in modern society requires the ability to decode text, interpret the meaning of words and grammatical structures, and construct meaning. It also entails the ability to read between the lines and reflect on the purposes and audiences to whom the texts are addressed. Reading ability involves, therefore, the ability to understand and interpret a wide variety of text types and thus make sense of what is read by linking it to the contexts where they appear.
The results of the study provide evidence of validity and reliability of the final set of test items for measuring the level of reading comprehension of university students. This test is offered as a useful instrument to detect first-time university students who could have reading comprehension problems and, therefore, would see their academic performance decrease. In this sense, this instrument would help improve the reading comprehension levels of students corresponding to different areas of knowledge in the field of university education and social science. For example, it could be used as an initial measurement to determine whether some students require certain reinforcements in reading to successfully embark on their degree courses. To cover this need, the ELCU Test presented for newly admitted university students in its first version, despite its advantages derived from its cognitive design, is not free from limitations such as the presence of items with low discrimination that we hope to improve in future versions of the instrument.
In summary, it can be stated that reading comprehension and meta-comprehension skills are fundamental for good academic performance. Therefore, the detection of a substantial number of graduates with deficiencies in these basic skills is a point of attention that must be taken into account in accessing a university and thus reduce early dro-pout or failure to graduate. The observation that only 55.93% of those admitted have clearly consolidated reading comprehension and meta-comprehension skills should make us reflect on the possible implementation of reinforcement programmes at levels prior to university. The ELCU is an instrument for evaluating and predicting the reading performance of college freshmen. It is useful for preparing students for academic literacy tasks. The learning process of almost all disciplines and areas of knowledge is based on the proper management of cognitive and metacognitive strategies of analog and digital reading comprehension. The utility is observed particularly in texts that have a certain length, density of content and diversity of content and disciplines.
