English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Cited by Google 
Similars in
SciELO 
Similars in Google 
Permalink
1The definition of quality and its assurance is essentially a political rather than a technical process (Skolnik, 2010). Therefore, if quality is a polysemous concept (González & Espinoza, 2008), its measurement and assurance will also involve a multifaceted process (Schindler et al., 2015). For this reason, rather than worrying about the concept of quality as a phenomenon with a unique characteristic, greater attention should be paid to the mechanisms of quality assurance once defined in a given time and context. From this point of view, the procedures of accreditation of quality in higher education systems are relevant. Thus, concern for the quality of higher education entails the need to review institutional practices, in order to know the difficulties and limitations for the achievement of educational purposes, which would better guide the use of public resources (Garbanzo, 2007).
We would like to emphasize that the concept of quality in higher education is not a process that follows a single trajectory, since its interpretation and assurance involves the participation of interest groups that struggle to establish quality benchmarks and their respective forms of certification. Indeed, in Chile, quality is understood operationally and normatively as the fulfillment of the purposes declared by higher education institutions in the areas of institutional management, in teaching actions leading to a degree, in postgraduate programs and research, and in strategies of linkage with the environment (Comisión Nacional de Acreditación de Chile, 2015). This generates performative effects in all higher education training programs. This is especially true for initial teacher training (ITT), so much that institutional quality assurance and its results have an explanatory capacity of 55.1% on initial teacher training (Pedraja-Rejas et al., 2012). Subsequently, this has impacted professional practice immediately following training (Almeyda, 2016; Hirmas, 2014; Solís et al., 2016). Therefore, in a context of globalization and internationalization of educational policies-which are strongly hegemonized by neoliberal thinking-regulations, support, monitoring, and research are required to promote the quality of new teacher training (Ávalos, 2014).
This creates a substantial procedural problem for how to assess teacher training quality, which inherently should reflect training-its theoretical component-and the application of acquired knowledge. Unlike physical entities, social constructs are conceptualizations that cannot be directly measured. In this regard, quantification in education-and especially in initial teacher training-is a living and dynamic process of the utmost importance, since it is through quantification that the achievements of the teaching and learning process will become evident to a large extent. Therefore, this problem and a proposal to overcome it are the subject of what follows in this article. However, first, a conceptual framework on the subject is provided. This is followed by a proposal for measuring this correlation between training and professional practice, with the demonstration of its concrete application in a real teacher training program.
A Brief Outline of the Tension Between Theory and Practice in ITT from the Perspective of National Policies
ITT has been examined from different perspectives: some researchers focus on the structure and curriculum; others on professional practices and theoretical courses; while yet others focus on the relationship between theory and practice (Darling-Hammond, 2017) and the lack of connection between certain courses (Flores & Al-Barwani, 2016). For his part, Craig (2016) addresses the effects of the administrative political organization in teacher training schools and its impact on training, revealing the political, historical, and cultural influence on training in education.
The divide between theory and practice in ITT programs is well documented and their articulation has been identified as an ongoing challenge (Korthagen, 2010). The discourse of the deficit of beginning teachers in the classroom has given impetus to educational policies to ensure that teacher education programs are strictly regulated and their quality is assured. The effect of these regulations-for the certification of future teachers and/or the accreditation of teacher training programs-orients the curriculum of teachers and influences their entry, maintenance in the teaching career, and their recognition or approval.
Among the international debates on the results of governmental teacher training policies, Darling-Hammond (2017) highlights the decisions of the systems of Finland, Australia, Canada, and Singapore. She concludes that, despite their differences, these countries have consolidated an efficient education system-with teacher training, integration of theory into actual practice, and teacher career development. In examining these practices, she highlights as examples a closer match between the design and quality of preparation, the connection between theory and practice through well-thought-out course design, and the integration of high-quality practical work in settings where exemplary experiences take place.
Training practices in Latin America are relatively different from those mentioned above, but very similar to the conditions in Brazil, because they arise as adaptations of measures and practices of 1st-world countries (Marcondes et al., 2017), in a context in which schoolteachers, in general, have low salaries, most public schools are administered by municipalities, and districts lack the infrastructure and resources necessary for quality teaching. Predictably, private schools have better salaries and working conditions. These differences discourage the selection of teaching careers by young people. The response to this is the implementation of policies, in some countries such as Chile, to provide scholarships and to promote the choice of teaching careers by talented students.
The Complexities of Curricular Organization in ITT
Flores (2017) analyzes the curricular organization of several countries and identifies problematic aspects in the curricular conformation of ITT programs, including: the lack of connection between some subjects; the gap between theory and practice; the duration of teaching practice; the lack of clear communication between the university and the school; the lack of coordination between departments, supervisors, and coordinators of teaching practice; and the neglect of the needs of student-teachers during their practice.
Regarding the curricular structure of ITT programs, Zuzovsky and Donitsa-Schmidt (2017) describe two main models of program organization depending on the ways in which the general and professional components of the curriculum are organized: a concurrent or integrated model and a consecutive model. The former is characterized by degree-specific content knowledge being taught alongside pedagogical studies throughout the pre-service course of study. This model combines degree-specific content, pedagogical theory, research, and educational practice, which are taught simultaneously. The distribution of practice during teacher training in this model aims at the articulation of the different components of the program and makes it possible to distinguish three stages of the practice process: initial, intermediate, and final or professional. As for the consecutive model, this begins after the degree-specific content has been studied in depth, that is, once training in a discipline has been completed with the awarding of a degree. Once the degree has been awarded, pedagogical studies are undertaken. In contrast, the consecutive model focuses mainly on general studies in education and pedagogy, with a shorter period of practical experience. This implies less integration between theory, research, and practice.
There is also the three-dimensional model developed by Kitchen and Petrarca (2016) that situates the programs with respect to the emphasis with which they orient their courses in regard to theory, practice, and reflection. In this sense, Marcondes et al. (2017) state that the concern for the articulation between theory and practice, and between research and reflection, characterize teacher training in Brazil-also in other Latin American countries- which is promoted by the regulations contained in the National Curriculum of that country. This legal document defines the principles, teaching and learning conditions, and educational procedures to be followed by institutions of higher education when developing their curricula.
Chile follows the international trend of establishing policies and standards for pedagogical and degree-specific training, which we summarize below. The enactment of the General Education Law (Law number 20.370, of 2009) establishes learning objectives and outcomes for all educational levels, which effectively affects the prioritization of the degree-specific competencies promoted in ITT programs (Ponce & Camus, 2019). In addition, the standards for graduates of teaching careers (CPEIP & Ministry of Education of Chile [Mineduc], 2012) establish the curricular structure that will guide the accreditation evaluations of teacher training institutions. Among these curricular provisions for teacher training, the articulation of degree-specific, didactic, and pedagogical knowledge and the implementation of early and progressive practices as a training system in a guided practice stand out.
As for practical experiences, reflective practice is promoted from the observation of situations and problems in schools. The research thus conducted by student teachers turns out to be an integration of theoretical knowledge, built on existing explanations and relating them to real data, and their reflective practice (Ponce & Camus, 2019; Ripoll-Rivaldo, 2021). In this sense, the training practice allows a dialogue between the individual perspective teacher in training and the social context to critically reinvent the teacher, the school, and society (Molina & Cañas, 2019).
The curricular structure of the ITT in the university in which this study was carried out establishes for all its programs the format of training by competencies. In this curricular format, the dimensions of comprehensive, pedagogical, degree-specific, and professional practice training are articulated. The core subjects contribute to the integral formation of the student, including development of critical thinking, communication skills in a second language, citizenship training, and discipline-specific knowledge, such as mathematics, biology, language, and communications, among others.
In addition, the core subjects refer to the training in pedagogical and didactic knowledge of students, and professional practice dedicated to the integration of knowledge and the practical application of knowledge and skills demanded in a school-specific context (UPLA, 2017).
Why Measure the Effectiveness of ITT?
As indicated above, measurement in the social sciences-especially in education-is not a direct process, and therefore requires methods that are not free of errors or uncertainties. The measurement of the correspondence between what is theoretically acquired and what is actually carried out does not escape from this. However, there are generic methodological developments that can make this process more feasible. For example, Brennan (2010) presents a substantial improvement with the theory of generalizations relating it to Item Response Theory. Also, the fit of a good model is a key factor in this perspective. Therefore, Patarapichayatham et al. (2011) propose a series of statistical alternatives ranging from classical to Bayesian tools.
On the other hand, there is a line of research concerned with studying the statistical analyses used in quantifications in order to enrich them (Thomas & Zumbo, 2012), and from that perspective improve the metric characteristics of the tests. For example, Moses and Kim (2012) guide how to carry out a good process of comparison and classification of groups, while Jacob et al. (2014) indicate the importance of not wasting any type of information. In addition, the role of context and integrated assessment should not be ignored (Wilson & Sloane, 2000).
In this sense, it is interesting to study methods that analyze the effectiveness of certain pedagogical experiences that are difficult to classify, either as a form of learning or as an evaluation of it (Crisp, 2012). Such is the case of the so-called professional practices in pedagogy. These constitute a subject that most higher education training programs have, generally, established for advanced years of these programs, closer to graduation. Their purpose is to validate the competencies defined in the graduate profile of each career, especially with regard to the link between what knowledge is acquired theoretically and what is applied in real situations. These training activities have demonstrated their ability to configure the professional identity (Pereira-Bravo et al., 2022), what becomes relevant in changing scenarios, as establish by Almonacid-Fierro et al. (2021) with the emergence of COVID-19 and its effects on said training instance.
Thus, based on the Chilean context, the proposal made in this article should be considered significant, given that the quality assurance of pedagogical careers must be accredited, according to a 2006 mandate of the National System of Quality Assurance in Higher Education (Law number 20.129, of 2006). Therefore, pedagogical careers need to provide evidence through instruments or indicators, regarding the consistency of its entire curricular structure with the declared graduate profile. Hence, this article responds to the need to validate the training process prior to the professional practice experience in student teachers, in order to visualize the prediction that success in training subjects translates to similar success in practice.
The graduate profile took on special relevance during 2011 and 2012, after obtaining the first results of the Inicia Test, which made it possible to collect information on the quality of the training of students graduating from teaching careers, evaluating 34% of those who underwent the assessment as insufficient (Mineduc, 2013). Consequently, the first evidence was obtained where the skills of the graduate profile were established but were not achieved by a large majority of future teachers, revealing that the teaching staff does not reach the expected performance standards, which is partly due to structural elements of deregulation of the same teacher training (Bastías-Bastías & Iturra-Herrera, 2022). Currently, the challenge is even greater, since the new Professional Teaching Career together with teacher evaluation have added new challenges and a greater number of skills that practicing teachers and future teachers must acquire (Mineduc, n. d.).
The present proposal aims to specify optimization criteria and quantification of impacts on the relationship between academic performance and success in professional practices in teaching careers, proposing a stochastic index to quantify the specificity and sensitivity of academic performance on success in professional practices. This proposal is based on Bayes' Theorem (Barnard & Bayes, 1958), since it allows updating probabilities that are too complex to obtain a priori. Further, the realization of this proposal will serve as an evidence-based input to improve quality assurance processes in initial teacher training, and perhaps as a training validity mechanism for the entire higher education system, providing social utility (Tejedor, 2018) for pedagogical training.
Methods
Research Objective
This study focuses on knowing the ability of academic performance to predict success in the professional practices of teachers in training, through a stochastic index of predictive validity.
Unit of analysis, total population, and sample
The total population comprised students of scientific pedagogy careers of the Universidad de Playa Ancha de Ciencias de la Educación, Valparaíso, Chile, belonging to the 2006 to 2013 cohorts, with a size of 842 students. The 2013 cohort includes students who have recently completed their professional practice. The unit of analysis is the academic performance in the degree-specific2, core3, and professional practice subjects.
A probabilistic sample was used in which 377 students were included from the indicated population, exceeding a confidence level of 95%, and an estimation error of +5%.
Data collection process
We consulted the institutional database containing the academic records of the students in the sample-while protecting their identity-provided by the Institutional Analysis Unit of the Universidad de Playa Ancha de Ciencias de la Educación.
Proposal
The purpose of the educational quantification process is to contribute to educational diagnosis, seeking evidence in the data, and confirming or rejecting hypotheses. Therefore, it is important to know how to correctly interpret the impact of academic performance on professional practices. The result of the quantification of the impact can entail the classification of a student or decide their approval or rejection. Another consequence is that a good diagnosis or quantification of impact will allow us to guide pedagogical interventions, provide information on their status, or contribute to the implementation of preventive measures.
Regardless of the context or objective of the quantification process, it will be said that E (event) represents the possession of the characteristic of interest to be measured, or equivalently according to the problem if it develops a successful professional practice. This approach is from the perspective of the learner and particularly their processes of successful development in professional practice. Based on this notation we will say that CE (complementary event) will represent the opposite event, that is, not achieving the quantitative level expected to declare the professional practice as successful.
This notation allows us to indicate that the group of students under study is assumed to be partitioned into two sections, those belonging to E or exclusively to CE. It is observed that the notations E and CE are constructs that are not accessible to direct measurements. Therefore, in the measurement process there will exist intrinsically a process of measurement of random phenomena. This conception justifies the use of probability measures as a modeling of uncertainty. In other words, E and CE are platonic elements-gold standard-that characterize the actual possession of the trait of interest under study that we seek to quantify.
Now from the perspective of the measurement instrument or academic performance used for classification, academic performance prior to professional practice-realized in E or CE- is said to be positive if the quantification of learning reaches the predefined cut-off threshold, which is associated with degree-specific academic success. This event is symbolized by "+"; otherwise, academic performance is said to be negative if the quantification of learning does not reach the predefined cut-off threshold, which we symbolize by "-".
The relationship between success in professional practice and academic performance described above is shown in table 1.
If quantification by means of academic performance, as a predictor of success in professional practice, were perfect, then a = d = 0, but perfection is only platonic. The false positive (FP) will be the quotient , which is an estimate of the probability that, knowing that the student is not successful in professional practice, the academic performance will classify them as positive, which is symbolized by and is formally known as conditional probability. Similarly, the quotient is called false negative (FN), which is an estimate of the probability that, after it is known that the student possesses the characteristic of interest or is successful in his professional practice, the academic performance classifies it as negative, which is symbolized by . These two types of quotients or probabilities allow us to have objective expressions in relation to the errors that the academic performance can generate in relation to the predictive power on the professional practices. Based on this notation and objective concepts of initiation, the fundamental elements of the proposal are defined.
sensitivity: The sensitivity of academic performance is defined as the probability that, knowing that the student reached the level of success in professional practice, they will be classified as positive in terms of the cut-off threshold for academic performance. This concept is symbolized by .
specificity: Specificity is defined as the probability that, knowing that the student did not reach the level of success in professional practice, they will be classified as negative according to the cut-off threshold for academic performance. This concept is symbolized by.
predictive value: When academic performance is used for predictive purposes, it is of interest to quantify the probability that, knowing that academic performance reached its positive threshold in a student, they will be successful in their professional practice or equivalently. On the other hand, when the student's academic performance is classified as negative-that is, they did not reach the desired threshold-they will indeed not have the level for a successful professional practice. Both situations are symbolized as and , which are identified as positive predictive value and negative predictive value, respectively.
Note that the notations and represent a priori probabilities to academic performance and the notations and represent a posteriori probabilities to academic performance.
the role of Bayes' theorem: The importance of Bayes' theorem in this proposal lies in the updating of the probabilities indicated above, i.e., it allows estimating a complex problem as and in terms of simple probabilities such as and . Consequently, we can obtain an interesting result, being able to express the positive or negative predictive values in terms of sensitivity and specificity.
The expressions that allow us this relationship are the following:
Note that the probabilities and have a confirmatory interpretation, that is, they allow us to know whether success in professional practice is a predictor of academic performance. On the other hand, in and , the interest is in the professional practices, whose decision support is given by the previous academic performance.
On the other hand, and can be considered as a probabilistic approximation of validity, in the sense of measuring what it was designed to measure. For example, consider =as a percentage estimate of validity, which has the following properties: if then if and then . Therefore, improving sensitivity and specificity implies improving validity if and then
Thus, by increasing the predictive values and respectively, and decrease, which are called positive and negative predictive errors, respectively. The reason for this implication is the fact that the positive predictive value and the negative predictive error are complementary, that is: and . This means that, as the predictive value increases, the predictive error decreases (figure 1).
Results
The pedagogical careers of the Universidad de Playa Ancha have four internships, which are broken down into three pre-internships and one final internship, which is the professional internship. This process needs a supervising professor and a guiding teacher. The role of the supervising professor is intended to be an accompanying entity, a facilitator of the student's process, rather than a supervisor of the process. The guiding teacher must be a teacher of the specialty they are supervising, otherwise they must have a degree in basic pedagogy with a scientific specialty.
This process is evaluated based on a final work called the portfolio, which has a pre-established guideline with the criteria to be considered by both the supervising professor and the guiding teacher to determine the achievement of this professional practice. Thus, the cutoff threshold considered to categorize the student's performance at an optimal or reliable level is 6.0-grade on a scale of 1 to 7. That is, if the student has a grade higher than or equal to 6.0, they are considered a student who possesses the competencies required to professionally practice their career-choice. Otherwise, doubts arise regarding the possession of the necessary competencies to perform professionally.
Applying the proposal to the context described, we note that E represents the possession of the characteristic of interest to be measured, which may be having the necessary competencies to pass a discipline, course, or in a diagnostic context, if one possesses the necessary level to initiate a specific training process. In concrete terms, this is the positive quantitative evaluation of professional practice.
On the other hand, CE represents the complementary event, that is, it does not have the expected quantitative level. It should be remembered that E and CE are constructs that are not accessible via precise measurements. However, universities have a gold standard or platonic construct for identifying their training process, which in this case corresponds to having a quantitative evaluation of 6.0. It should be noted that the maximum grade in this metric system is 7.0 and the minimum is 1.0.
Table 2 Contrast of criteria for the relationship between the degree-specific and professional practices
Furthermore, "+" represents positive academic performance in degree-specific or core subjects, while "-" represents the opposite; that is, negative academic performance in degree-specific subjects or in core subjects.
These formulae proposed above have been applied to the sample of 377 students of pedagogical careers, and a summary of the results of the analysis are presented below in tables 2, 3, and 4.
Table 5 summarizes the interpretation of the results set out in table 4.
Table 5 Interpretation of the estimates of the Edumetric indicators
Note. * This quantifies whether success and failure in degree-specific subjects measures or explains success in professional practice.
It should be noted that the interpretation of validity is related to its proximity to 100%. This explains an individual's success in professional practice, which ultimately is the best reference of the achievement of the defined graduate profile. In both cases, this figure does not reach 50%, which in terms of the interpretation of predictive validity is very low.
Discussion and Conclusion
Assessment is dynamic and alive, so it must be constantly updated in order to gain greater precision and to take into account the characteristics of human nature. In this sense, measuring the quality of initial teacher training, as a social process, faces the challenge of generating pertinent methods that make it possible to establish consistency between the theoretical content delivered and professional practice. Thus, this proposal contributes metric tools to measure assessment instruments such as tests.
Measurement processes seek to objectify the platonic conceptualizations of an institution with respect to its intervention in teacher training, so that conceptualizations such as "having the necessary competencies for" are confirmed by means of indexes in the process of calibrating instruments for quantifying and diagnosing learning. Thus, it is possible to establish a better correlation between content and practice or, in other words, the adjustment to certain quality standards (Blanco, 2013), addressing the concern of Flores (2008) regarding the distance generated between theory and teaching practice.
From the technical point of view, the results obtained by means of this proposal make it possible to demonstrate the potentialities and weaknesses of an instrument in an objective manner, without falling into institutional or professional biases, which can definitively leave a mark on a student. Likewise, the proposal provided above allows us to quantify in a stochastic way the predictive validity of a variable on another response variable, using conditional probabilities and with the updating of probabilities that Bayes' theorem permits (Barnard & Bayes, 1958).
On the other hand, and considering the applied case, it is indicated that the predictive validity of both the academic performance in the degree-specific and in the core subjects, does not reach 50%, which demonstrates the existing disconnect between the success in the professional practices and the role of the university in how the student carried out their quasi-labor incursion. Thus, in correlational terms, the quantitative evaluation in the professional practices is independent of the academic performance in the degree-specific and nuclear areas. This could be explained by variables not considered in this index, either of a quantitative or qualitative nature, which of course opens the door to future research that scrutinizes the disconnect made explicit in the index. This, while serving as a test scenario for the proposal, puts initial teacher training in check, at least in the institutional context analyzed, since there is an important disconnect between what is theoretically delivered and what is executed as a supposed product of this theoretical preparation. This accentuates the theoretical-practical dichotomy as one of the main problems in the quality of training in this social discipline (Flores, 2008; McIntyre, 2009; Pedraja-Rejas et al. 2012), a disconnect with significant negative effects at the social level, both for the exercise and prestige of the teaching profession, as well as for its direct users.
These results necessitate further analysis and correction of the internal processes of initial teacher training quality assurance-which is part of an uncritical and neoliberalized macrosystem (Buendía, 2007) -in at least two ways: (1) to analyze the relevance of theoretical content in practice; and (2) the correct application of theoretical content in practice. That is, subsequent research should delve into the reason for this disconnect and the direction in which it occurs, favoring the adjustment of the purpose of teacher training (Schindler et al., 2015). Otherwise, this proposal offered here should be minimally considered an advance in educational research in general from its contribution as an input to improve practice-part of the social utility of research established by Tejedor (2018). Likewise, this methodology serves as an example for the entire quality assurance system in higher education training, and especially in training programs that divide their curriculum into theoretical periods and practical testing periods prior to graduation.
