Mathematical Model

In factor analysis, a linear model is assumed:

where

X(p x 1) is the observable random vector, with mean vector µ and covariance matrix Σ;

L(p x m) is the matrix of factor loadings;

F (m x 1) are common factors, unobserved values of factors which describe major features of members of the population;

ε (p x 1) are error specific factors, measurement error and variation not accounted for by the common factors;

µi is the mean of variable I;

ε _i is the ith specific factor;

F _j is the jth common factor; and

L _ij is the loading of the ¿th variable on the j th factor.

We assume that the unobservable random vectors F and ε satisfy the following conditions:

F and ε are independent;

E(ε) = 0 and Cov(ε) = Ψ; where Ψ, is a diagonal matrix; and E(F) = 0and Cov(F) = 1.

Thus, the factors are assumed to be uncorrelated. This is called the orthogonal factor mode.

Factoring, given the model: X = µ + LF + ε

The implied covariance structure for X is,

Cov( X) = LL' + Ψ

If, V(X _i ) = l ² _i1 +… + l ² _im + Ψ _i and;

Cov(X _(i) ,...,X( _k) ) = l ² _i1 l ² _k1 , + … + l ² _(im) l ² _km

Furthermore, Cov(X,F) = L so that Cov(X _(i) ,...,F _(j) ) =l _(ij)

The portion of variance of the ¿th variable that is explained by the m common factors is called the communuality of the ¿th variable: σ _ii = h ² _i + Ψ _i where σ _ii is the variance of X _i , i.e., the ith diagonal of Σ; h ² _i = (LL´) _ii = l ² _i1 , + … + l ² _(i2) + l ² _(im) . is the communality of X _i ; and Ψ _i is the specific variance or uniqueness of X _i .

Note that the communality h ² _i is the sum of squared loadings for X _i (^{Harman, 1968}).

In this case, thirty-three items or quantitative variables are presented, so the factor analysis technique is applied with the extraction method of main components to obtain two dummy variables that allow to relate and summarize the teaching staff survey. This allows to evaluate the relevant aspects of the teacher, within the teaching-learning process.

Analysis of the original information regarding its relevance and validity

An exploratory analysis is made of the data obtained from the application of the evaluation instruments of 33 items with 5 answer choices (see Table 1), which were carried out by 6 110 students of the engineering, science and higher level technological programs for the professors of the Escuela Politécnica Nacional. These students were enrolled in 8 faculties and schools, studying 24 different degrees. The higher percentage of male students is representative of the population of students of polytechnic studies, in which 68,60% were male and 31,40% were female. The average age was 22,30 years old. These 6 110 students attended 1 380 different subjects which were distributed into 1 812 class-groups. The teacher sample consisted of 670 teachers, who represented a varied sample in terms of age, category, and teaching experience. More than half of these teachers were male (62,80%). The application of the scale of 33 items was carried out at the end of semester 2017-A (October 2017-March 2018), before the students knew their final grades. All teachers were evaluated by the students in the same term. All students had to evaluate the teachers to be able to access their final grades. The student teaching evaluation was conducted through an electronic platform, obtaining 19 527 records (original data matrix) in which the data were recorded (the same student was able to evaluate several professors since he/she took several subjects).

From the original data matrix, a correlation matrix is elaborated between all the considered variables (items). Several tests are carried out to determine if it is pertinent, from a statistical point of view, to carry out factor analysis with the information available from the correlation matrix. The main tests are:

The Bartlett sphericity test: it is based on chi-square distribution, where high values lead to rejecting the null hypothesis (H₀) that states that the variables are not correlated within the population. Thus, Bartlett's test of sphericity determines whether the correlation matrix is an identity matrix, which would indicate that the factorial model is inadequate. If the significance value (p-value) is less than 0,050, we reject the null hypothesis (H₀) and continue with the factor analysis.

The Kaiser-Meyer-Olkin Index (KMO): it allows the comparison between the magnitude of the observed correlation coefficients and the magnitude of the partial correlation coefficients. The KMO statistic varies between 0 and 1. Those less than 0,500 indicate that factor analysis is not required for the data in question.

The partial correlation coefficient: it describes the linear relationship between two variables while controlling the effects of one or more additional variables. These coefficients should tend to zero, when they are lent for factor analysis (Montoya O. 2005).

Extraction of Main components

Interpretation of the main components is often difficult, so the initial extraction is rotated to achieve a solution that facilitates it. Varimax with ^{Kaiser Normalization (Kaiser, 1958}) is the rotation method that uses the orthogonal rotation of factors previously normalized. In other words, it maintains the independence between the rotated factors. This method achieves that each rotated component presents correlations with only a few variables. Therefore, this method minimizes the number of variables with high loads by one factor and is adequate when the number of components is reduced.

Statistical analysis of teacher evaluation instruments

Bartlett's sphericity test was applied before using the multivariate factor analysis technique in order to verify if the correlation matrix is an identity matrix, which means that the correlations between the variables are zeros. The test consists of an estimation of the chi-square indicator, where high values lead to rejecting the null hypothesis. The test must have a significance value lower than the 0,050 limit, which would indicate that the variables are not correlated within the population. Table 3 shows the result of the Bartlett's Sphericity test that is 0,000. This demonstrates that the null hypothesis is rejected. Therefore, factor analysis is applicable in this case.

Table 2 KMO and Bartlett Tests 

Source: Authors (data analysis performed by SPSS v.22)

Table 3 Factor loadings and total variance explained 

Source: Authors

The analysis tool that was used was the Kaiser-Meyer-Olkin test (KMO). It is an index that compares the magnitude of the correlation coefficients observed with the magnitude of the partial correlation coefficients, eliminating the effect of the remaining variables included in the analysis. Since the partial correlation between two variables must be small when the factorial model is adequate, the denominator must increase a little compared to the magnitude of the correlation coefficients observed if the data corresponds to a factorial structure, in which case KMO will have a value close to 1.

Table 3 shows the result of the KMO test using the SPSS statistical analysis software, which has a value of 0,990, very close to the unit and therefore fulfills the requirement.

The partial conclusion that can be reached about this first part is that the two types of analysis on the pertinence and validity of the data matrix are satisfactorily verified.

Now, we proceed with the second part, which consists of extracting the principal components by grouping the 33 items or original variables into new variables called "factors". It is based on an exploratory analysis and shows that there is a large number of stereotyped responses, defined as those in which students respond with a single type of score along the whole scale, be it 1, 2, 3, 4 or 5. The data from these students is eliminated and, finally, the number of records on which the analysis is based is 15 771.

The results of the factor analysis of the sample reveal the existence of two factors, dimensions, or different constructs, as can be seen in the sedimentation chart in Figure I. These are chosen when the components have eigenvalues greater than 1.

Source: Authors

Figure 1 Sedimentation chart 

The total variance explained in Table 3 analyzes in detail the selection of the two components, factors, or constructs: factor 1 explains 70% of the variation in the scores of the scale, and factor 2, 3,20%. Only the first two factors have eigenvalues greater than 1 and explain 73,20% of the original problem, resulting in a loss of 26,80% of the original information due to the fact that the survey has a very high number of items, among other aspects.

Factor 1 is composed of items 17 to 33. All the items have high saturations between them and the factor (0,780 to 0. 680). This is related to the factor or set of items. The items with higher load or saturations are, in this order, items 32, 28, 31, 30, 25, 26, 24, 29, 27, 33, 22, and 20, to the less representative 23,18,19, 21 and 17.

Given that all these items refer to the teacher-student relationship, this factor can be called Teacher-Student Relationship and establishment of a good learning environment.

Extraction Method: Principal component Analysis rotation Method: Varimax with Kaiser Normalization

Factor 2 is composed of items 1 to 16. All the items have saturations or high relationships between each one of them and the factor (0,761 to 0,670); the items with higher loads or saturations are, in that order, the items 3, 4, 5, 2, 6, 7, 1, 9, and 8. In addition, these items refer to what may be called Planning, mastery, and clarity in the explanation of the subject.

Given that Factor 1, displays a greater variance percentage than Factor 2, this indicates that the students of the Escuela Politécnica Nacional give the greatest importance to the teacher-student relationship, or, in other words, perform the assessment of the teacher depending on the quality of this relationship, to a greater extent than the aspect of Planning, mastery and clarity in the explanation of the subject.

Another requirement that any questionnaire or rating scale must meet is reliability. If all the items amount to or contribute to measure the same, the reliability will be high. As indicated above, the most used statistical tool to calculate reliability is Cronbach's Alpha (α) internal consistency coefficient. It evinces an adequate reliability when α values range from 0,650 to high values such as 0,800 and above.

To do this, the reliability of each of the factors obtained in the factorial analysis was calculated using Cronbach's α internal consistency coefficient; being α = 0,970 the reliability of Factor 1, and Factor 2, α = 0,950. I was very high in both cases.

Given that it is possible that both factors or aspects are related, the total reliability of the 33 item scale, that amounted to α = 0,980, was obtained. This implies that a total score of the scale can be obtained, as well as scores for each of the previous factors or sub-scales.

Once it is confirmed that the reliability of each of the sub-scales is very high, it is possible to determine which item contributes more to the reliability of the scale and which items are redundant. Moreover, these can be eliminated without decreasing the reliability of the scale.

The sub-scale/Factor 1, composed of 17 items, has a reliability α = 0,970. If redundant items are eliminated, this scale can be reduced to 6 items: 26, 28, 30, 31, 32, and 33, with reliability α = 0,950, which is still very high.

The sub-scale/Factor 2, composed of 16 items has a reliability α = 0,950. If redundant items are eliminated, this scale can be reduced to 5 items: 3, 4, 5, 6, and 7, with reliability α = 0,930, which remains considerably high.

Thus, the 33-item questionnaire can be reduced to about 11 items without loss of validity or reliability α = 0,960, and with practically the same informative value as the original evaluation instrument . These items would be, as indicated in Table 4:

Table 4 Reduced questionnaire of11 items 

Source: Authors

Factor 1/scale 1: Items 26, 28, 30, 31, 32, and 33. 27 and 29 could also be included in this order, with reliability α = 0,960.

Factor 2/scale 2: Items 3, 4, 5, 6, and 7. We could also include 9, with α = 0,940 and 2, with α = 0,940 in this order.

Then, the question that arises is: What happens with the other items and with the other theoretical aspects included in the scale as resources, Methodology and evaluation?

The answer explains that they contribute very little to the assessment of the teaching staff, given what the selected items of the reduced scale do.

However, the dimensions or aspects related to resources, Methodology and evaluation are important enough to be included in the scale, for which it is necessary to incorporate items that better represent these dimensions than the previous scale of 33 questions.

Therefore, a factor analysis was carried out to determine the extent to which the dimensions or aspects of resources, Methodology and evaluation influence the results, thus forcing the appearance of four factors, out of which two are new: evaluation, Methodology-resources and -two factors that had previously appeared- Teacher-student relationship, and Planning, mastery and clarity in the subject's explanation.Table 5 summarizes the variance parameters, factor loadings, as well as the Cronbach's Alpha internal consistency coefficients (α) for each aspect.

Table 5 Factorial analysis forced to 4 factors 

Source: Authors

Proposal of a reduced scale

Based on the information displayed in section A, it is considered convenient to better define the items on the evaluation and Methodology-resources aspects. To do this, 15 items are proposed, since they commonly appear in most universities (^{Alaminos and Castejón, 2006}). This scale could include the most effective items of the original questionnaire, along with some new items introduced from other questionnaires, based on the theoretical dimensions of the aspects that are to be measured (^{Casero, 2008}).

The analysis of the data obtained is structured in four aspects or dimensions: Teaching Development and Planning, Teacher-student relationship, evaluation and a global Assessment question, as indicated in Table 6, which shows each question with subscripts that express the following information:

1 = combined items of the aspects in the original questionnaire.

2 = relevant items of the original questionnaire.

3 = New items included

The analysis of the proposed reduced scale observed in Table 6 indicates that two of the included items, related to the Teacher-student relationship, were the same ones as in the original questionnaire because they provide relevant information about the evaluation to the professor. In addition to this table, there are items, such as 7, 12, and 15 that include two alternatives, which coincide in their meaning, but are expressed differently. Therefore, for those questions, it is necessary to choose between the options when applying the new survey.

From the results, two possible options are proposed for item 11, which are items 31 and 26 of the original questionnaires, offering the possibility of choosing between one or the other, since both alternatives have practically the same importance.

Finally, regarding item 16, which is related to the global Assessment is included in the questionnaire to evaluate the general performance of the professor. However, as it not a relevant aspect, it can be considered as a replacement to item 15.