COYUNTURA ECONÓMICA REGIONAL

 

School cover policy and academic achievement: the case of Medellin's concession program

 

 

Jorge Barrientos Marín*

* Jorge Barrientos Marin is researcher at the Centro de Investigaciones y Consultarías –CI&C–, University of Antioquia. E-mail: jbarr@economicas.udea.edu.co. Any error or mistake is responsibility of the author. We acknowledge the financial support of the local government of Medellin. This paper was presented at Sociedad de Economia de Chile-SECHI seminar in Valparaiso, Chile, 2008.

 

–Intoduction. –I. The Literature about Academic Achievement. –II. Methodology and Empirically Strategy. –A. The Fundamental Evaluation Problem-FEP. –B. The average treatment effect on the treated-ATT.- III. Statistical Analysis. –A. Descriptive stadistics.–B. The Average Treatment Effect on the Treated. –Conclusions. –References.

 

---

ABSTRACT

In recent years the secretary of Education of Medellín started the Concession Schools Program for school-age children who were not covered by the educational public system. This program was designed to provide private educational management by means of public resources where public supply was reduced. In this work we are interested in investigating the impact of this program on students' academic achievement. To this aim we match individual at concession schools with those at public ones by computing the probability of attending concession institutions. In spite of the good reputation of the concession strategy we found that students at concession schools had a lower performance than their peers at public schools.

Key words : Concession, matching, counterfactual, treatment and control groups, propensity score.

---

RESUMEN

En años recientes la Alcaldía de Medellín lanzó un programa de cobertura escolar para jóvenes por fuera del sistema educativo público. El programa, conocido como Colegios en Concesión, fue diseñado para proveer educación pública con administración privada en lugares donde la oferta de cupos escolares era reducida. En esta investigación indagamos por el impacto del programa sobre el logro académico de los estudiantes. Metodológicamente nos inclinamos por la metodología de pareo o matching. A pesar de la buena reputación de los colegios en concesión nosotros encontramos evidencia de que sus estudiantes tuvieron un desempeño inferior que sus pares en colegios administrados por medios públicos.

Palbras clave : concesión, pareo, contra factual, grupos de tratamiento y control, propensity score.

---

RÉSUMÉ

Récemment la Ville de Medellín a mis en place un programme de couverture scolaire pour les jeunes qui se trouvent hors le système éducatif public. Ce programme, connu depuis comme '' lycées en concession '', procure une éducation publique gérée par des privées dans les quartiers où l'offre scolaire a été réduite. Dans cet article, nous étudions l'impact de ce programme sur la réussite scolaire des élèves tout en utilisant la méthodologie appelée du matching. Malgré la bonne réputation des '' lycées en concession '' considérés par notre étude, nous trouvons que leurs élèves ont eu une performance scolaire inférieure à celle constaté dans les lycées gérés directement par la Ville.

Most clef: concession, matching, contrefactuel, groupes de traitement et de contrôle, propensity score.

Clasificación JEL: C14, C21, I20.

---

 

 

Introducción

In 2002 the local government of Medellin¹ started an educational program, namely Concession School Program-CSP, which consisted in providing public education services through private management. More precisely, 14 public institutions were given to be managed by 11 private operators (concessionaires) in order to provide educational services at every grade (elementary, secondary and high school) for five years, 2002-2006. On the one hand, the local Secretary of Education (SE²) designed the contract for the supply of educational services and, on the other hand, it endowed those institutions with enough resources to work. Every concessionaire should be responsible for the investment of these resources given by the local government.

The contract stipulated, among other commitments, to attend the maximum number of students per classroom at every grade. This amount would depend on characteristics of each institution like number of classrooms, the size of the buildings (square meters), etc. In addition, the concessionaires had to get furniture, the needed equipment for working and also the maintenance of the building. But, the most important feature of those institutions was the freedom to choose and manage the staff of professors, managers and principals, as well as the design of the strategy for interacting with communities. The idea behind CSP was to solve inflexibilities concerned to management and hiring personnel (especially professors), perhaps the main weakness of the public education in Colombia.

This program included 16290 children per year; this number corresponds to the number of places available at the 14 institutions, which was stipulated in the contract. So, according to the numbers of the SE, for 2006 nearly to 9.1% of the children at secondary school would be attended through the Concession scheme. It is worth noting that the cost per child and per year was established unilaterally by the SE; for instance, each child at secondary school in 2006 needed a monthly spending of $81.630 (Colombian pesos), so the annual amount spent per child was $979.560 (nearly to US$436), and this amount increased every year according to the contract. In summary, the CSP had a total cost of almost to 125 thousand million Colombian pesos (nearly to US$6250000) during the period 2002-2006.

In this paper, we are interested in investigating the impact of CSP on the academic achievement. We just take into account the last three years, the period of which the data is more complete and reliable. In addition, it was just in 2004 that the database, called Online Enrollment System (OES) started to operate, this one allows to us follow the evolution of the educational system in Medellin. OES is an important source of information which to some extent helps us to overcome little information contained in the Icfes³ form since 2001.

We going to use the matching methodology for estimating the average treatment effect on the treated (ATT) and thus overcome the well known fundamental evaluation problem- FEP. In this context we have two different groups of persons at the same time, we observe the score of the treated but we couldn't do this if they weren't being treated. For evaluating the impact of the CSP we take among our variables of interest the score attained by students in standardized tests, like ICFES's test⁴ (which is carried out by Icfes). Statistical procedures performed here take into account total results about score, language and math scores of ICFES. Our results suggest that, in spite of the good reputation of Concession Programs, students at concession schools in Medellin had a lower performance than their peers at public schools.

This paper is organized as follow: in Section I we present some relevant aspects of academic achievement concerning the Colombian literature (especially those aspects based on the production function approach). In Section II we show in short how matching and the propensity scores work. Section III provides us with the results about impact evaluation. Finally, we conclude.

 

I. The Literature about Academic Achievement

Most of the research papers investigating the relationship between academic achievement and quality of education are based on the production function approach. More precisely, this approach assumes that scores in standardized tests are the results of interactions between individual attributes and school inputs. It means that school is like a black box where both family background and school inputs interact to produce something we will call academic achievement; it is usually measured by the obtained score in standardized tests like ICFES.

In mathematical terms, let (Y_i), the educational attainment, let X_i a vector of characteristics of agent i, and let (Z_j) the set of school inputs at the school j, the production function approach tells us that (Yij) can be related with the matrix (X_ij, Z_j) through the function F(X_ij, Z_j), many of the times considered as a linear relationship. Assuming linearity we have:

Where E[ε|X, Z] = 0 and we get E[ε] = 0. Analysis based on the relationship given by the function F_ij(.) depends on several and strong assumptions widely discussed in the literature. In Colombia there is a wide literature based on the production function approach, using different methodologies and obtaining similar results. For instance, we have the paper by Gaviria and Barrientos (2001a, 2001b), Restrepo and Alviar (2002), Nunez et al. (2003). Since the early studies of Hanushek (1986) many papers have been published and recently we have, among others, Hanushek (1996), Murnane et al. (2005) and Hanushek (2005a, 2005b). The evidence found in the domestic literature suggests in summary that school is important but individual attributes could be the main contributor to students' academic achievement.

An interesting paper very related to ours is due to Barrera (2006). He makes an extensive analysis of the concession program carried out in Bogota, D.C. (a 15-year program similar, but not equal, to the Medellin one). In his research Barrera is interested in investigating the average concession program effect on the educational attainment measured by results in standardized tests. He finds that students at concession schools, on average, get one additional point in math and almost two additional points in language.

 

II. Methodology and Empirically Strategy

A. The Fundamental Evaluation Problem- FEP

Let Y be the dependent variable of interest, for instance the score in a single test, or an index of dropout rate. Let Y_1i be the score of student i (at moment t) who is affected by the concession program and let Y_0i be the score of student out of the concession program. We have that the impact of the treatment (on the treated) is given by the non observable quantity:

However, this difference is unknown because it is not possible to observe these two terms for the same student at the same time. This difficulty is known as the fundamental evaluation problem-FEP, Holland (1986). Instead of (1), we really observe:

where D = 1 if individual i has been treated and D = 0 if otherwise. There exist many ways to face this problem, but we are interested in applying matching by using propensity score, see Rosenbaum y Rubin (1983). It has to do with the construction of two groups to be compared; the first group is formed by people who went to concession school (treatment group) and the other is composed by people who went to publics schools (control group or comparison group)⁵.

B. The average treatment effect on the treated-ATT

In this paper we use the idea of matching to overcome the fundamental evaluation problem; in this context we have two different groups of people at the same time so we observe the score of the treated but we could not observe the score of the treated if they weren't being treated. The idea behind matching is to look for and compare various features of the control group similar to the ones in the treatment group. If we assume that the status of treated depends on some matrix X of observable characteristics we can set up a control group which is similar in X to the treated. It ensures that both groups have similar probability distribution function of X.

In this paper we suppose that students belonging neither to control nor treatment group are affected by the program. This assumption is not restrictive in our case and helps us to eliminate in some extent the FEP. Additionally, we assume that the impact of the concession program is independent (from a statistical viewpoint) from the score gotten by students into comparison group.

According to Rosenbaum and Rubin (1983), many observations in X can lead us to the well known problem (curse) of the dimensionality, they suggest us to use a probability measure p(X), which summarizes properties of matrix X, instead of matrix itself, they call this measure Propensity Score, and so we compute the quantity:

by estimating a sort of dependent qualitative model, which could be Logit specification (other like Probit could be possible). Once we have calculated the probability measurement (3) conditioned to X, we estimate the ATT by means of:

There are many ways to compute the counterfactual of non treatment for treatment group based on X (unfortunately (4) does not control for unobservable) with similar results, for instance we have matching kernel which is defined by:

where D₁ is the set of treated and D₀ is the comparison group. In doing so, matching not only gives us an estimation of ATT but also gives us an appropriate comparison group. We can also find the counterfactual by using nearest neighbor method, which is understood as the individual j nearest to the individual i such that:

In this paper we provide estimation of quantity (4) based on (5) and (6), we use as variables of interest the score obtained in ICFES test by students at concession and public school; additionally, we evaluate the performance of concession school by assessing dropout and failing rates. In summary, in practice we have to follow the next steps (Vinha, 2005):

• Estimate a qualitative dependent variable model to get: p(X) = p(D|X), with these probabilities we construct the counterfactual of no treatment for treated.

• Choose the number of comparison observation.

• Define the matching method (or the way in which the counterfactual is determined for each treated observation) in order to get an estimator of E(Y₀|p(X)).

• Finally, we get the ATT.

 

III. Statistical Analysis

A. Descriptive Statistics

The data set used in this paper is the result of linking three different databases, i) Online Enrollment System carried out since 2004, ii) ICFES test result during 2004- 2006, iii) C600⁶ and C100 survey which gives us information about dropout and passing rate and infrastructure (physical characteristic of the school), respectively. Unfortunately, information on infrastructure for concessionaires is not complete; so in many cases the ratio teacher/pupil is not available. Additionally, for both types of schools the most recent information about infrastructure was collected in 2002 (so we have to assume that it has not changed since then).

There are two features about the dataset which should be mentioned before going on. Firstly, the number of students at concession school who were tested is underestimated; because many of them at the moment of the registration at Icfes had an I.D different from the one they had when they enrolled at school (many of them had Identity Card which is the I.D for young people up to 18). Secondly, names and last names of students in the OES are different from the ones they had when they were registered at the moment of testing by the Icfes.

For statistical analysis we take into account students whose age ranges from 16 to 25 at the moment of testing. We eliminated schools which by error inform zero classrooms or zero square meters of building, etc. It leaves us a sample of 12028 students in 11^th grade in 99 schools in Medellin in 2004, 10736 in 2005 and 12749 in 2006, from which 609, 675 and 808 students, respectively, were enrolled at concession schools.

It is worth noting that analysis to be presented focuses on the total, math and language score (without the score on English test). Because of performance on natural and social science scores relies on memory more than on skills or student's abilities, we get rid these topics from the analysis.

Table 1 show us some descriptive statistics from the main variables used in this paper. On average, language score is two points higher at public schools than concession ones in each year; this trend holds for total score, public schools have on average 10 additional points that the concessionaries, which means better performance. The performance in math is very similar in public as in concession schools, especially within each year. In spite of differences in score average we did not find much dispersion between scores.

 

In both samples we have more women than men, so this confirms the female trend toward a higher schooling than the men. Students' age is on average near 17 with little difference between public schools and concessionaries; that number is according to most papers about academic achievement in Colombia. Note that most of the students belong to stratum 1, 2 or 3, as public schools as concession ones, in addition, there are not students at concession schools placed at stratum 4, 5 or 6 (variable stratum is defined by the local government for every household, so that the poorest students are supposed to be included in stratum 1 and the richest in stratum 6).

Schedule of schools in Colombia is usually divided in three: complete, morning and afternoon (other schools have evening schedule but this one wasn't considered here); in this sample, most of the students are enrolled in the afternoon (57%) or in the morning (41%). Table 1 shows us that public schools are better endowed than concession ones (they have more inputs), for instance the number of laboratories and desktop connected to internet is higher in public schools.

ATTE is performed by matching students whit similar individual characteristics contained in X and summarized through the value p(X), which is computed for every observation in the sample. In order to get the counter factual of the treatment we perform Nearest Neighbor (N.N) and Kernel Matching (K) based on propensity score. The kernel matching needs the choice of a set of smoothing parameters (bandwidths), in our case we choose it by performing Silverman (1986)'s rule of thumb which is given by h_s = 1.06σ(X_i)n^-1/5 where σ(.) is the standard deviation of X_i, and n is the sample size. We performed estimations trying bandwidths higher than Silverman's one.

B. The Average Treatment Effect on the Treated

1. Estimating ATT based on ICFES tests

Table 2 shows us estimation of propensity score by fitting Logit models; in doing so we get the counterfactual for the treatment, thus propensity score gives us the probability of being treated conditioned to some characteristics like gender, age, stratum, etc. According to the results, students residing in strata 1 or 2 increase the probability to be treated.

 

On the other hand, Table 3 provides marginal effects, showing that, on average, an additional year of age increases the probability of being treated by almost 1.5%. Male condition contributes less than 1% in such probability. The effect of strata on treated status is very strong, so we have that to reside in a neighborhood placed at strata 1 and strata 2 increases the probability of being treated by 18% and 6.3%, respectively (averaging over three years).

 

According to table 4, in every case the resulting estimations tell us that students at public schools on average score better than students at concessionaires ones. In 2004, for instance, students enrolled at concession institutions scored less than those enrolled at publics ones in total score, language and math, 10, almost 3 and 1 point, respectively. We carried out the same exercise by performing nearest neighbor matching obtaining similar results. We observe a slight improvement in both 2005 and 2006, but students at public school keep obtaining better scores than those enrolled at concession institutions. However, concerning math, students in both types of schools obtained similar results.

 

Table 5 shows us Kernel matching with Gaussian kernel using bandwidth h=1,25 > h_S, where hs is Silverman's bandwidth. Several exercises were performed using higher and lower bandwidths than hs with similar results. In fact not only the sing but also the length holds. These results are confirmed if we use N.N matching. Finally, we estimated ATT by performing kernel matching using Epanechnikov Kernel, choosing smoothing parameter by Silverman's rule of thumb procedure; however, we get the same conclusion: the ATT in 2004 tells us that students at concession school had worse performance that their peers at public ones (-9,7 in total score, -2,5 and -0,9 in language and math).

 

Also, we test the hypothesis that the mean of propensity score is not statistically different between both types of groups (public and concessionaire) by splitting up the sample into spaced blocks (optimal number of blocks are five) arriving to the conclusion that there isn't statistical difference between the mean of the distribution of X for students at public and concessions school. We compute critical values for t-student by performing 100 bootstrap replications. In every case the estimated parameters are significant at 5% level.

We found out strong effects at the regions defined by 0,01<p(X)<0,4 and 0,02<p(X)<0,30, for 2004, 2005 and 2006, respectively. We eliminate observation which probability to be treated equals one; in doing so we just take into account students whose characteristics are placed at common support region. Either heterogeneous treatment among students or systematic changes of observables might produce these strong effects at these regions. This hypothesis was not tested, however. Anyway, for sake of the simplicity, we have assumed common effects.

Table 6 shows the estimated ATT after removing students enrolled at six concession schools with the worst performance in language, math and total score in 2006 (also for 2004 and 2005 with similar results). The criterion to make the choice of concession schools with the worst attainment on the ICFES test is contained in Barrientos et. al (2007). They applied an institutional test called School Institutional Self-Evaluation, which was answered by every concession school. Estimations obtained by reducing the number of treated observations (in addition to a low number of covariates) produce non robust results (too low statistical significance).

 

Conclusions

According to the evidence, students at concession schools had a modest performance in ICFES test compared with their peers at public ones. We think that in spite of good reputation of the concession scheme, good results could arrive later than sooner. It is also the opinion in Sarmiento's paper (2005), which assessed the Concession program in Bogota, whose results do not favor the program.

Unlike results from Barrera (2006), our negative results might come because of the way in which concessionaires were chosen; we mean that needed requirements to provide educational services were not completely fulfilled by concessionaires and maybe the contract terms could have been dominated by quantity but not quality's criterion. Finally, it is possible that local government failed to watch the concessionaires as has been pointed out by Barrientos et al., (2007).

 

References

1. Alviar, M.; Restrepo, P. (2006). ''La Educación en Medellín: visión de futuro''. Documento ECSIM, Medellín.         [ Links ]

2. Barrera, F. (2006). ''Impact of private provision of public education: Empirical evidence from Bogotá's concession schools''. HDNED, World Bank.         [ Links ]

3. Barrientos, J.; Retrepo, P.; Tobón, D. y Zapata, M. (2007). ''La Evaluación del Programa de Colegios en Concesión en Medellín 2002-2006''. Grupo de Microeconomía Aplicada, Departamento de Economía, Universidad de Antioquia.         [ Links ]

4. Gaviria, A.; Barrientos, J. (2001a). ''Calidad de la Educación y Rendimiento Académico en Bogotá'', Coyuntura Social, Fedesarrolllo, No. 23.         [ Links ]

5. Gaviria, A.; Barrientos, J. (2001b). ''Características del Plantel y Calidad de la Educación en Bogotá'', Coyuntura Social, Fedesarrollo, No. 24.         [ Links ]

6. Gaviria, A.; Barrientos, J. (2001c). ''Determinantes de la Calidad de la Educación en Colombia''. Archivos de Economía, Departamento Nacional de Planeación- DNP-, Documento 159, Bogotá.         [ Links ]

7. Hanushek, E. A. (1986). ''The Economics of Schooling: Production and Efficiency in Public Schools'', Journal of Economic Literature, 24(3): pp. 1141-1177.         [ Links ]

8. Hanushek, E. A. (1996). ''Measuring Investment in Education'', Journal of Economic Perspectives, 10(4), pp. 9-30.         [ Links ]

9. Hanushek, E. A. (2005a), ''The Economics of School Quality'', German Economic Review. Vol. 6, No. 3, pp. 269-286.         [ Links ]

10. Hanushek, E. A., J. F., Kain, S. G. Rivkin and Gregory F. Branch. (2005b). ''Charter School Quality and Parental Decision Making With School Choice,'' NBER Working Papers, 11252, National Bureau of Economic Research, Inc.         [ Links ]

11. Heckman, J (1979). ''Sample Selection Bias as a Specification Error'', Econometrica. Vol. 47, No.1, pp. 153-162.         [ Links ]

12. Heckman, J.J; Lalonde, R.J y Smith, J. A (1999). The Economics and Econometrics of Active Labour Market Programs, in Ashenfelter A y D. Card (eds.) Handbook of Labor Economics, Vol. 3, Elsevier, Amsterdam.         [ Links ]

13. Holland, P. W. (1986). ''Statistics and Causal Inference'', Journal of the American Statistical Association, No. 81, pp. 945-970.         [ Links ]

14. Restrepo, P.; Alviar, M. (2005, enero-junio). ''El logro académico y el efecto colegio en las pruebas Icfes en Antioquia'', Lecturas de Economía. No. 60, Universidad de Antioquia.         [ Links ]

15. Rosenbaum, P.; y Rubin, D. (1983). ''The Central Role of Propensity Score in Observational Studies for Causal Effects''. Biometrica, 70 (1), pp. 41-55.         [ Links ]

16. Sarmiento, Alfredo; Alonso, C. E; Duncan, G; Garzón, C. A (2005). ''Evaluación de la Gestión de los Colegios en Concesión en Bogotá 2000-2003''. Archivos de Economía, Departamento Nacional de Planeación – DNP–, No. 291, Bogotá.         [ Links ]

17. Vinha, K. ( 2005). ''The impact of the Washington Metro on development patters, Ph.D. Dissertation'', Department of Agricultural and Natural Resource Economics, University of Maryland, College Park.        [ Links ]

 

 

Primera versión recibida el 25 de septiembre de 2009; version final aceptada el 9 de noviembre de 2009

 

 

Notas

1 Medellin is the second biggest city in Colombia, after Bogotá D.C.

2 Secretary of Education is the local agency which manages public education in Medellin, Colombia.

3 Instituto Colombiano para el Fomento de la Educación Superior.

4 By Icfes we mean the Institute and by ICFES the test. ICFES test is the equivalent to American SAT.

5 Control Group term is used when people inside the comparison group have been chosen randomly. Fortunately, at the moment of the enrollment students unknown they had been enrolled at Concession Schools.

6 Departamento Administrativo Nacional de Estadísticas.