Fundación Sergio Paiz Andrade

 Fundación Sergio Paiz Andrade - FUNSEPA
Monitoring & Evaluation Project
Final Report
July 2012
Table of Contents Executive Summary 3 Introduction 4 The Sergio Paiz Andrade Foundation (FUNSEPA) 5 Literature Review 6 Methodology 11 Qualitative Evaluation 11 Quantitative Evaluation 12 Sampling 12 Hypothesis and Statistical Model 14 Control Variables 17 Methodological Limitations 19 Results 22 Qualitative Results 22 Quantitative Results 27 Conclusions 34 References 35 Appendix 37 Monitoring & Evaluation Project -­‐ FUNSEPA 1 Table of Acronyms FUNSEPA Fundación Sergio Paiz Andrade TPE Tecnología para Educar AF Abriendo Futuro ICT Information and Communications Technology IDB Inter-­‐American Development Bank DIGEDUCA General Department of Educational Evaluation and Research of Guatemala DIPLAN Department of Educational Planning of Guatemala INE National Institute of Statistics of Guatemala Monitoring & Evaluation Project -­‐ FUNSEPA 2 Executive Summary This report presents the results of an evaluation of the programs of the Sergio Paiz Andrade Foundation: Tecnología para Educar y Abriendo Futuro. Tecnología para Educar aims at providing computers to public schools in Guatemala, while Abriendo Futuro looks to train public school teachers in the use of computers and their effective incorporation into traditional teaching. The assessment focuses primarily on the effect of these programs on student academic performance, as well as their effect on dropout rates and the probability of students passing to the next grade in the period 2005-­‐2010. The evaluation is a quasi-­‐experimental study that utilizes two groups of analysis: one group affected by FUNSEPA programs and a comparison group not affected by such programs. This methodology aims at comparing the difference observed in the FUNSEPA group before and after the implementation of the programs with the observed difference in the comparison group during the same time period. The statistical analysis was conducted using fixed effects and probabilistic models and included control variables such as class size, academic level, proportion of computers per student, proportion of trained teachers per student, and poverty levels, among others. The analysis alternated the variables of the programs (TPE effect and AF effect) and then looked at the combined effect of the programs (BOTH effect: TPE+AF). The main methodological limitations identified include school self-­‐selection into the FUNSEPA programs, the use of non-­‐standardized tests to estimate results, and the level of transfer of teachers between public schools, among others. After taking these limitations into account, the results of this report can be still considered as solid and as evidence of the effectiveness of FUNSEPA programs. The results of this evaluation show that, in general, the academic performance of students in schools benefited by the FUNSEPA programs is greater than that of students in comparison schools. This difference is particularly greater for schools that have participated in TPE, followed by schools that received the AF program. Schools benefited by BOTH programs (TPE+AF) gain an additional positive effect on academic performance when a second FUNSEPA program is incorporated to the school. In the case of dropout rates and the probability of being promoted to the next grade, students in schools benefited by FUNSEPA programs also show a greater performance than students in comparison schools. Schools that participated in TPE showed the greatest effect on dropout rates as well as on the probability of passing to the following grade. Students in schools with BOTH programs also gain an additional positive effect when a second FUNSEPA program is incorporated. Monitoring & Evaluation Project -­‐ FUNSEPA 3 Introduction This evaluation aims at determining the effect of the programs of the Sergio Paiz Andrade Foundation on the academic performance of students in schools benefited by these programs in comparison to students in non-­‐participating schools. The programs analyzed in this study are Tecnología para Educar and Abriendo Futuro. The evaluation observed the academic performance of 6,089 students in public schools in Guatemala between 2005 and 2010. Additionally, the evaluation included a qualitative analysis to better understand what changes could be expected from the programs and thus provide context to the quantitative analysis. After considering a series of methodological limitations, the results of the evaluation show that the FUNSEPA programs have, in general, a positive effect on the academic performance of students in schools that have participated in its programs. The first part of the report provides a brief overview of the FUNSEPA programs and later presents a literature review of evaluations of similar programs. These evaluations also look to estimate the effect of educational technology and teacher training programs on student academic performance. The purpose of the literature review is to understand the type of evaluations conducted in this field, the methodology used, and finally the observed results. The second part of the report describes the methodology used in the qualitative and quantitative analyses. This section details the characteristics of the sample, sampling techniques, the main hypothesis of the study, and the model specifications used for the quantitative analysis. This section also explains the methodological limitations of the study and the implications these have for the interpretation of results. The third part presents the qualitative and quantitative results of the evaluation. The first portion of this section is dedicated to the qualitative results and describes the perspectives of interviewed beneficiaries: students, teachers, and school principals. This section also presents the perspectives of external stakeholders interested in the improvement of public education in Guatemala. The second portion of this section presents the statistical results of the quantitative analysis, including results from the models on academic performance as well as the models on dropout rates and probability of students being promoted to the next grade. Finally, the fourth part presents a series of conclusions and recommendations for FUNSEPA to follow up and assess its programs adequately. Monitoring & Evaluation Project -­‐ FUNSEPA 4 The Sergio Paiz Andrade Foundation The Sergio Paiz Andrade Foundation (FUNSEPA) is a nonprofit organization established in 2004 with the purpose of contributing to the social and economic development of Guatemala through the use of technology as a tool for improving education. The Foundation was created in memory of Guatemalan businessman Sergio A. Paiz Andrade, who worked for the sustainable development of this Central American country using technology and education as key instruments. To this purpose, FUNSEPA focuses its efforts and activities in the provision of computer equipment to public schools and training teachers to effectively incorporate technology into traditional teaching. These initiatives are channeled through two key programs: Tecnología para Educar and Abriendo Futuro. Tecnología para Educar (TPE) started in 2006 and consists of providing public schools with computers, at no cost to the schools, which include a software with educational material. This software is donated by Microsoft and suited to strengthen skills in math, reading, and science. The Ministry of Education of Guatemala had previously approved this software for the purposes of the project implementation. The donated equipment may be new or used and refurbished in FUNSEPA’s Reconditioning Center. Additionally, a teacher is trained to maintain the computers and on what to do in case of computer failure. Through TPE, the Foundation has refurbished and delivered over 6,400 computers, directly benefiting more than 450 schools and approximately 150,000 children in the country. Abriendo Futuro (AF) was created in 2007 by the Ministry of Education, who appointed FUNSEPA as the executing branch of the program. This program aims to develop the technological capabilities of public school teachers in Guatemala so that they are better able to integrate the use of technology in the classroom. The training and resources developed for the program are based on the National Basic Curriculum of the Ministry of Education. As part of this program, the Ministry of Education facilitated the acquisition of personal computers for the teachers at affordable prices and with accompanying financing plans. TPE and AF are implemented separately, that is, there are schools that receive computer labs and their teachers are not necessarily trained through AF. Similarly, schools with teachers who have been trained by AF do not necessarily have a computer lab provided by FUNSEPA. For this reason and as later explained, this study considered appropriate to analyze the individual effects of the programs (TPE effect and AF effect), as well as the combined effect of these programs (TPE + AF). Monitoring & Evaluation Project -­‐ FUNSEPA 5 Literature Review The methodology for this evaluation is based on an extensive literature review that includes recognized studies on the impact of technology on education in both developed and developing countries. The purpose of this review was to use these studies as a guide for determining the statistical model and the variables required to perform a methodologically sound evaluation. Previous evaluations The evaluations discussed below were selected as a guide to assess FUNSEPA’s programs for three main reasons: (a) the main objective of these programs is to use technology to improve learning and teaching processes; (b) these studies evaluate existing programs, that is, they are not predesigned academic experiments; and (c) their general research question is whether access to technology improves students' academic performance. Additionally, these studies are traditional points of reference for academic and official publications, since they are methodologically sound and are based on longitudinal analyses—an evaluation methodology that observes the change of various individuals over a given period of time. While some of these studies have been conducted in the United States—a developed country with socioeconomic characteristics and an educational system that considerably differ from those of Guatemala—, the main purpose of using such studies is to observe the statistical model and the number and type of variables used to estimate the effect of technology on academic performance. In this sense, the literature review was conducted for methodological purposes and not based on socioeconomic similarities or differences between these countries. The literature review shows mixed results regarding the impact of technology on student performance. Although results are generally positive, many studies emphasize that the simple introduction to using computers or teacher training does not automatically improve academic performance. To generate a substantial positive impact, programs must ensure that teachers actually change their teaching methods and incorporate computers into their daily classroom activities, such as using the computer as support to teach general subjects and not only to teach how to use the computer itself. Because the adoption of information and communications technologies (ICTs) in education is a relatively recent phenomenon, there is no standard methodology or a universally accepted set of indicators for impact assessments. The few existing rigorous statistical evaluations to date suggest that implementation and program design are key factors to significantly affect academic performance. While factors such as standardized test scores are relatively easy to control for, other indicators necessary for a thorough evaluation—such as quality of education, teaching pedagogy, and critical-­‐thinking skills—are more difficult to measure.1 However, as more impact assessments are published, more successful models will emerge and better practices will be incorporated into the design and monitoring of future educational programs. 1
InfoDev (2005), “Knowledge Maps: ICTs in Education”, November, www.infoDev.org. Monitoring & Evaluation Project -­‐ FUNSEPA 6 In this sense, this evaluation has the ultimate goal of contributing to the documentation of the impact of this type of educational programs and thus collaborating with the improvement of existing and future programs. Computadoras para Educar – Colombia2 Computadoras para Educar (CPE) was initiated by the Ministry of Communications of Colombia in March 2002. The basic structure of the CPE program is very similar to that of the FUNSEPA programs, since they are based on public-­‐private partnerships and involve computer donations, restoration, and renovation of the donated equipment, as well as teacher training on the use of computers. The program was primarily designed to support education in literature, math, and science through the use of ICTs and to promote collaborative learning, creativity, and increased technological confidence in teachers and students. The evaluation was performed on 100 randomly selected schools previously interested in participating in the program, which were further divided into treatment and control groups. The questionnaires were administered to students, teachers, and school principals and several socioeconomic indicators—such as attendance, student grade, repetition rate, and desertion—
were collected. The program was successful in increasing the number of computers and the use of computers, but found no effect on test scores. Even after the donation of computers, teacher training, and technical assistance, teachers only taught basic computer skills and did not use the new technology in the teaching of school subjects. For this reason, the study highlights the importance of the application of technology in the classroom and evaluating effective techniques to ensure that teachers actually incorporate technology into their teaching. The Apple Classrooms for Tomorrow – United States3 The Apple Classrooms for Tomorrow (ACOT) project was initiated in five school sites across the United States with the purpose of assessing the impact of technology on teaching and learning. As in the case of FUNSEPA programs, ACOT provided students with computers at school and teachers with computer training. ACOT also provided computers to both students and teachers at home. The University of California—Los Angeles (UCLA) conducted a 3-­‐year controlled evaluation focused on those initial sites. UCLA researchers compared students’ basic skills performance to nationally reported norms, then compared student progress and achievement over time, and finally compared treatment classrooms with demographically similar classrooms. A final evaluation step was to gather data on classroom practices and parents’ background characteristics to explain student outcomes. 2
Barrera-­‐Osorio, F. & Linden, L. (2009). “The use and misuse of computers in education: Evidence from a randomized experiment in Colombia”. 3
Baker, E.L., Gearhart, M., & Herman, J.L. (1994). “Evaluating the Apple Classrooms of Tomorrow.” Monitoring & Evaluation Project -­‐ FUNSEPA 7 Student performance was assessed using different measures: a) achievement on standardized tests; b) performance in written composition; and c) student attitudes. The outcomes measured in the initial year of the project were used as baseline for comparison with the subsequent years of the study. Teacher outcomes were gathered through surveys and a quantitative classroom observation method. The UCLA team used both the observational data and survey data to test the effect of technology on student and teacher performance. The results of this study suggested that ACOT students had at least maintained their performance levels on standard measures of educational achievement in basic skills and had sustained positive attitudes. In the case of teachers, although results indicated that the ACOT project affected instruction and impacted teachers personally and professionally, it was not possible to identify causal changes. The West Virginia Basic Skills/Computer Education Program (BS/CE) – United States4 The West Virginia Basic Skills/Computer Education (BS/CE) program was an eight-­‐year program that started in 1990. BS/CE was created to provide computers to both students and teachers. The program began in the earliest grades and moved upward each year. The evaluation of the BS/CE program was conducted by the West Virginia Department of Education and the Milken Exchange on Education Technology in 1999. Student data was collected from all fifth graders in 18 elementary schools. Data for teachers was collected from surveys given to 290 teachers in the studied schools. Control groups were selected from schools in the same districts as the program schools. Student test data consisted of scaled scores based on the Stanford-­‐9 achievement test.5 Other variables used to explain student performance were student prior achievement in the Stanford-­‐9 test and student socio-­‐
demography. Because of time and budget constraints, neither control nor treatment groups were randomized. The study used an empirically derived model based on two of the components that will be also used in the case of FUNSEPA programs: access to technology and teacher training. Researchers added a third component, attitude toward technology, as researchers assumed that schools must believe that technology can help teachers and students improve their achievements for the program to be successful. The evaluation study found powerful positive effects with significant gains in reading, writing, and math. The BS/CE program helped all children perform better, but results also indicated that the program helped children that struggled in school the most. Low income and rural students and students without computers at home obtained the biggest gains. Overall results showed 4
Mann, D., Shakeshaft, C., Becker, J., Kottkamp, R. (1999). “West Virginia Story: Achievement gains from a statewide comprehensive instructional technology program.” 5 The Stanford-­‐9 is a standardized achievement test used by school districts in the United States to assess students from kindergarten through high school. It covers various subjects such as reading comprehension, mathematical problem solving, and science. Monitoring & Evaluation Project -­‐ FUNSEPA 8 that the BS/CE program accounts for 11% of the total variance in the basic skills achievement scores of fifth-­‐grade students. The IBM’s Reinventing Education (RE) Program – United States6 The IBM’s Reinventing Education (RE) is a grant program that started in 1994 and is focused on the adoption and adaptation of customized technology solutions. As in the case of FUNSEPA programs, computers and technology training are provided to students and teachers. The program was designed to solve some educational deficiencies in schools in ten initial sites across the United States. The three-­‐year evaluation of this program was conducted by the Center for Children and Technology (CCT) and was initiated in 1998 in the ten initial sites. As in the evaluations discussed above, the RE evaluators acknowledged that—as in any other school-­‐level program—
technology is rarely an independent variable and works instead as an amplifier enhancing other efforts. This issue was also considered when evaluating FUNSEPA programs. Researchers conducted the evaluation in three phases that combined qualitative and quantitative strategies. The first phase involved visits to the RE sites, interviews with IBM leaders, visits to participating schools, and review of internal documentation about the technology and its use. The second phase consisted of the selection of key indicators: standardized test scores, teaching practices, and school-­‐parent contacts. The third phase included instrumentation and data gathering. The evaluation found positive impacts of technology on learning and teaching. Where technology was used to teach core disciplines, there were significant gains in grades 7 through 11.7 At the same time, students with an initial high performance maintained their performance throughout the evaluation. They also found significant and consistent improvements in reading levels, word recognition and comprehension relative to similar students that were not involved in the program. Teacher training, professional development, and process and organizational changes within the school districts were also positively affected by the RE program. Literature Review Implications This literature review served to shape the design and implementation of this evaluation. Among other things, this literature review influenced the following areas of the assessment of the FUNSEPA programs: Methodology: the review of previous evaluations helped cement the importance of combining qualitative with quantitative techniques in order to properly interpret statistical results. Additionally, the rigor of these assessments led to the use of two groups of analysis and the selection of a random sample with a size statistically proportional to that of the real 6
Center for Children and Technology (2001). IMB Reinventing Education: Research Summary and Perspective. 7 Equivalent to grades 1 to 3 of básica and 1st of diversificada in Guatemala. Monitoring & Evaluation Project -­‐ FUNSEPA 9 population—this with the purpose of ensuring that the results of this evaluation are robust and representative. Hypothesis: the results achieved by many of these evaluations helped establish that the introduction of technology in education itself can have an impact on students' academic performance and that, therefore, it is plausible to consider as a hypothesis that the FUNSEPA programs can have some effect on the grades of students in participating schools. It was also feasible to consider that, beyond grades, FUNSEPA programs could have some effect on dropout rates. Control Variables: the literature review allowed for the exploration and determination of factors and variables, besides the FUNSEPA programs, that could influence changes in student scores and therefore needed to be included in the final model. Among these factors are: class size, number of computers available per student, number of trained teachers per student, and poverty levels. Finally, the FUNSEPA evaluation was based on some of the characteristics of these previous evaluations and has adapted and incorporated them into a rigorous evaluation that ultimately aims at contributing to this effort of scientifically understanding the impact of technology on education.
Monitoring & Evaluation Project -­‐ FUNSEPA 10 Methodology This evaluation is comprised of qualitative and quantitative elements. The qualitative assessment was performed prior to the quantitative analysis to allow for a better understanding of how FUNSEPA programs are implemented and what changes could be expected from their implementation. The quantitative evaluation was performed in order to estimate the effect, over time and controlling for a number of socioeconomic factors, of FUNSEPA programs on academic performance. Groups of Analysis TPE: This group includes students in schools that have received computer labs through Tecnología para Educar. AF: This group includes students in schools where teachers received training on computer skills through Abriendo Futuro. BOTH (TPE+AF): This group includes students in schools that received computer labs through Tecnología para Educar and also received training for their teachers through Abriendo Futuro. Comparison: This group includes students in schools that have not received any of the FUNSEPA programs. QUALITATIVE EVALUATION The project involved conducting a series of surveys in order to understand how computers and skills from teacher training are actually articulated in the regular activities of the schools benefited by FUNSEPA. The purpose of this qualitative assessment is to give context to the quantitative results. The qualitative instruments were administered on three main stakeholders: students, teachers, and school principals. It is important to mention that it was not possible to interview the same number of teachers, students, and principals in all visited schools. In some cases it was only possible to interview teachers and students, while in other cases only principals were interviewed. The following table shows the total number of respondents for each group of interest and the percentage of respondents in schools that received only the AF program, in schools that received only the TPE program, and in schools that received BOTH programs (AF + TPE) respectively: Monitoring & Evaluation Project -­‐ FUNSEPA 11 Table 1. Teachers, Students, and Principals Interviewed Teachers Students Principals 18 34 6 % in AF schools 5 (28%) 9 (26%) 0 (0%) % in TPE schools 3 (17%) 6 (18%) 1 (17%) % in AF + TPE schools 10 (56%) 19 (56%) 5 (83%) Total of interviewees Additionally, the qualitative evaluation included interviews with stakeholders who are not necessarily directly linked to FUNSEPA activities, but who are involved in activities related to the improvement of education in Guatemala. The purpose of interviewing these external stakeholders was to understand what are the most critical issues in the area of education in Guatemala and thus better contextualize the quantitative results of the study. External stakeholders interviewed include the Ministry of Education, USAID Guatemala, the Research Triangle Institute (RTI) -­‐ Alianzas, Empresarios por la Educación, the World Bank, and the Inter-­‐American Development Bank, among others. To see the complete list of external stakeholders interviewed for this study, please see Appendix 1. QUANTITATIVE EVALUATION This quantitative evaluation is a quasi-­‐experimental study comprised of two main groups: one group affected by FUNSEPA programs (hereinafter called the "FUNSEPA group") and a comparison group not affected by FUNSEPA programs (hereinafter the "comparison group"). The purpose of this technique is to compare the difference observed in the FUNSEPA group before and after implementation of the programs with the observed difference in the comparison group during the same period of time. This technique allows for an approximation to what would have been the performance of students in FUNSEPA schools if the programs had not been implemented. In this way, we can conclude that any difference between the FUNSEPA group and the comparison group comes from the FUNSEPA intervention. The FUNSEPA group includes students in public elementary schools who received computer labs from FUNSEPA (through TPE), students in schools that have trained teachers by FUNSEPA (through AF), or student in schools that received both programs (AF+TPE). The comparison group includes students in schools with comparable characteristics of those in the FUNSEPA group, that is, public elementary schools with similar demographic characteristics but that have not received any of the FUNSEPA programs. Random Sampling Methodology & Sample Size The selection of the sample for the evaluation was performed following a two-­‐stage random sampling technique: schools were first randomly selected and then data was collected for all students in selected schools. The sampling was done in this way because of the inability to Monitoring & Evaluation Project -­‐ FUNSEPA 12 access a complete list of the students, both in FUNSEPA schools and comparison schools, from which to select individuals ex-­‐ante. The evaluation gathered data for all students in each school until the sample size reached a minimum of 1,067 students per group, as this is the minimum size acceptable for the results to be statistically significant and representative. In other words, data was collected for at least 1,067 students to evaluate TPE, for at least 1,067 to evaluate AF, and for at least 1,067 to evaluate BOTH programs. This means the final sample exceeds 6,000 students. This sample was calculated considering the total population of students in public elementary schools in Guatemala. The formula for calculating the minimum sample size was based on a confidence level of 95% and a margin of error of 5%.8 The following table shows the total number of students per program from which data was collected for this evaluation: Table 3. Distribution of Students and Schools per Group of Analysis Group of Analysis Students Schools TPE AF AF + TPE Comparison 1,243 1,075 1,315 2,456 9 7 9 5 Total 6,089 30 Note: The complete list of schools identified with its respective UDI codes per group of analysis can be found in Appendix 2. Data Collection Methodology & Sample Characteristics The final sample included academic and socio-­‐demographic information of the students in both groups of study for the years 2005 to 2010. Year 2005 served as baseline to evaluate TPE and year 2006 to evaluate AF. This timeframe allowed for sufficient time for the effect of the programs to be observed and thus for it to be captured by this evaluation. The final set of variables included in the statistical analysis is listed in Appendix 3. The data utilized in the evaluation was collected from several sources. Randomly selected schools provided the academic records (PRIM tables) of elementary students from 2005 to 2010. The Department of Educational Planning (DIPLAN) of the Ministry of Education provided the dropout and promotion rates for selected schools between 2005 and 2010. The General Department of Educational Evaluation and Research (DIGEDUCA) provided the results from standardized tests applied to some of the schools in the sample between 2006 and 2009. Finally, the National Institute of Statistics (INE) and the Inter-­‐American Development Bank (IADB) provided statistics on poverty levels in the country for the years 2000, 2006, and 2011. 8
The confidence level means that we can be 95% certain that the answer we received from our sample accurately represents that of the general population. The standard error is the margin error of obtaining an answer that would not necessarily be present in the true population. Monitoring & Evaluation Project -­‐ FUNSEPA 13 Hypothesis & Statistical Model Specification The general question this evaluation seeks to answer is: what is the impact of the FUNSEPA programs on academic performance? More specifically, in the case of Abriendo Futuro, the question is: what is the effect of training teachers on the use of the computers on the academic performance of their students? In the case of Tecnología para Educar, the question is: what is the effect of providing computer labs to Guatemalan public schools on the academic performance of students in benefited schools? The evaluation looks to answer these questions from different perspectives. First, the evaluation observes the individual effect of each program and then it looks at their combined effect; that is, when schools have participated in a single program (AF or TPE) and when they have participated in both programs (AF+TPE). In this sense, the main assumption of this analysis is that having access to computers will expand the resources available to students at schools to increase their knowledge about certain subjects, to conduct better research and complete assignments, and to develop new skills related to the use of the technology itself. All these factors may ultimately contribute to improvements in academic performance. Likewise, providing computer training to teachers will help them learn new pedagogic techniques, prepare classes and class material in a more didactic way, and help their students with the use of technology itself. The assumption in this case is that the familiarization of teachers with the technology through training will have an impact on their students’ learning process and should therefore generate a positive effect on their academic performance. In order to test these assumptions and to estimate the effect of the programs, a main baseline model was used to analyze the data. This model allows for estimating the individual effect of each program, as well as their combined effect. The model is represented by the following equation: yit = β0 + β1FUNSEPAit + uit
where yit is the academic performance of student i in year t in the school where teachers were trained through Abriendo Futuro, in the school that received computer labs from Tecnología para Educar, or in the school that received BOTH programs. More specific models derived from this general model are explained in detail below. Model 1. TPE Effect on Academic Performance: This model estimates the individual effect of Tecnología para Educar on the academic performance of students in schools that only received TPE and in relation to the academic performance of students in comparison schools. The following equation describes the fixed effect model used to estimate this effect: Monitoring & Evaluation Project -­‐ FUNSEPA 14 t
i
FinalGradeit = β 0 + β1TPE it + γ it + ∑δYearit + ∑α i + µit 2005
2
where FinalGradeit is the academic performance of student i in year t in the school that only received computers from TPE, controlling for variables γit. €
Model 2. AF Effect on Academic Performance: This model estimates the individual effect of Abriendo Futuro on the academic performance of students in schools that only received this program, relative to the academic performance of students in comparison schools. In this case, a fixed effect model was also used to estimate this effect and is described by the following equation: t
i
FinalGradeit = β 0 + β1 AFit + γ it + ∑δYeart + ∑α i + µit 2005
2
where FinalGradeit is the academic performance of student i in year t in the school where teachers received the AF training, controlling for variables γit. €
Model 3. BOTH (TPE + AF) Effect on Academic Performance: This model estimates the combined effect of BOTH programs, Tecnología para Educar and Abriendo Futuro, on the academic performance of students in schools that received computer labs from TPE and also received training for their teachers from AF. In this case, a fixed effect model was also used as described by the following equation: t
i
FinalGradeit = β 0 + β1BOTH(TPE + AF) it + γ it + ∑δYeart + ∑α i + µit 2005
2
where FinalGradeit is the academic performance of student i in year t in the school that received computers from TPE and also received training for its teachers from AF, controlling for €
variables γit. Demonstrative Schools of the Future The Demonstrative Schools of the Future (in Spanish, Escuelas Demostrativas del Futuro -­‐ EDF) are schools that received the FUNSEPA programs but that, during their inception, counted with important support from the Ministry of Education of Guatemala for the implementation of the programs. Among other resources, this support consisted in providing a more prolonged technical training for teachers and training a greater number of teachers at the school than is currently provided to schools benefited by FUNSEPA. The EDF project was implemented until 2007 due to the financial burden it entailed for the Ministry. More specifically, EDF schools received technical training on the use of computers for nine months. This training did not include any pedagogical component, but was instead focused on mastering the technology. Additionally, all teachers in EDF schools had to participate in the Monitoring & Evaluation Project -­‐ FUNSEPA 15 training and not only interested teachers. In this way, EDF schools received an initial support that is significantly different than that received by non-­‐EDF schools. Therefore, it is plausible that schools that received the FUNSEPA programs and also benefited from this additional EDF training could show a significantly higher effect on student academic performance than schools that did not receive this additional training. In an attempt to understand if there is in fact any important difference between EDF schools and other schools benefited by FUNSEPA, this evaluation also explored the effect of the FUNSEPA programs (TPE, AF, or BOTH) when schools are EDF schools. To this purpose, the evaluation utilizes models 1, 2, and 3 described above to determine such effect. Other Statistical Models Even if they do not respond to the main research question of this evaluation, the statistical analysis also utilized other models to explore if the FUNSEPA programs could have any effect on other variables beyond academic performance. More specifically, these models looked to identify any effect on student dropout levels and the probability of students being promoted to the next grade. Model 4. Effect on Dropout Rates: This model estimates the effect of the FUNSEPA programs on dropout rates reported by the Ministry of Education at the end of each year. A fixed effect model was used to conduct such estimation, alternating the independent variable to observe first the individual effect of TPE and AF and then the combined effect of BOTH (TPE+AF) programs. The following equation describes this model: t
i
Dropout it = β 0 + β1FUNSEPAit + γ it + ∑δYeart + ∑α i + µit 2005
2
where Dropoutit is the dropout rate in the school of student i in year t that only received computers from TPE, that only received training for its teachers from AF, or that received BOTH € controlling for variables γit. programs, Model 5. Effect on Probability of Students Being Promoted to the Next Grade: This model estimates the effect of the FUNSEPA programs on the probability of students being promoted to the following grade. To this purpose, a binary varible (passed) was created and equals 1 if the student achieved a final grade equal or greater than 60 points.9 To estimate this probability, a fixed effect logit model (xlogit, fe) was applied and marginal effects (mfx) were later calculated, as described by the following equation: 9
According to the PRIM tables, the minimum grade to pass a subject or the school year is 60 points out of 100 total points. Monitoring & Evaluation Project -­‐ FUNSEPA 16 t
i
δPassedit = β 0 + β1FUNSEPAit + γ it + ∑δYeart + ∑α i + µit
2005
2
where Passedit is the probability that student i in year t, in the school that received any of the FUNSEPA programs (TPE, AF or BOTH), is promoted to the next grade with a score equal to or € 60 points, controlling for variables γit. greater than Model 6. Score Difference in Standardized Tests: The General Department of Educational Evaluation and Research (DIGEDUCA) of the Ministry of Education applies every year, since 2006, standardized tests to public schools nationwide. These standardized tests assess the performance of students in math and reading. Evaluated schools are randomly selected every year, as well as the grade and class to be evaluated. In other words, not all students in the school take the DIGEDUCA test. DIGEDUCA provided the results of these tests for schools included in this evaluation sample. Because the sample of schools from the FUNSEPA and comparison groups selected by DIGEDUCA is small, the estimation of the effect of the FUNSEPA programs on test scores was limited to an analysis of the statistically significant difference between the score obtained by FUNSEPA schools and the score achieved by comparison schools. In this sense, this analysis aims at determining whether schools benefited by FUNSEPA achieved a score in math and reading that is statistically higher, equal, or lower to the score attained by comparison schools. The following equation describes this model: +
AchievementFUNSEPA = AchievementNonFUNSEPA −
Control Variables €
The models described above include a series of variables to control for the effect of factors other than participation in FUNSEPA programs that also have an influence on academic performance. Although not all models use the same set of variables, they in general control for the following factors: Class Size. Number of students in the classroom for a given year. The more students in a classroom, the less attention the teacher can give to students and this can affect their academic performance. Furthermore, more students in the classroom means that students must either share a computer with other students or spend less time in the computer lab as the class needs to be divided into two groups—and this can certainly affect the time that the student is exposed to the technology. Similarly, this affects the teacher's ability to incorporate technology into their lessons to reinforce concepts taught in the classroom. Because all this can affect academic performance, it was necessary to control for the size of the classroom of the student. Grade Level. Grade level of the student in a given year. It is possible that in lower grades students are more interested in studying than in higher grades. It is also possible that in lower Monitoring & Evaluation Project -­‐ FUNSEPA 17 grades students are more interested in using technology to reinforce concepts learned in the classroom, while for higher graders the technology may no longer be a novelty. Moreover, higher graders can begin to feel pressured to enter the job market. Students who repeat grades may also be discouraged to study. Because all this can affect students’ academic performance as they move up to higher grades, this evaluation considered important to control for this factor. For the purpose of this analysis, this variable will be interpreted as the change in academic performance as the student is promoted to the next grade. Gender. Gender of the student. Some studies suggest that girls tend to be more studious than boys and that boys tend to begin working earlier than girls. For this reason, it is important to control for the gender of the student so as to determine whether there is in fact a significant difference in academic performance given that the student is a female or a male. Number of Trained Teachers. Number of trained teachers through AF in a given year. It is possible that FUNSEPA programs have a greater effect when a greater number of teachers are trained on incorporating the technology in their traditional teaching for a given school and a given year. Therefore, it is feasible that the more teachers incorporate technology in their daily teaching, the greater the academic performance of their students will be. For this reason, the functional form of this variable was created to be the proportion of trained teachers per student. Number of Donated Computers. Number of computers donated by FUNSEPA through TPE. If the number of donated computers is not sufficient—relative to the average class size for a given school—the effect of technology on the student’s academic performance could be smaller, as each computer would need to be shared by at least two students or be used for less time. The functional form of this variable was also changed to represent the proportion of donated computers per student. Dropout. Dropout rate in the school of the student for a given year. It is possible that dropout patterns in the school discourage students if, for example, the culture in the community where the school is located values more economic productivity than academic productivity. It is thus feasible that a student is more likely to drop out of school if one or several of his/her classmates dropped out and now works and economically supports his/her family. Poverty. Incidence of poverty in the department in which the school is located for a given year. The higher the poverty levels, the higher the pressure on the student to become economically active to help his/her family. This can certainly affect students’ academic performance if they need to work while studying or if they need to temporarily or permanently abandon their studies to work. NOTE: The specific set of control variables used in each model will be reported in the results section, as well as in the output tables from the statistical software in the Appendix. Monitoring & Evaluation Project -­‐ FUNSEPA 18 Methodological Limitations The methodology used in this evaluation is considered as one of the most robust techniques for quasi-­‐experimental studies. Nonetheless, there is a series of factors and conditions that limit the capacity of generalizing the results obtained through this methodology. The following are the main limitations identified for this study: The study has been carried out in a real context, meaning that factors affecting academic performance are difficult to control for with absolute precision. Additionally, because the selection of schools into the FUNSEPA programs is not performed randomly, the model cannot control entirely for factors that have an effect on academic performance (for example, personal motivation). Nonetheless, the evaluation is controlling for some of this influence by randomly selecting the evaluation groups, using a comparison group, and analyzing results through a fixed effects method. Moreover, the model included as many control variables as possible that could influence academic performance, such as class size and poverty levels. Schools voluntarily participate in the FUNSEPA programs. This makes the comparison between FUNSEPA schools and non-­‐FUNSEPA schools less accurate, as participating schools understand the importance of having a computer lab or trained teachers to enhance the academic performance of students. Their interest in participating can come from these schools being in better conditions or having greater resources—teachers better trained or more qualified, better informed principals, schools with better infrastructure, or with the capacity to install the infrastructure necessary for the computer labs—than schools that do not participate. These differences may mean that the FUNSEPA and comparison groups are not entirely similar. Nonetheless, the sampling was conducted in a way that mitigated this problem. First, the schools in the sample were randomly selected from a full list of schools benefited by FUNSEPA. Second, to guarantee that the comparison was done between groups with similar characteristics, schools in the comparison group were also randomly selected from a list of schools interested in participating in the FUNSEPA programs, but that, for some reason, have not received the programs yet. This allowed the evaluation to compare “apples to apples”. The analysis does not control for additional programs in which the schools—FUNSEPA schools and comparison schools—could have participated during the period of study. If benefited schools have participated in programs similar to FUNSEPA programs during the period of analysis (2005-­‐2010), the results of this study would be overestimating the real effect of the FUNSEPA programs on academic performance. If, on the contrary, comparison schools participated in similar programs during the period of the study, the results would then be underestimating the real effect that FUNSEPA programs have on academic performance, assuming those programs had a positive impact on control group schools. The effect on academic performance has been estimated using grades from non-­‐standardized tests, that is, students have not been evaluated using the same test. This implies that the academic performance of students cannot be easily compared since the evaluated material, the Monitoring & Evaluation Project -­‐ FUNSEPA 19 level of difficulty, and the time to complete the test can vary considerably across schools. Although it is difficult to control for this variation, the Ministry of Education of Guatemala establishes a minimum set of standards that school curriculums and evaluation plans must comply with. These standards include the content per subject that needs to be covered both in the classroom and in evaluations. It is possible then to assume that this “standardization” in the content for the classes and for the evaluations mitigates the variation across schools. The level of transfer of public school teachers from one school to another tends to be high in Guatemala. This implies that teachers that were trained by FUNSEPA in a given year may have transferred to other schools in subsequent years and this analysis does not control for these transfers. The severity of this limitation depends on whether the teachers transferred to a school outside the sample, to another FUNSEPA school within the sample, or to comparison schools in the sample. If trained teachers were transferred to other FUNSEPA schools in the sample, this limitation does not have severe implications since the FUNSEPA effect is kept within the intervention group. If trained teachers transferred to schools outside the sample, this factor is not as serious because the program trains teachers to reproduce the skills acquired and train other teachers in the school. In this sense, the effect stays in the school to a certain extent, though not with the same intensity. On the contrary, if trained teachers transferred to schools in comparison schools in the sample, the results would be underestimating the effect of FUNSEPA programs on academic performance. The poverty levels used in this study are linear projections built with the poverty incidence by Guatemalan departments reported for 2000, 2006, and 2011 by the National Survey of Living Conditions (ENCOVI), collected by the National Institute of Statistics of Guatemala. These projections were created to include estimations of poverty for every year in the study. A limitation of creating this variable as a linear projection is that the model assumes that the variation in poverty across time increased or decreased progressively without important fluctuations. The variable thus does not capture increases or decreases in poverty levels due to specific conditions in the political and socioeconomic context of Guatemala. This implies that the estimations reported in this study do not capture these fluctuations, but do capture the general change in academic performance given a certain poverty level across time and across schools. This study is an ex-­‐post analysis, that is, the analyzed data was collected after the implementation of the FUNSEPA programs. This implies that the FUNSEPA schools and the comparison schools are currently similar in regards to their interest in having a computer lab and trained teachers. Nonetheless, comparison schools may have not been similar to the FUNSEPA schools in the first years of the implementation of the programs—for example, these schools may have not met infrastructure prerequisites needed to participate in the programs during those initial years. Monitoring & Evaluation Project -­‐ FUNSEPA 20 If the comparison schools are in fact systematically different than FUNSEPA schools, this evaluation could be overestimating the effect of the FUNSEPA programs. However, average grades and average poverty levels are similar in both groups before and after the implementation of the programs, meaning that we could assume that schools in the sample have similar characteristics in terms of infrastructure and staff or that they do not differ systematically during the studied period. This study controls for the number of trained teachers but not for the number of teachers effectively competent to incorporate the technology in the classroom. Controlling for the level of competency or skillfulness of teachers at the end of the FUNSEPA trainings would make the estimation of results more robust. If at the end of the training, teachers are still not capable of effectively incorporating technology in their teaching, or if the acquired skills vary significantly across teachers, then the models could be overestimating the effect of the FUNSEPA programs by simply controlling for the total number of teachers trained in a given year. Finally, the analysis does not include information on the family environment of students. The results of the study could be more robust if household characteristics were controlled for. These characteristics include the income level, education achievement, and occupation of the parents of the student, among others. These variables are especially relevant to estimate the effect of the FUNSEPA programs on school dropout rates. Monitoring & Evaluation Project -­‐ FUNSEPA 21 Evaluation Results QUALITATIVE RESULTS This section presents the results of the interviews with students, teachers, and principals in schools benefited by FUNSEPA, as well as with external stakeholders interested in the improvement of public education in Guatemala. Perspective of School Principals The principals interviewed have on average between 7 and 9 years in the position of principals. Fifty percent (50%) of them said to have 10 or more years in this position, while 33% said to have at least 4 years as principals. The remaining 17% has 3 or fewer years holding this position. All interviewed principals indicated they have the double function of managing the teachers and the curriculum plan for the school. They also estimated that more than 80% of their teachers received training from AF to successfully incorporate computers in their teaching. All visited schools (100%) received their first computer lab from FUNSEPA and 33% received new computers from other organizations (mostly from the Ministry of Education) after the implementation of the FUNSEPA program. Another 17% acquired new computers with their own funding. Interviewed principals had the impression that by improving the infrastructure of the school to receive computers from TPE or already having in place a computer lab provided by FUNSEPA, the schools could access new donations or funding that otherwise they could have not obtained. One of the principals interviewed stated: “the school received additional computers from the Ministry of Education because we already had a lab in place to install a computer […] and we also received program Modelo 3:110 through which we could have more students using the same computer because we already had computers.” Therefore, the expansion of opportunities to access greater resources and consolidate the use of technology in the classroom is certainly a non-­‐intentional, positive effect of the FUNSEPA programs. 10
Modelo 1:3 is a project implemented by the Ministry of Education, in partnership with EDUINNOVA from the Pontifical Catholic University of Chile and Microsoft. This is an innovative model designed to have 3 mice connected to the same computer. The screen is divided into three equal parts so that each student has a work area to perform the activities requested by the teacher. Monitoring & Evaluation Project -­‐ FUNSEPA 22 Figure 1. Computer Provider Organizations and Trained Teachers Perspective of Teachers Interviewed teachers have on average almost 5 years working as educators and most of them (67%) said they have 7 years of experience. Most of the interviewees teach first grade (44%), work more than 21 hours a week (67%), and have more than 25 students per class (78%). Eighty-­‐nine percent (89%) of interviewees teach more than two subjects—for the most part, basic subjects such as math, literature, and science. Fifty percent (50%) of interviewed teachers received training from Abriendo Futuro and over 80% of these teachers received the basic training. From the total trained by AF, 67% considered they have incorporated the majority of the training in their daily teaching and 73% said to use the computer lab to teach between 1 to 2 times a week. Sixty-­‐seven percent (67%) of teachers considered that all subjects—and not only math, literature, and science—can be successfully taught using a computer. Figure 2. Teachers trained by Abriendo Futuro and Learning Incorporated by Trained Teachers Among the limitations mentioned by interviewed teachers are the lack of additional training and further investment in computer labs, as well as the lack of computer connection to the Monitoring & Evaluation Project -­‐ FUNSEPA 23 Internet. Teachers expressed that although the AF training was an important step for incorporating technology in their teaching, it was not sufficient to significantly improve their teaching and the learning process of their students. Seventy-­‐eight percent (78%) of teachers think this training should have had continuity to more advanced levels. Some teachers commented: “often times students know more than us and that makes it difficult to incorporate the technology in regular classes in a successful way.” Fifty-­‐six percent (56%) of interviewed teachers commented that schools rarely make additional investments in continuing teacher training or in improving computer labs. Teachers also mentioned that the lack of connection to the Internet in computer labs limits research capabilities for both teachers and students. Additionally, interviewed teachers said the number of computers in the lab is not sufficient to adequately serve students. A teacher commented: “we can only espose students for a maximum of two hours a week because they are so many that we have to divide the class in two groups and take them in different days; in other cases, we have to divide the class and only take them to the computer lab for a half hour to give each group the opportunity […] and because we want all grades to go at least once a week, we cannot extend the times of each class because otherwise some of them will not be able to use the lab.” Figure 3. Perception on AF Training and Additional Investments Perspective of Students Most of the interviewed students (56%) were in fifth or sixth grade, named math as their favorite subject (53%), and said they would like to be a teacher when they grow up (29%). All interviewed students said Spanish is the language they normally use both at school and at home. Students also indicated that they learn other languages at school such as English (65%) or some indigenous dialect like cachiquel (18%). Seventy-­‐one percent (71%) of students said they do not have computers at home, while 56% said they use computers outside the school (i.e. cyber cafes) mainly to do homework or study (89%), browse the Internet (84%), and play video games (74%). About 85% said they use the Monitoring & Evaluation Project -­‐ FUNSEPA 24 computer lab for their regular classes, mostly for math (72%), literature (59%), and science (41%) classes. Figure 4. Percentage of Students with Computers at Home & Classes Taught at the Computer Lab In contrast to teachers’ comments, students believe that not all subjects are adequate to be taught using computers. Fifty-­‐six percent (56%) of the students consider that math and science are the most suitable subjects to learn using a computer, while only 29% of them think that literature is adequate to be taught using the technology. All interviewed students (100%) think the use of the computers has helped them learn better and 85% of them said they apply what they learn in the computer lab outside the school. Perspective of External Stakeholders The study consulted several organizations that although are not directly related to FUNSEPA programs and activities, they are involved in the mission of improving public education in Guatemala. These organizations include public institutions, nonprofit organizations, and multilateral entities. In general, these external stakeholders were asked about the most important challenges faced by public education in Guatemala and more specifically about the benefits of incorporating the use of technology in education. The following points summarize the most important factors mentioned by the interviewed stakeholders: 1. Education quality is as or more important than coverage. All interviewees agreed that although it is important to ensure that every child in Guatemala has access to education, the quality of it is perhaps more important. Felix Alvarado, Education and Health Advisor at Academy for Educational Development (AED), 11 noted that "one of the problems is that access to education is not yet universal in Guatemala, but a more serious problem is the quality of education as even those who access it seem not learn enough." 11 AED currently functions under the name of FHI 360 and is a nonprofit organization focused on human development. For more information on FHI 360, visit: http://www.fhi360.org/ Monitoring & Evaluation Project -­‐ FUNSEPA 25 A similar view was expressed by Cynthia del Aguila, Education Manager at the Research Triangle Institute (RTI) – Alianzas, who said that "rather than the quantity, improving the quality of education is essential and it has to do with increasing resources—such as reading materials, math exercise books, and school infrastructure, among others—as well as training teachers." 2. Technology is a tool, not the solution itself. Stakeholders thought that having a computer is not the solution, but a means to improve education levels. This is because the effectiveness of technology on education is directly related to how the computer is incorporated into everyday teaching of core subjects, as well as the preparation level of teachers. Carlos Perez-­‐Brito, Human Development Officer at the World Bank, noted that "most teachers in the public sector only have secondary education and have very limited technical knowledge to be able to incorporate technology into school lessons successfully." Armando Godinez, Education Senior Specialist at the Inter-­‐American Development Bank (IDB), added that "a significant part of the programs incorporating technology in education are wrong in assuming that technology changes everything and when they include some teacher training component, these programs often limit the training to providing skills on how to use Microsoft Office instead of incorporating a comprehensive educational guidance on using the computer to teach." An interesting point was expressed by Luisa Müller, General Director at the General Department of Education Evaluation and Research (DIGEDUCA) of the Ministry of Education of Guatemala, who noted that some of the assessments made by the Ministry found that while technology alone has in fact no effect on the improvement of education, "the computers themselves seem to be a palliative to student dropout." At the end of the survey stakeholders were asked to provide recommendations for FUNSEPA to improve its programs and these were their main suggestions: •
Integrate both programs instead of providing them separately. As explained above, interviewees consider that technology is not effective by itself and thus must be accompanied of a plan to train teachers on how to successfully incorporate the use of the computer in their daily teaching. •
Focus on quality as opposed to quantity. Interviewees think that beyond taking computers to a great number of schools, FUNSEPA should dedicate more resources to assuring the effective incorporation of computer usage in curricular plans of benefited schools. •
Looking for partnerships that permit expansion without sacrificing quality. Stakeholders understood that as a nonprofit organization with limited resources, efforts to expand program coverage may often compromise the quality of the programs. Therefore, interviewees recommended establishing partnerships with organizations that have greater capacity to scale up, such as regional governments or municipalities, in order to expand the programs without sacrificing quality. Monitoring & Evaluation Project -­‐ FUNSEPA 26 QUANTITATIVE RESULTS The statistical analysis shows that, in general and relative to comparison schools, the FUNSEPA programs have a positive effect on the academic performance of students in benefited schools. When analyzed separately, the program that has the greatest effect is Tecnología para Educar, followed by the individual effect of Abriendo Futuro (see Appendix 4). The combined effect of BOTH programs seems at first lower than the individual effect of TPE and AF. Nonetheless, this is because for schools that receive both programs, TPE and AF are in practice implemented independently. In other words, the programs are not necessarily implemented simultaneously in a given year, but there may be a significant time gap between the implementation of one and the implementation of the other. When the second program is implemented, the effect of BOTH programs seems smaller because there has already been an initial benefit on academic performance generated by the first program. For this reason, the effect of BOTH programs could be interpreted as a marginal effect in addition to the effect already generated by the first program (see section BOTH Programs Effect for more information). General Implication of the Quantitative Results In general, the FUNSEPA programs increase the final grades of students in benefited schools by 1 to 4 points compared to the grades of students in comparison schools. This increment is a substantial effect, especially considering that students in the sample use the technology only 1 to 2 hours per week. This effect is also significant considering the relatively small investment to provide schools with approximately 16 computers (equivalent to 0.011 computers per student) and training on average 2 teachers per school (equivalent to 0.049 trained teachers per student). The TPE Effect Students in schools that only received computers from TPE increased their grades by 1.32 points, in contrast to students in the comparison group. This result is statistically significant and was obtained controlling for class size, student grade, number of computers per student, and poverty levels (see Appendix 5). The individual effect of TPE in specific subjects is equally positive and significant, with the effect on math scores being the greatest. Students in schools with TPE received math grades that were 3.31 points higher than students in schools not benefited by TPE. In the case of literature, FUNSEPA students scored 3.26 points more than students in comparison schools. In science, grades increase by approximately 1.73 points. For these results, the model controlled for class size, school grade, number of computers per student, and poverty levels (see Appendix 6). Monitoring & Evaluation Project -­‐ FUNSEPA 27 In this model, as well as in other models analyzed in this evaluation, it is interesting to observe that the FUNSEPA programs have a greater effect on math scores than on literature or science scores. This difference could be related to a greater emphasis given by teachers—intentional or not—to this subject when incorporating the technology into the classroom. The results of the qualitative evaluation seem to validate this assumption, as interviewed students said to use the computer lab mostly for their math classes (72%). Table 3. Results from the TPE Effect on Final Grades TPE Effect per Subject Final Math Literature Science Average Change in Grades + 1.32 points + 3.31 points + 3.26 points + 1.73 points In this model, it is also important to mention the effect of some control variables on the academic performance of students. It is interesting to observe that as students move up to higher grades, their final grade decreases. This could be related to students losing interest in school, the computer no longer being a novelty, or students beginning to engage in different activities such as work. It is also interesting that as the proportion of computers to students increases, the academic performance of students also increases. In other words, as fewer students have to share the same computer, the effect on their academic performance is greater. Additionally, the model explored if gender had any influence on students’ final grade or subject grades. The analysis found that gender is not a significant factor in the academic performance of students (see Appendices 5, 6, and 11). The TPE Effect on Demonstrative Schools of the Future The Demonstrative Schools of the Future (EDF) are schools that received the FUNSEPA programs but that in their initial phase had significant support from the Ministry of Education of Guatemala for their implementation. In an attempt to understand if there is in fact a substantial different between EDF schools and other schools benefited by TPE, this evaluation explored the individual effect of this program on EDF schools. When schools are EDF and received TPE, the academic performance of students increases by approximately 2.86 points. In other words, the TPE effect increases from 1.32 points to 2.86 points (or additional 1.5 points) if the school is EDF. This result is statistically significant and was obtained controlling for class size, school grade, number of computers per student, and poverty Monitoring & Evaluation Project -­‐ FUNSEPA 28 levels (see Appendix 11). 12 These results are particularly important from a programmatic perspective, as they suggest that the individual effect of TPE could be significantly higher if all teachers received a longer training on the use of the technology. The AF Effect AF also showed statistically positive results on school grades. Students in schools with teachers trained by AF scored on average 1.19 points more than students in comparison schools. In this case, the model controlled for class size, proportion of trained teachers per student, and poverty levels (see Appendix 7). The results per subject are also positive and statistically significant. Students in schools benefited by AF obtained 1.85 points more in math grades than students in comparison schools. In the case of literature, students in AF schools achieved 1.55 points more in this subject and 1.28 points more in science than students in the comparison group. These results were obtained controlling for class size, school grade, proportion of trained teachers per student, and poverty levels (see Appendix 8). Table 4. Results from the AF Effect on Final Grades AF Effect per Subject Final Math Literature Science Average Change in Grade + 1.19 points + 1.85 points + 1.55 points + 1.28 points In the case of this model, it is also relevant to discuss the effect of some of the control variables. As in the TPE model, final grades decrease as students move up to higher grades. As explained before, this can be due to students in higher grades losing interest in school, feeling that computers are not longer a novelty, or that they are taking on new responsibilities outside the school such as work. Likewise, it is interesting to observe that as the proportion of trained teachers per student increases, final grades also increase. In other words, the more trained teachers per student in the classroom, the greater the effect on academic performance. Most of these control variables are statistically significant. As in the case of TPE, gender did not show any significant effect on academic performance in this model (see Appendix 7 and 8). 12 Although these results are statistically significant, the sample size does not permit to generalize the results to all public school students in Guatemala. The model was applied to 889 students in EDF and non-­‐EDF schools that received computer labs from TPE—a number that is below the statistically minimum required for the results to be generalized (minimum of 1,067 students). However, the results are representative for the group of TPE schools and valid for programmatic purposes. Monitoring & Evaluation Project -­‐ FUNSEPA 29 The BOTH Programs Effect In general, the effect of BOTH programs—that is, when schools have received training for their teachers from AF as well as computers from TPE—is positive. Students in schools with BOTH programs have higher final grades. These results were obtained controlling for class size, school grade, proportion of computers per student, proportion of trained teachers, and poverty levels. The increase in the final grade for students in schools with BOTH programs is 0.49 points (see Appendix 9). Though the effect may seem small, it is important to understand that it likely represents a marginal benefit of implementing a second program once the first one is already in place. This is because the programs are implemented independently. For example, a school could have received computers from TPE but not have had its teachers trained by AF until 2 or 3 years later. Likewise, a school could have trained teachers but had no computer lab to reinforce teachings until years later. Due to this gap in the implementation of both programs, there is an initial impact on academic performance generated by the first program implemented. When the second program is implemented—and the BOTH programs effect is turned on—a positive effect on the final grade coming from the first FUNSEPA program is already present. Therefore, the effect of BOTH programs can be interpreted as a marginal effect or an additional impact to the one already produced by the first program. In order to exemplify this marginal effect, take the case of a school that received TPE in 2007 and AF in 2009. Students in this school increase their final grade by on average 1.3 points (individual effect of TPE). When teachers at this school are trained through AF in 2009, the effect on final grades increases marginally, as an initial benefit from TPE is already in place. Hence, students at this school increase their grades by 1.8 points (1.3 +0.49) when the school has received BOTH programs. The same interpretation applies when analyzing the effect of BOTH programs on subject grades. In the case of math, BOTH programs generated a marginal increase of 1.82 points relative to math grades in the comparison group. As for literature and science, the marginal grade increase is 1.5 and 0.65 points respectively compared to students in comparison schools (though this effect on science grades is not statistically significant). This model controlled for class size, proportion of computers per student, proportion of trained teachers per student, and poverty levels (see Appendix 10). Table 5. Results from BOTH Programs Effect on Final Grades BOTH Effect per Subject Final Math Literature Science Average Change in Grade + 0.49 points + 1.82 points + 1.50 points + 0.65 points w w Results not statistically significant Monitoring & Evaluation Project -­‐ FUNSEPA 30 As in other models, final grades decrease as students move to higher grades and increase when the proportion of computers to students and trained teachers to students increase. This model did not find any effect of gender on academic performance (see Appendices 9, 10, and 12). The BOTH Programs Effect on Demonstrative Schools of the Future Because the effect of BOTH programs is the combination of AF and TPE, the evaluation considered appropriate to explore the effect of a second FUNSEPA program on Demonstrative Schools of the Future (EDF). In this case, the results are even greater than those observed in EDF schools that only received TPE. When EDF schools receive a second FUNSEPA program, the academic performance of their students increases marginally by 4.55 points. In other words, the marginal effect of BOTH programs increases from 0.49 points to 4.55 points (over 4 additional points) if the school is EDF. This result is statistically significant and was obtained by controlling for class size, school grade, proportion of computers per student, proportion of trained teachers per student, and poverty levels (see Appendix 12). 13 These results are also interesting from a programmatic perspective since they suggest that if, in addition to providing the technology and pedagogical components, FUNSEPA programs include a solid technical training on the use of computers, the combined effect of BOTH programs on academic performance could be greater. Results from Other Models Although the main question of this evaluation focuses on the effect of the FUNSEPA programs on academic performance, the study also explored other potential effects that these programs may have on issues such as dropout rates and the probability of students being promoted to the next grade. However, it is important to mention that the data used for this evaluation includes variables directly related to academic performance, so other factors that might influence school dropout rates and promotion to upper grades have not necessarily been controlled for in the models reported below. Particularly in the case of dropout rates, the model does not include some important variables that may influence the decision to abandon school. These variables are related specifically to the student's household characteristics: family structure (if the family is comprised by both 13 Though statistically significant, the sample size does not allow these results to be generalized to all public school students in Guatemala. The model was applied to 470 students in EDF and non-­‐EDF schools that received BOTH programs—a number that is below the statistically minimum required for the results to be generalized (minimum of 1,067 students). However, the results are representative for the group of schools with BOTH programs and valid for programmatic purposes. Monitoring & Evaluation Project -­‐ FUNSEPA 31 parents or a single parent), family resources (education, occupation, and income of the parents), and patterns within the family (family members who have dropped out of school). Including such variables could make the dropout rates model more robust. Nonetheless, the model is still considered valid to estimate the relationship between FUNSEPA programs and dropout rates. In general, FUNSEPA programs show a positive effect on dropout rates and the probability of students passing to the following grade. When estimating the individual effect of the programs, the greatest effect observed is that of TPE. Moreover, there is an increase in the effect for schools that receive BOTH programs. Effect on Dropout Rates The average student dropout rate for schools in the sample is 6% per year. This overall average decreases when schools have participated in TPE, AF, or BOTH programs. In the case of TPE, the average annual dropout rate is reduced by about 1.43% relative to schools in the comparison group. This effect is statistically significant and was obtained controlling for the size of the school, final Table 6. Average Change in Dropout Rates grades, proportion of computers per student, and poverty levels (see Appendix 13). TPE -­‐ 1.43% -­‐ 1.14% Similarly, the average dropout rate decreases by 1.14% AF when schools have teachers trained by AF—controlling BOTH -­‐ 3.15% for the size of the school, final grades, proportion of trained teachers per student, and poverty levels (see Appendix 14). The effect of BOTH programs is even higher, showing an additional reduction of 3.15% in the average annual dropout rate when schools have received computers from TPE and also have trained teachers by AF. This effect is also statistically significant and was obtained controlling for the size of the school, final grades, the proportion of computers per student, proportion of trained teachers per student, and poverty levels (see Appendix 15). Finally, although there is a positive relationship between FUNSEPA programs and dropout rates, further research that would include other control variables is needed to assess the causal impact of the programs on dropout rates. Effect on Probability of Students Being Promoted to the Next Grade FUNSEPA programs also show a positive effect on Table 7. Average Change in the Probability the probability of students being promoted to the of Being Promoted to the Next Grade next grade. In the case of TPE, students in schools + 13.85% that received computers from FUNSEPA are nearly TPE 14% more likely to be promoted to the following AF + 9.26% grade than students in comparison schools. This BOTH + 9.30% model controlled for variables such as class size, Monitoring & Evaluation Project -­‐ FUNSEPA 32 school grade, proportion of computers per student, and poverty levels (see Appendix 16). Meanwhile, students in schools with teachers trained by AF are 9% more likely to advance to the next grade than students in comparison schools, controlling for the size of the class, grade, the proportion of trained teachers per student, and poverty levels (see Appendix 17). Students in schools that received computers from TPE and also have teachers trained by AF experience an additional 9% boost in their probability of being promoted to the following grade relative to students in comparison schools. This result is also statistically significant and was obtained by controlling for class size, school grade, proportion of computers per student, proportion of trained teachers per student, and poverty levels (see Appendix 18). Results from Standardized Tests The statistical analysis of the scores from the standardized tests implemented by DIGEDUCA did not yield conclusive results in areas evaluated (math and reading). It is important to mention that the model used for this analysis is different from the model used for the results discussed above. This model only tested if the average score of schools benefited by FUNSEPA and the average score of the comparison group were statistically different—see the Methodology section for further details. Based on the results of this model, there is no evidence that the scores in math or reading obtained by FUNSEPA schools are statistically higher, equal, or lower than the test scores of schools in the comparison group. This inconclusive result may be due to the small sample available for this model—which includes scores of only 22 schools (see Appendices 19 and 20). Monitoring & Evaluation Project -­‐ FUNSEPA 33 Conclusions The results of this evaluation show that FUNSEPA programs have in general a positive effect on the academic performance of students in schools benefited by these programs, as well as on dropout rates and the likelihood that a student is promoted to the following grade. Additionally, the analysis identified unintended effects of the FUNSEPA programs, such as schools access to new programs due to their participation in FUNSEPA programs. Principals in visited schools, for example, believe that schools accessed the benefits of other programs because they already had computer labs donated by FUNSEPA. Moreover, the evaluation revealed important insights that can help improve FUNSEPA’s activities so that the impact of its programs can be greater: §
The effect of BOTH programs could be higher if FUNSEPA implemented TPE and AF simultaneously. The statistical analysis showed that when schools have received BOTH programs, there is an additional effect to that generated by the first program implemented. Consequently, the effect of BOTH programs has the potential to be greater than the isolated impact of individual programs. §
Teacher training should go beyond the basic level. Most teachers trained by AF within the sampled schools received the basic training. During interviews for the qualitative analysis, teachers mentioned that while this initial training was useful, it was not enough and should have continued to more advanced trainings. The statistical effect of AF can also be increased if teacher training goes beyond the basic level. §
The individual effect of TPE and AF may also be greater if the number of computers donated were proportional to the average number of students in the classroom or school. The qualitative evaluation revealed that the time students can spend in the computer lab, as well as the time that trained teachers can use the lab to reinforce skills, is limited due to class size and overall school size. It is therefore possible that the effect of the programs would increase if more computers were donated to schools with a greater number of students per class. This would allow students to spend more time in the computer lab. §
The results observed in the case of the Demonstrative Schools of the Future show that the effect of the FUNSEPA programs on academic performance might be higher if the programs also provide solid technical training on the use of the technology. These results imply that there is potential to significantly increase the impact of the programs if they are complemented with technical preparation. Finally, the evaluation shows that the effect of FUNSEPA programs is generally positive and significant and that there are also opportunities to increase and enhance such effects. Specifically, both quantitative and qualitative results suggest that there is opportunity to deepen the implementation of the programs and focus resources on ensuring sustained quality. Monitoring & Evaluation Project -­‐ FUNSEPA 34 References Baker, E.L., Gearhart, M., & Herman, J.L. (1994). “Evaluating the Apple Classrooms of Tomorrow.” Los Angeles, CA: UCLA Center for the Study of Evaluation/Center for Technology Assessment. Barrera-­‐Osorio, F. & Linden, L. (2009). “The use and misuse of computers in education: Evidence from a randomized experiment in Colombia.” Policy Research Working Paper 4836. Washington, DC: World Bank. Barrera-­‐Osorio, F. (2006). “The Impact of Private Provision of Public Education: Empirical Evidence from Bogotá’s Concession Schools.” Policy Research Working Paper 4121. Washington, DC: World Bank. Center for Children and Technology (2001). IMB Reinventing Education: Research Summary and Perspective. New York, NY: Center for Children and Technology. Colorado State University. (2012). Sampling Procedures and Methods. Retrieved in March 2012 from: http://writing.colostate.edu/guides/research/com4b1a.cfm Mann, D., Shakeshaft, C., Becker, J., Kottkamp, R. (1999). “West Virginia Story: Achievement gains from a statewide comprehensive instructional technology program.” Santa Monica, CA: Milken Family Foundation. Ministerio de Educación de Guatemala (2010). Orientaciones Pedagógicas Curriculares – Nivel Medio Ciclo Básico. Guatemala, Guatemala. Retrieved in August 2010 from: http://www.mineduc.gob.gt/PORTAL/plantilla.asp?load=contenido/anuncios/orientacionesPed
agogicas2010/index.html&height=500px Ministerio de Educación de Guatemala (2010). Orientaciones Pedagógicas Curriculares – Nivel Medio Ciclo Diversificado. Guatemala, Guatemala. Retrieved in August 2010 from: http://www.mineduc.gob.gt/PORTAL/plantilla.asp?load=contenido/anuncios/orientacionesPed
agogicas2010/index.html&height=500px Ministerio de Educación de Guatemala (2010). Orientaciones Pedagógicas Curriculares – Nivel Primario. Guatemala, Guatemala. Retrieved in August 2010 from: http://www.mineduc.gob.gt/PORTAL/plantilla.asp?load=contenido/anuncios/orientacionesPed
agogicas2010/index.html&height=500px Mondragón, A. (2002). ¿Qué son los indicadores?. Revista Cultura Estadística y Geografía, N. 19. México, DF: INEGI. Retrieved in May 2012 from: http://www.inegi.gob.mx/prod_serv/contenidos/espanol/bvinegi/notas/notas19.pdf Spielvogel, R., et. al (2001). IMB’s Reinventing Education Grant Partnership Inititative – Individual Site Reports. New York, NY: Center for Children and Technology. Monitoring & Evaluation Project -­‐ FUNSEPA 35 Other bibliography: Linden, L., Banerjee, A., and Duflo, E. (2003) Computer-­‐Assisted Learning: Evidence from a Randomized Experiment. Poverty Action Lab Paper No. 5. Boston, MA: Poverty Action Lab. Ringstaff, C., and Kelley, L. (2002). The Learning Return On Our Educational Technology Investment. San Francisco, CA: WestEd RTEC. Schacter, John (1999). Education Technology on Student Achievement: What the Most Current Research Has to Say. Santa Monica, CA: Milken Exchange on Education Technology. Monitoring & Evaluation Project -­‐ FUNSEPA 36 Appendix Appendix 1. List of External Stakeholders Interviewed during Field Visit Organization People Interviewed and Position Academy for Educational Development (AED) Felix Alvarado, Education and Health Specialist Inter-­‐American Development Bank (IADB) Armando Godínez, Education Senior Specialist World Bank Carlos Pérez-­‐Brito, Human Development Official German International Cooperation in Guatemala Mario Raúl Moreno, Director of Programa de Apoyo a la Calidad Educativa (PACE) Empresarios por la Educación Verónica Spross de Rivera, Executive Director Ministry of Education Luisa Müller, General Director at DIGEDUCA Mario von Ahn, Assistant Director of Analysis, Statistics and Information Education Jaime Reyes, Educative Innovation at DIGECADE Research Triangle Institute (RTI) / Alianzas Guatemala Tere Ligorría, Director Cynthia del Aguila, Education Manager Manuel Antonio Román, Monitoring & Evaluation Manager Roots and Wings International Erik Swanson, Founder & Director of Operations USAID Guatemala Juan Luis Córdova, Education Programs Specialist Monitoring & Evaluation Project -­‐ FUNSEPA 37 Appendix 2. Distribution of Students and Schools per Group of Analysis Program TPE AF AF + TPE Number of Students 1,243 1,075 1,315 Number of Schools UDI Code Demonstrative Schools of the Future (EDF) 9 02-­‐06-­‐0142-­‐43 07-­‐08-­‐0248-­‐43 09-­‐11-­‐0345-­‐43 13-­‐05-­‐0311-­‐43 15-­‐04-­‐0143-­‐43 15-­‐04-­‐0145-­‐43 17-­‐09-­‐0267-­‐43 18-­‐01-­‐0077-­‐43 22-­‐07-­‐0331-­‐43 EDF EDF EDF -­‐ EDF EDF EDF EDF -­‐ 7 03-­‐06-­‐0879-­‐43 08-­‐05-­‐0282-­‐43 09-­‐04-­‐0012-­‐43 14-­‐16-­‐0652-­‐43 16-­‐13-­‐7558-­‐43 20-­‐01-­‐0069-­‐43 20-­‐09-­‐0381-­‐43 -­‐ -­‐ -­‐ -­‐ -­‐ -­‐ -­‐ 9 05-­‐01-­‐0066-­‐43 12-­‐06-­‐0305-­‐43 14-­‐04-­‐1328-­‐43 14-­‐06-­‐0260-­‐43 14-­‐12-­‐0445-­‐43 14-­‐15-­‐0605-­‐43 14-­‐15-­‐0612-­‐43 18-­‐02-­‐0549-­‐43 20-­‐01-­‐1194-­‐43 -­‐ EDF EDF EDF -­‐ EDF -­‐ -­‐ EDF 12 Comparison 2,456 5 01-­‐11-­‐1838-­‐43 04-­‐05-­‐0250-­‐43 08-­‐01-­‐0151-­‐43 08-­‐01-­‐1808-­‐43 16-­‐01-­‐0108-­‐43 Total 6,089 30 Monitoring & Evaluation Project -­‐ FUNSEPA 38 Appendix 3. List and Description of Variables Variable (variable name in statistical software) Type Description Final Grade (final_grade) Continuous Final grade reported by schools in the PRIM tables at the end of the school year. This grade comprises grades obtained for all evaluated subjects. Minimum possible: 0; maximum possible: 100; minimum to pass: 60. Math Grade (math_grade) Continuous Final math grade reported by schools in the PRIM tables at the end of the school year. Minimum possible: 0; maximum possible: 100; minimum to pass: 60. Literature Grade (math_grade) Continuous Final literature grade reported by schools in the PRIM tables at the end of the school year. Minimum possible: 0; maximum possible: 100; minimum to pass: 60. Continuous Final science grade reported by schools in the PRIM tables at the end of the school year. Minimum possible: 0; maximum possible: 100; minimum to pass: 60. TPE (TPE) Binary Indicates that the student is in a school that only received the TPE program; 1= received TPE, 0= did not received TPE. AF (AF) Binary Indicates that the student is in a school that only received the AF program; 1= received AF, 0= did not received AF. Binary Indicates that the student is in a school that received both programs, TPE and AF. 1= received BOTH programs, 0= did not received BOTH programs. Binary Indicates that the student is in a school that did not receive any of the FUNSEPA programs. 1= did not receive FUNSEPA programs, 0= otherwise. Class Size (class_size) Continuous Total number of students in the class of the student in a given year. School Size (school_size) Continuous Total number of students in elementary grades in the school of the student in a given year. Science Grade (science_grade) BOTH (BOTH) COMPARISON (CONTROL) Monitoring & Evaluation Project -­‐ FUNSEPA 39 Appendix 3. List and Description of Variables – Cont. Part 2 Variable (variable name in statistical software) Type Description Binary Indicates whether the student is a female or a male. 1=female; 0=otherwise. Binary Indicates whether the student belongs to an ethnic group (regardless of which specific ethnic group). Ethnicity is reported in the PRIM tables. 1=pertains to an ethnic group; 0=otherwise. Grade (grado_cat) Categorical School grade of the student in a given year. This variable ranges from first to sixth grade of elementary school. 1=first grade; 2=second grade; 3=third grade; 4=forth grade; 5=fifth grade; 6=sixth grade. Dropout (desercion) Continuous Dropout rate of a school in a given year (2005-­‐2010). Rates provided by the Ministry of Education of Guatemala. Promotion (promocion) Continuous Promotion rate of a school in a given year (2005-­‐2010). Rates provided by the Ministry of Education of Guatemala. Donated Computers (comp_donadas) Continuous Number of computers donated by FUNSEPA to a school through the TPE program. Number of Teachers (no_maestros) Continuous Number of teachers trained by FUNSEPA through the AF program at the school of the student in a given year. Teachers with basic level training (nivel_inic) Continuous Number of teachers that received basic training through AF at the school of the student in a given year. Teachers with intermediate level training (nivel_int) Continuous Number of teachers that received an intermediate level training through AF at the school of the student in a given year. Teachers with advanced level training (nivel_int) Continuous Number of teachers that received advanced level training through AF at the school of the student in a given year. Continuous Incidence of poverty by department by year of study. Variable scale is 0-­‐100, where 100=maximum poverty and 0=no poverty. Variable built for the 2005-­‐2010 series using the levels of poverty reported by the INE in 2000, 2006, and 2011. Gender (female) Ethnicity (etnia) Poverty (pobreza) Monitoring & Evaluation Project -­‐ FUNSEPA 40 Appendix 3. List and Description of Variables – Cont. Part 3 Variable (variable name in statistical software) Type Description Department (department) Categorical Department where the school is located Municipality (municipio) Categorical Municipality where the school is located Computers per student in the classroom (no_comp_class) Continuous Proportion of computers donated through TPE per student in the classroom. Computers per student in the school (no_comp_student) Continuous Proportion of computers donated through TPE per student in the school. Trained teachers per student in the classroom (no_maestro_class) Continuous Proportion of teachers trained through AF per student in the classroom in a given year. Trained teachers per student in the school (no_comp_student) Continuous Proportion of teachers trained through AF per student in the school in a given year. Continuous Score achieved in the standardized math test applied by the Ministry of Education of Guatemala to students in randomly selected public schools at the end of each school year (since 2007). Final scores range from 1 to 100, where 100 is the best performance possible. Continuous Score achieved in the reading standardized test applied by the Ministry of Education of Guatemala to students in randomly selected public schools at the end of each school year (since 2007). Final scores range from 1 to 100, where 100 is the best performance possible. Performance in standardized test on math (matemática) Performance in standardized test on reading (lectura) Monitoring & Evaluation Project -­‐ FUNSEPA 41 Appendix 4. Summary of Quantitative Results Average Change in Grade Program Final Math Literature Science TPE + 1.32 points + 3.31 points + 3.26 points +1.73 points AF + 1.19 points + 1.85 points +1.55 points + 1.28 points BOTH + 0.49 points + 1.82 points + 1.50 points + 0.65 points w w Results not statistically significant Monitoring & Evaluation Project -­‐ FUNSEPA 42 Appendix 5. TPE Fixed Effects Model – Final Grade Modelo / Model TPE
(4)
(5)
(6)
final_grade
0.066
-0.09
-0.109
-0.109
1.032
1.005
(0.235) (0.235) (0.236) (0.236) (0.260)** (0.265)**
-0.081
-0.082
-0.082
-0.116
-0.119
(0.013)** (0.013)** (0.013)** (0.013)** (0.014)**
0.001
0.001
0.007
0.006
(0.002) (0.002) (0.002)** (0.002)**
0.125
0.228
-0.311
(1.319) (1.307) (1.307)
-0.676
-0.651
(0.060)** (0.066)**
0.011
(0.013)
(1)
(2)
(3)
(7)
(8)
1.318
(0.374)**
-0.124
class_size
(0.015)**
0.005
school_size
(0.002)*
-0.299
female
(1.452)
-0.617
grado
(0.075)**
-0.015
desercion
(0.025)
-0.818
comp_per_class
(0.421)
-0.066
pobreza
(0.038)
74.647
77.183
76.778
76.716
77.581
78.108
78.72
80.483
Constant
(0.044)** (0.395)** (0.713)** (0.972)** (0.970)** (1.019)** (1.101)** (1.510)**
23800
23800
23785
23785
23785
22218
20227
20227
Observations
6089
6089
6086
6086
6086
5994
5500
5500
Number of group(id)
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
TPE
1.302
(0.373)**
-0.124
(0.015)**
0.006
(0.002)**
-0.352
(1.457)
-0.587
(0.072)**
-0.018
(0.025)
-0.808
(0.419)
(9)
(10)
(11)
1.317
1.359
1.326
(0.373)** (0.361)** (0.362)**
-0.124
-0.12
-0.118
(0.015)** (0.014)** (0.014)**
0.005
0.006
(0.002)* (0.002)**
-0.618
-0.642
-0.595
(0.075)** (0.065)** (0.064)**
-0.015
(0.025)
-0.816
-0.766
-0.776
(0.421) (0.410) (0.412)
-0.067
-0.054
-0.083
(0.038) (0.036) (0.034)*
80.337
79.545
81.893
(1.307)** (1.266)** (0.933)**
20227
21794
21809
5500
5592
5595
Note: This model was used to estimate the effect on the academic performance of students in schools that received TPE relative to students in comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 6. TPE Fixed Effects Model by Subject Modelo / Model TPE
Matemática / Math
math_grade
TPE
3.316
(0.419)**
class_size
-0.176
(0.016)**
grado
-1.924
(0.079)**
comp_per_class
-0.6
(0.447)
pobreza
-0.037
(0.042)
Constant
82.652
(1.121)**
Observations
21809
Number of group(id)
5595
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Modelo / Model TPE
Literatura / Literature
lit_grade
TPE
3.263
(0.454)**
class_size
-0.174
(0.016)**
grado
-1.396
(0.079)**
comp_per_class
-1.025
(0.571)
pobreza
-0.064
(0.043)
Constant
81.884
(1.139)**
Observations
21809
Number of group(id)
5595
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Modelo / Model TPE
Ciencias / Science
science_grade
TPE
1.731
(0.429)**
class_size
-0.16
(0.015)**
grado
-1.402
(0.074)**
comp_per_class
-0.802
(0.526)
pobreza
0.03
(0.039)
Constant
80.848
(1.046)**
Observations
21809
Number of group(id)
5595
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Note: The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 43 Appendix 7. AF Fixed Effects Model – Final Grade (1)
AF
0.388
(0.201)
(2)
(3)
(4)
Modelo / Model AF
(6)
(7)
final_grade
1.261
0.974
(0.245)** (0.270)**
-0.118
-0.118
(0.014)** (0.014)**
0.005
0.005
(0.002)* (0.002)**
-0.378
-0.372
(1.318) (1.317)
-0.682
-0.696
(0.069)** (0.070)**
0.013
0.013
(0.013) (0.013)
1.036
(0.547)
(5)
0.16
0.131
0.13
1.422
(0.201) (0.207) (0.207) (0.238)**
-0.08
-0.08
-0.08
-0.113
(0.013)** (0.013)** (0.013)** (0.013)**
0.001
0.001
0.006
(0.002) (0.002) (0.002)**
0.114
0.153
(1.320) (1.317)
-0.734
(0.062)**
(8)
(9)
0.917
(0.269)**
-0.118
(0.014)**
0.004
(0.002)*
-0.329
(1.313)
-0.722
(0.073)**
0.014
(0.013)
1.053
(0.547)
-0.058
(0.036)
2.199
(0.317)**
-0.111
(0.014)**
0.005
(0.002)*
-0.336
(1.324)
-0.695
(0.072)**
0.014
(0.013)
(10)
(11)
(12)
(13)
(14)
0.906
1.277
0.916
1.133
1.197
(0.256)** (0.271)** (0.269)** (0.268)** (0.267)**
class_size
-0.12
-0.119
-0.118
-0.114
-0.112
(0.014)** (0.014)** (0.014)** (0.013)** (0.013)**
school_size
0.005
0.002
0.004
0.005
(0.002)** (0.002) (0.002)* (0.002)**
female
-0.351
-0.324
(1.313) (1.466)
grado
-0.737
-0.627
-0.722
-0.757
-0.723
(0.073)** (0.074)** (0.073)** (0.065)** (0.064)**
desercion
0.015
-0.01
0.014
(0.013) (0.025) (0.013)
no_maestros_class
1.053
0.952
0.848
(0.547) (0.547) (0.547)
pobreza
-0.033
-0.054
-0.051
-0.059
-0.039
-0.064
(0.037) (0.036) (0.037) (0.036) (0.035) (0.034)
nivel_inic
-0.18
(0.039)**
nivel_int
0.243
(0.057)**
nivel_avan
-6.128
(0.303)**
Constant
74.599
77.085
76.806
76.749
78.154
78.646
78.573
80.107
79.11
79.811
80.761
79.948
79.3
81.513
(0.032)** (0.395)** (0.714)** (0.975)** (0.982)** (1.027)** (1.027)** (1.432)** (1.467)** (1.433)** (1.515)** (1.262)** (1.226)** (0.915)**
Observations
23800
23800
23785
23785
23785
22218
22218
22218
22218
22218
20227
22218
23785
23800
Number of group(id)
6089
6089
6086
6086
6086
5994
5994
5994
5994
5994
5500
5994
6086
6089
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Note: This model was used to estimate the effect on the academic performance of students in schools that received AF relative to students in comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 8. AF Fixed Effect Model by Subject Modelo / Model AF
Matemática / Math
math_grade
AF
1.851
(0.347)**
class_size
-0.168
(0.016)**
grado
-1.95
(0.079)**
no_maestros_class
3.086
(0.660)**
pobreza
-0.034
(0.041)
Constant
82.458
(1.103)**
Observations
23800
Number of group(id)
6089
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Modelo / Model AF
Literatura / Literature
lit_grade
AF
1.553
(0.345)**
class_size
-0.166
(0.015)**
grado
-1.411
(0.079)**
no_maestros_class
2.701
(0.705)**
pobreza
-0.068
(0.042)
Constant
81.744
(1.115)**
Observations
23800
Number of group(id)
6089
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Modelo / Model AF
Ciencias / Science
science_grade
AF
1.282
(0.334)**
class_size
-0.146
(0.015)**
grado
-1.544
(0.074)**
no_maestros_class
2.829
(0.653)**
pobreza
0.033
(0.038)
Constant
80.632
(1.025)**
Observations
23800
Number of group(id)
6089
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Note: The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 44 Appendix 9. BOTH Programs Fixed Effects Model – Final Grade (1)
BOTH
class_size
school_size
female
0.262
(0.276)
(2)
(3)
(4)
0.126
0.094
0.094
(0.274) (0.279) (0.279)
-0.081
-0.081
-0.081
(0.013)** (0.013)** (0.013)**
0.001
0.001
(0.002) (0.002)
0.119
(1.319)
grado
desercion
comp_per_class
no_maestros_class
pobreza
74.639
77.132
76.796
76.737
(0.022)** (0.388)** (0.716)** (0.976)**
23800
23800
23785
23785
Observations
6089
6089
6086
6086
Number of group(id)
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Constant
Modelo / Model AMBOS
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
final_grade
1.19
1.189
0.976
0.432
0.457
0.457
0.455
0.344
1.228
0.493
(0.298)** (0.301)** (0.317)** (0.412) (0.413) (0.413) (0.413) (0.407) (0.295)** (0.04)**
-0.116
-0.12
-0.119
-0.118
-0.118
-0.118
-0.118
-0.113
-0.115
-0.112
(0.013)** (0.014)** (0.014)** (0.014)** (0.014)** (0.014)** (0.014)** (0.014)** (0.013)** (0.014)**
0.006
0.005
0.005
0.005
0.004
0.004
0.004
0.006
(0.002)** (0.002)* (0.002)* (0.002)* (0.002) (0.002) (0.002) (0.002)**
0.184
-0.371
-0.371
-0.357
-0.312
-0.312
(1.308) (1.308) (1.458) (1.460) (1.456) (1.456)
-0.637
-0.605
-0.55
-0.579
-0.6
-0.6
-0.6
-0.635
-0.629
-0.593
(0.058)** (0.064)** (0.071)** (0.073)** (0.076)** (0.076)** (0.076)** (0.066)** (0.060)** (0.064)**
0.013
-0.012
-0.013
-0.011
-0.011
-0.011
(0.013) (0.026) (0.026) (0.026) (0.026) (0.026)
0.162
0.175
0.182
0.182
0.183
0.263
0.208
(0.322) (0.323) (0.324) (0.324) (0.324) (0.327)
(0.321)
1.316
1.219
1.219
1.219
1.587
1.384
(0.667)* (0.672) (0.672) (0.672) (0.667)*
(0.660)*
-0.056
-0.056
-0.056
-0.042
-0.082
-0.07
(0.038) (0.038) (0.038) (0.037) (0.033)* (0.034)*
77.936
78.582
78.888
78.736
80.213
80.213
80.061
79.196
81.815
81.469
(0.977)** (1.024)** (1.111)** (1.114)** (1.530)** (1.530)** (1.328)** (1.283)** (0.917)** (0.927)**
23785
22218
20227
20227
20227
20227
20227
21794
23800
21809
6086
5994
5500
5500
5500
5500
5500
5592
6089
5595
(5)
Note: This model was used to estimate the effect on the academic performance of students in schools that received BOTH programs relative to students in comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 10. BOTH Programs Fixed Effects Model by Subject Modelo / Model AMBOS
Matemática / Math
math_grade
BOTH
1.821
(0.521)**
class_size
-0.162
(0.016)**
grado
-1.926
(0.083)**
comp_per_class
1.785
(0.620)**
no_maestros_class
2.717
(0.785)**
pobreza
-0.004
(0.042)
Constant
81.656
(1.113)**
Observations
21809
Number of group (id)
5595
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Modelo / Model AMBOS
Literatura / Literature
lit_grade
BOTH
1.506
(0.518)**
class_size
-0.158
(0.016)**
grado
-1.368
(0.081)**
comp_per_class
1.426
(0.625)*
no_maestros_class
2.404
(0.839)**
pobreza
-0.036
(0.043)
Constant
80.885
(1.124)**
Observations
21809
Number of group (id)
5595
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Modelo / Model AMBOS
Ciencias / Science
science_grade
BOTH
0.659
(0.492)
class_size
-0.153
(0.015)**
grado
-1.467
(0.074)**
comp_per_class
0.301
(0.351)
no_maestros_class
3.488
(0.773)**
pobreza
0.058
(0.039)
Constant
80.261
(1.037)**
Observations
21809
Number of group (id)
5595
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
Note: The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 45 Appendix 11. TPE Fixed Effects Model in Demonstrative Schools of the Future – Final Grades Modelo / Model TPE
Escuelas Demostrativas del Futuro (EDF)
final_grade
2.865
(0.589)**
-0.162
class_size
(0.024)**
-0.972
grado
(0.152)**
-1.502
comp_per_class
(0.619)
-0.094
pobreza
(0.086)
81.327
Constant
(2.109)**
Observations
6944
Number of group(id)
1744
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
TPE
Note: The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 12. BOTH Fixed Effects Model in Demonstrative Schools of the Future – Final Grades Modelo / Model AMBOS
Escuelas Demostrativas del Futuro (EDF)
final_grade
4.559
(0.660)**
-0.129
class_size
(0.022)**
-0.784
grado
(0.140)**
0.295
comp_per_class
(0.380)
-1.415
no_maestros_class
(0.854)
0.109
pobreza
(0.093)
Constant
76.208
(2.147)**
6944
Observations
1744
Number of group(id)
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level
BOTH
Note: The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 46 Appendix 13. TPE Fixed Effects Model – Dropout Rates Modelo / Model TPE
desercion
-1.433
TPE
(0.164)**
0.023
school_size
(0.001)**
-0.006
final_grade
(0.003)
-0.383
comp_per_student
(2.622)
0.066
pobreza
(0.013)**
-2.532
Constant
(0.405)**
20227
Observations
5500
Number of group(id)
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level Note: This model was used to estimate the effect of TPE on dropout rates in schools benefited by this program and in relation to comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 14. AF Fixed Effects Model – Dropout Rates Modelo / Model AF
desercion
-1.139
AF
(0.101)**
0.005
school_size
(0.002)**
0.012
final_grade
(0.006)*
-1.089
no_maestros_student
(1.636)
0.286
pobreza
(0.022)**
-2.226
Constant
(0.682)*
22218
Observations
5994
Number of group(id)
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level Note: This model was used to estimate the effect of AF on dropout rates in schools benefited by this program and in relation to comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 47 Appendix 15. BOTH Programs Fixed Effects Model – Dropout Rates Modelo / Model AMBOS
desercion
-3.151
BOTH
(0.135)**
0.025
school_size
(0.001)**
-0.006
final_grade
(0.003)
-8.252
comp_per_student
(1.531)**
-13.311
no_maestros_student
(1.651)**
0.055
pobreza
(0.013)**
-3.249
Constant
(0.483)**
20227
Observations
5500
Number of group(id)
Robust standard errors in parentheses
** significant at 5% level; * significant at 1% level Note: This model was used to estimate the effect of BOTH programs (TPE+AF) on dropout rates in schools benefited by both programs and in relation to comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 16. TPE Probabilistic Model (xtlogit) – Probability of Students Being Promoted to the Next Grade Modelo / Model TPE
(Efectos Marginales de xtlogit)
aprobo
0.138
TPE
(0.049)**
-0.025
grado
(0.008)**
0.104
comp_per_class
(0.084)
-0.0001
pobreza
(0.003)
5465
Observations
1339
Number of group(id)
Standard errors in parentheses
** significant at 5% level; * significant at 1% level
Note: This model was used to estimate the effect of TPE on the probability of a student in a school benefited by this program of being promoted to the following academic grade, relative to students in comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 48 Appendix 17. AF Probabilistic Model (xtlogit) – Probability of Students Being Promoted to the Next Grade Modelo / Model AF
(Efectos Marginales de xtlogit)
aprobo
0.092
AF
(0.032)**
-0.027
grado
(0.007)**
0.392
no_maestros_class
(0.085)**
-0.0001
pobreza
(0.003)
5953
Observations
1457
Number of group(id)
Standard errors in parentheses
** significant at 5% level; * significant at 1% level Note: This model was used to estimate the effect of AF on the probability of a student in a school benefited by this program of being promoted to the following grade, relative to students in comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Appendix 18. BOTH Programs Probabilistic Model (xtlogit) – Probability of Students Being Promoted to the Next Grade Modelo / Model AMBOS
(Efectos Marginales de xtlogit)
aprobo
0.093
AMBOS
(0.047)*
-0.040
grado
(0.008)**
0.243
comp_per_class
(0.045)**
0.363
no_maestros_class
(0.097)**
0.004
pobreza
(0.003)
5465
Observations
1339
Number of group(id)
Standard errors in parentheses
** significant at 5% level; * significant at 1% level Note: This model was used to estimate the effect of BOTH programs (TPE+AF) on the probability of a student in a school benefited by both programs of being promoted to the following grade, relative to students in comparison schools. The model was tested using the Hausman Test to verify that it actually produces the best estimations. The table above shows robust standard errors. Monitoring & Evaluation Project -­‐ FUNSEPA 49 Appendix 19. Score Difference in Math Standardized Tests Note: This ttest assesses whether scores in math obtained by students in FUNSEPA schools (in the standardized tests implemented by the Ministry of Education) are statistically higher, equal, or lower than scores obtained by students in comparison schools. Appendix 20. Score Difference in Reading Standardized Tests Note: This ttest assesses whether scores in reading obtained by students in FUNSEPA schools (in the standardized tests implemented by the Ministry of Education) are statistically higher, equal, or lower than scores obtained by students in comparison schools. Monitoring & Evaluation Project -­‐ FUNSEPA 50 MANAUS Consulting Manaus provides consulting services to companies and organizations working with corporate responsibility and international development projects. MANAUS works with companies, nonprofit organizations, and multilateral organizations to help them understand the impact their programs and initiatives have in the communities where they are implemented. Research Team Principal Researchers: Carlued Leon, Global Research Manager Tamar Benzaken Koosed, President Technical Support: Santiago Bunce, Associate Alicia Cooperman, Summer Associate www.manausconsulting.com Monitoring & Evaluation Project -­‐ FUNSEPA 51