Operationalizing IE: Case Study example: Textbooks and Teacher Training in Sierra Leone APEIE Workshop, May 10 -14 2010
Operational Key Steps 1. Policy 6. Analysis, Reports, Policy Advice 5. Survey 2. IE Design 3. Baseline Survey 4. Intervention Starts
IE Design – methodological issues l l Available data and required data Identify control group and treatment group – – l Sampling and sample sizes – – l Randomization Transparency Power calculations Budget and access constraints Identify outcome indicators
Data Instruments l l Utilize survey experts Who do you survey? – l Households, community, facilities, government… Pilot instruments – – Clear questions and coding Includes all information needed for analysis and covariates
Intervention Roll Out l Possible contamination of evaluation results! – – l Intervention benefits rolled out to control and treatment schools evaluation compromised Intervention or treatment schools only receive additional benefit evaluation compromised Monitor roll out – – – Train implementers – they need to know why! Hire supervisors Continue supervision throughout roll out
Background l In June 2007, the World Bank and a team from the Sierra Leone Ministry of Education and Sports (MOES) met to rigorously evaluate the impact of two pilot studies on student performances. l Two questions – – What is the effect of textbooks on student performance? What is the effect of teacher training on student performances?
How do we design the study to get rigorous results? l Random Selection – 5 districts in Sierra Leone l – Neediest schools in district (using EMIS data) l l l Kailahun, Kambia, Pujehun, Western Rural, Western Urban Student to teacher ratio, presence of latrine, school construction 360 schools total Random Assignment – – – Comparison schools: receive regular allotment of books Books schools: receive additional books Books and training schools: receive additional books + teacher training
Timeline Sept: WB and MOE agree on pilot Oct: Met with local councilors to select districts TARGET Nov: Baseline data collection and teacher training manual developed Dec: Roll out of textbooks and teacher training 2007 REALITY Nov: Mid-term data collection 2009 June: Baseline data collection NO teacher training up to this point 2008 June: Roll out of textbooks Dec: Mid-term data collection
What happened? l Met with several challenges on the ground that caused significant delays in the project: – – – Textbook distribution Teacher Training Data Collection
Challenge 1: Textbook Distribution l l l Target date: December 2007 Actual date: June 2008 Number of books: – – l MOES accidentally distributed original books, so had to use Sababu books had fewer books than project initially called for Communication – information not filtering down – – – Sababu not properly informed Service Providers unwilling/unable to change distribution Headteachers in comparison schools unaware of project
Challenge 1: Textbook Distribution continued l l l Textbook distribution: June 2008 Data collection: December 2008 Consequences: – – Not enough books distributed in some districts to make a statistical comparison Distribution happened very late and at a bad time (right before summer vacation) l – Allowing for only a small window for students to utilize the extra textbooks Some Comparison schools received books
Challenge 2: Teacher Training l l Target date: December 2007 Actual date: has not yet occurred Time and budget constraints The ‘+’ part of the ‘intervention +’ so took a back seat to the textbook distribution
Challenge 3: Mid-term Data Collection l l l Target date: November 2008 Actual date: December 2009 Delayed funding disbursement Non-existent schools (poor quality of initial data) Headteachers unaware of project – not willing for enumerators to collect data
Solutions: l l l Back up list of schools on hand for enumerators Better communication with all levels of organization involved Constant project sensitization Greater monitoring of distribution Inclusion of Process Surveys Allow for sufficient time for procurement and disbursement
Next Steps l Analyze mid-term data – – – l l Was there a significant effect due to textbooks? Has education in Sierra Leone improved in the past 2 years? Where in the textbook distribution are there major informational stop gaps? Move forward on teacher training Write up findings and circulate report
Conclusion l l Impact evaluations a useful tool for policy considerations But need to understand every process involved in the impact evaluation – l Textbook distribution, not just textbooks Study can be ineffective if you don’t have full buy-in and understanding at all levels of government, all service providers, and headteachers: – – – SENSITIZATION! COMMUNICATION! MONITORING!
THANK YOU
Скачать презентацию Operationalizing IE Case Study example Textbooks and Teacher
f02736e9eec839d24a7badc1f29c8103.ppt