Impact Evaluation for Evidence Based Policy Making Arianna

Impact Evaluation for Evidence. Based Policy Making Arianna Legovini Lead, Africa Impact Evaluation Initiative AFTRL

Answer Three Questions p Why is evaluation valuable? p What makes a good impact evaluation? p How to implement evaluation? 2

IE Answers: How do we turn this teacher… 3

…into this teacher? 4

Why Evaluate? p Need evidence on what works n n p Improve program/policy overtime n n p Allocate limited budget Fiscal accountability Operational research Managing by results Information key to sustainability n n n Negotiating budgets Informing constituents and managing press Informing donors 5

What is different between traditional M&E and Impact Evaluation? p monitoring to track implementation efficiency (input-output) p impact evaluation to measure effectiveness (output-outcome) BEHAVIOR MONITOR EFFICIENCY INPUTS OUTCOMES EVALUATE EFFECTIVENESS $$$ 6

Question types and methods p Process Evaluation / Monitoring: ▫Is program being implemented efficiently? ▫Is program targeting the right population? ▫Are outcomes moving in the right direction? p Descriptive analysis Impact Evaluation: ▫What was the effect of the program on outcomes? ▫How would outcomes change under alternative program designs? ▫Does the program impact people differently (e. g. females, poor, minorities)? ▫Is the program cost-effective? Causal analysis 7

Which can be answered by traditional M&E and which by IE? p Are ITNs being delivered as planned? ØM&E p Does school-based delivery of malaria treatment increase school attendance? ØIE p What is the correlation between health ØM&E coverage and under fives receiving treatment within 24 hr of fever outbreak? p Does the house-to-house approach lead to ØIE an increase in under fives sleeping under ITNs relative to level in communities with other community-based approaches? 8

Types of Impact Evaluation p Efficacy: n n p Proof of Concept Pilot under ideal conditions Effectiveness: n n At scale Normal circumstances & capabilities Lower or higher impact? Higher or lower costs? 9

So, use impact evaluation to…. p p p Test innovations Scale up what works (e. g. de-worming) Cut/change what does not (e. g. HIV counseling) Measure effectiveness of programs (e. g. JTPA ) Find best tactics to e. g. change people’s behavior (e. g. come to the clinic) Manage expectations e. g. PROGRESA/OPORTUNIDADES (Mexico) n n Transition across presidential terms Expansion to 5 million households Change in benefits Battle with the press 10

Next question please p Why is evaluation valuable? p What makes a good impact evaluation? p How to implement evaluation? 11

Assessing impact p examples n n How much does an anti-malaria program lower under-five mortality? What is the beneficiary’s health status with program compared to without program? p Compare same individual with & without programs at the same point in time p Never observe same individual with and without program at same point in time 12

Solving the evaluation problem p Counterfactual: what would have happened without the program p Need to estimate counterfactual n p i. e. find a control or comparison group Counterfactual Criteria n n Treated & counterfactual groups have identical initial characteristics on average, Only reason for the difference in outcomes is due to the intervention 13

2 “Counterfeit” Counterfactuals p Before and after: n p Same individual before the treatment Non-Participants: n n Those who choose not to enroll in program Those who were not offered the program 14

Before and After Example p Food Aid Compare mortality before and after n Find increase in mortality n Did the program fail? n “Before” normal year, but “after” famine year n Cannot separate (identify) effect of food aid from effect of drought n p Epidemic 15

Before and After p Compare Y before and after intervention Y B A-B p Before-after counterfactual Estimated impact Before After C Control for time varying factors C A-C True counterfactual True impact A B B t-1 A-B is under-estimated Treatment t Time 16

Non-Participants…. p Compare non-participants to participants p Counterfactual: non-participant outcomes p Problem: why did they not participate? 17

Exercise: Why participants and non-participants might differ? p p p Mothers who came to the health unit for ORT and mothers who did not? Child had diarrhea Communities that applied for funds for IRT and communities that did not? Costal and mountain Children who received ACT and children who did not? Child had fever Access to clinic Epidemic and non-epidemic Access to clinic 18

Health program example Treatment offered p Who signs up? p n n Those who are sick Areas with epidemics p Have lower health status that those who do not sign up p Healthy people/communities are a poor estimate of counterfactual 19

Health insurance example p Health insurance offered p Who buys health insurance? p Who does not buy? p Compare health care utilization of those who got insurance to those who did not p Cannot separately identify impact of insurance and utilization on health 20

What's wrong? p Selection bias: People choose to participate for specific reasons p Many times reasons are directly related to the outcome of interest n p Health Insurance: health status and medical expenditures Cannot separately identify impact of the program from these other factors/reasons 21

Program placement example p Government offers family planning program to villages with high fertility p Compare fertility in villages offered program to fertility in villages not offered p Program targeted based on fertility, so n n p Treatments have high fertility Counterfactuals have low fertility Cannot separately identify program impact from geographic targeting criteria 22

Need to know… Why some get program and others do not p How some get into treatment and other in control group p p If reasons correlated with outcome n n p cannot identify/separate program impact from other explanations of differences in outcomes The process by which data is generated 23

Possible Solutions… p Guarantee comparability of treatment and control groups p ONLY remaining difference is intervention p In this workshop we will consider n n Experimental design/randomization Quasi-experiments Regression Discontinuity p Double differences p n Instrumental Variables 24

These solutions all involve… p Randomization n p Give all equal chance of being in control or treatment groups Guarantees that all factors/characteristics will be on average equal between groups Only difference is the intervention If not, need transparent & observable criteria for who is offered program 25

The Last Question p Why is evaluation valuable? p What makes a good impact evaluation? p How to implement evaluation? 26

Implementation Issues p Political economy p Policy context p Finding a good control n n n p Retrospective versus prospective designs Making the design compatible with operations Ethical Issues Relationship to “results” monitoring 27

Political Economy p What is the policy purpose? n n n In USA derail from national policy, defend budget In RSA answer electorate In Mexico allocate budget to poverty programs In IDA country pressure to demonstrate aid effectiveness and scale up In poor country hard constraints and ambitious targets 28

Political Economy p Cultural shift n From retrospective evaluation Ø n To prospective evaluation Ø Ø p Look back and judge Decide what need to learn Experiment with alternatives Measure and inform Adopt better alternatives overtime Change in incentives n n n Rewards for changing programs that do not work Rewards for generating knowledge Separating job performance from knowledge generation 29

The Policy Context p Address policy-relevant questions: n n p What policy questions need to be answered? What outcomes answer those questions? What indicators measures outcomes? How much of a change in the outcomes would determine success? Example: teacher performance-based pay n n Scale up pilot? Criteria: Need at least a 10% increase in test scores with no change in unit costs 30

Prospective designs p Use opportunities to generate good control groups p Most programs cannot deliver benefits to all those eligible n Budgetary limitations: Eligible who get it are potential treatments p Eligible who do not are potential controls p n Logistical limitations: Those who go first are potential treatments p Those who go later are potential controls p 31

Who gets the program? p Eligibility criteria n n Are benefits targeted? How are they targeted? Can we rank eligible's priority? Are measures good enough for fine rankings? Who goes first? p Roll out n Equal chance to go first, second, third? 32

Ethical Considerations p Do not delay benefits: Rollout based on budget/administrative constraints p Equity: equally deserving beneficiaries deserve an equal chance of going first p Transparent & accountable method n Give everyone eligible an equal chance n If rank based on some criteria, then criteria should be quantitative and public 33

Retrospective Designs p Hard to find good control groups n p Administrative data n p good enough to reflect program was implemented as described Need pre-intervention baseline survey n n p Must live with arbitrary or unobservable allocation rules On both controls and treatments With covariates to control for initial differences Without baseline difficult to use quasiexperimental methods 34

Manage for results Retrospective evaluation cannot be used to manage for results p Use resources wisely: do prospective evaluation design p n n n Better methods More tailored policy questions Precise estimates Timely feedback and program changes Improve results on the ground 35

Monitoring Systems p Projects/programs regularly collect data for management purposes p Typical content n n n p Lists of beneficiaries Distribution of benefits Expenditures Outcomes Ongoing process evaluation Information is needed for impact evaluation 36

Evaluation uses information to: Verify who is beneficiary p When started p What benefits were actually delivered p Necessary condition for program to have an impact: p benefits need to get to targeted beneficiaries 37

Improve use of monitoring data for IE p Program monitoring data usually only collected in areas where active n p Very cost-effective as little need for additional special surveys n p Collect baseline for control areas as well Add a couple of outcome indicators Most IE’s use only monitoring data 38

Overall Messages p Impact evaluation useful for n n n p Validating program design Adjusting program structure Communicating to finance ministry & civil society A good evaluation design requires estimating the counterfactual n n What would have happened to beneficiaries if had not received the program Need to know all reasons why beneficiaries got program & others did not 39

Design Messages p Address policy questions n Interesting is what government needs and will use Stakeholder buy-in p Easiest to use prospective designs p Good monitoring systems & administrative data can improve IE and lower costs p 40