Potential and Pitfalls of Experimental Impact Evaluation Reflections

Potential and Pitfalls of Experimental Impact Evaluation: Reflections on the design and implementation of an experimental Payments for Environmental Services project in Latin America Jos Vaessen, Maastricht University and University of Antwerp David Todd, GEF Evaluation Office Gert van Hecken, University of Antwerp

Outline 1. Introduction 2. Strengths and weaknesses of the experimental design and implications for analysis 3. Conclusions

Background • Growing debate on the use of (quasi-)experimental methods in IE • GEF EO decided to commission an evaluation of experimental design in practice as part of its annual report on impact • RISEMP project: one of the few recently completed GEF projects with underlying experimental design 1. Introduction

RISEMP project • Pilot project on PES in agricultural landscapes • Regional project in three countries (20022008) • 8. 72 m US$ of which 4. 77 m US$ GEF funding • Implemented by WB; three national NGOs 1. Introduction

(Research) objectives RISEMP project • To demonstrate and measure: – effects of payment incentives for environmental services (PES) on adoption of integrated silvopastoral farming systems in degraded pasture lands – resulting improvements in ecosystems functioning, global environmental benefits, and local socio-economic gains 1. Introduction

Experimental design RISEMP project • (in principle) random assignment of farmers to different ‘treatment’ groups • Purpose was to test: – Effectiveness of payments: PES – CG – Effectiveness of technical assistance: PES + TA – PES only – Effectiveness of payment modality: PES 4 yrs – PES 2 yrs • Annual monitoring of land use changes 1. Introduction

Objectives of the study • To assess the strengths and weaknesses of the experimental design underlying the RISEMP project; its design and implementation; • To assess the potential of the experimental design as a basis for analyzing the effectiveness of project incentives on land use changes; • To suggest alternative and/or complementary methods for impact assessment; • To draw lessons on the viability and utility of (quasi-) experimental designs as a future evaluation component in similar projects as the RISEMP project, in which GEF could play a role 1. Introduction

Conceptual framework for analysis: potential threats to (internal) validity • Selection bias • Treatment diffusion • Unintended behavioral responses – Expected behavior or compliance behavior – Compensatory equalization – Compensatory rivalry • Other aspects 1. Introduction

Methodology and delimitation • Focus on Nicaraguan pilot site • Interviews with stakeholders Ø Best source of information: field staff • Interviews with farmers in pilot site • Document review • Challenge: how to get accurate information on design, implementation of experimental design and possible threats to validity? Ø semi-structured interviews, triangulation between sources 1. Introduction

Validity of difference in means: PES – CG • Selection bias - • Treatment diffusion • Unintended behavioral responses • Conclusion: comparison of difference in means is invalid indicator of effect of payments in case of Nicaraguan pilot site 2. Strengths and weaknesses of the experimental design

Difference in means: PES - CG Figure 2. Incremental ESI points per hectare (2003 -2007), three countries Source: own calculations based on RISEMP project data, January 2008 Note: PSA refers to pagos por servicios ambientales, or PES. 2. Strengths and weaknesses of the experimental design

Validity of difference in means: PES+TA - PES only • Selection bias ~ • Treatment diffusion - • Unintended behavioral responses • Conclusion: comparison of difference in means is invalid indicator of effect of technical assistance in case of Nicaraguan pilot site 2. Strengths and weaknesses of the experimental design

Validity of difference in means: PES 2 yrs – PES 4 yrs • Selection bias • Treatment diffusion • Unintended behavioral responses • Conclusion: comparison of difference in means is valid and useful indicator of effect of payment modality in case of Nicaraguan pilot site 2. Strengths and weaknesses of the experimental design

Difference in means: PES 2 yrs – PES 4 yrs Δ ESI pts Source: own calculations based on RISEMP project data, January 2008 2. Strengths and weaknesses of the experimental design

How to make the experimental design work in practice • Clarity and shared vision on what experiment is about • Training and quality control of experiment • Separate budget 3. Conclusions

Experimental method as an approach for assessing impact and complementary methods • Does not fully ‘cover’ the scope of impacts as defined by the DAC “The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. These effects can be economic, sociocultural, institutional, environmental, technological or of other types” (OECDDAC, 2002: 24). • Better suited to assess outcomes such as changes in land use and short-term ecological and socio-economic effects, less suitable for addressing: – long-term ecological and socio-economic effects and sustainability of effects – unintended effects – institutional and replicatory effects 3. Conclusions

…continued… • Theory-based analysis of processes of change can complement the experimental analysis by helping to: – uncover causal mechanisms of change and variables affecting processes of change that are of interest – explain how policies (payments) affect types of subjects (farmers) in different ways 3. Conclusions

On the use of (quasi-)experimental designs in RISEMP and similar projects • (Quasi-)experimental methods have strong comparative advantage in addressing the attribution problem in evaluation research Lessons for practice: Ø Criteria (on e. g. applicability and desirability) for selecting interventions for (quasi-) experimental analysis are needed (WHEN) Ø Appropriate budget and strategic plan for (quasi-) experimental design is needed (HOW) Ø Analysis should be complemented with other methods (HOW) 3. Conclusions

THANK YOU jos. vaessen@metajur. unimaas. nl jos. vaessen@ua. ac. be