Evaluating Peace Operations Theoretical Perspectives and Operational Challenges

Evaluating Peace Operations: Theoretical Perspectives and Operational Challenges Benjamin de Carvalho, NUPI Presentation at the Monitoring and Evaluation of Peacekeeping and Peacebuilding workshop co-organized by the International Peace Institute (IPI) and the Norwegian Institute of International Affairs (NUPI) New York, 7 -8 May 2009 Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Overview of the Presentation 1. Why Evaluate Peacekeeping and Peacebuilding? 2. Perspectives on Evaluation 3. How, When and What to Evaluate in Peace Operations 4. Evaluating Peacebuilding in Complex and Unstable Environments: Challenges Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Working definitions: Evaluation and Monitoring Evaluation: ”Systematic assessment of policies, programs or institutions with respect to their conception and implementation as well as the impact and utilization of their results” (Source Paffenholz and Reichler) Monitoring: ”a continuing function that uses systematic collection of data on specified indicators” (Source OECD) Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

1. Why Evaluate and Monitor Peacekeeping and Peacebuilding? Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Peacekeeping and Peacebuilding Evaluation: Some Historical Perspectives • Distinguish between evaluations of peacekeeping and peacebuilding (evaluating PK less complex) • Peacebuilding from late 1990 s • Evolution of peacebuilding field: After 1990 s, focus on ‘lessons learned’ • From more ad-hoc in 1990 s to more professionalization today • Peacebuilding evaluation borrows heavily from development field Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Peacekeeping and Peacebuilding Evaluation: Some Historical Perspectives (cont’d) • Donors’ increased focus on accountability, as reluctant to fund activities which do not have clear impact • More focus on impact and effectiveness • New Public Management • UN Security Council focuses increasingly on indicators of progress and figures • UN system: increased focus on joint planning and indicators of progress (IMPP) Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

2. Perspectives on Evaluation Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

What do we evaluate? • • Norwegian Institute of International Affairs Evaluating Policies, Outcome(s) and Impact: Effective if outcomes are defined and met Evaluating Internal Process(es): Effective if internal processes work Symbolic Effect: Effective if perceived as effective System-wide Impact: Effective if system as a whole is progressing Norsk Utenrikspolitisk Institutt

What do we Evaluate? Effect s Impact Overall effect Outcome(s) Output(s) Results Immediate results Delivery of planned activities Activities Input Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt (Source Spurk IAM 2008 and OECD)

(i) Evaluating Outcome(s) and Impact • Most common form of evaluation • Sees activities and organizations as discrete activities and processes • Organization or activity effective if it has clear objectives and if these are met • Focuses largely on statistical analysis • Can include cost-benefit analysis Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(i) Challenges to Evaluating Outcome(s) • Outcomes and Impact often difficult to measure • Overly focused on statistical analysis? • Outcomes not necessarily the result of activity • Difficult to point at causality and clear link between input and outcome/impact Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(ii) Evaluating Internal Process(es) • Focus on processes within organizations • Organizations and processes as discrete entities • Effectiveness depends on whether internal processes work • Indicators of process: description of process rather than documented effects • Often easier than to evaluate outcome Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(ii) Evaluating Internal Process(es): Challenges • Assumes clear link between internal process and activities and outcomes • Can lead to too much emphasis on process at the expense of outcomes • Does not take into account coherence and coordination with other actors and processes Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(iii) Assessing Symbolic Effect(s) • Sees organization and process as interacting with environment • Effectiveness is about mastering signals and symbols, rather than substance • Internally: Evaluation as a way of boosting morale (is what we are doing important? ) • Externally: Effectiveness depends of whether organization or process is perceived as effective and legitimate • What to evaluate? Higher order impact? Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(iii) Assessing Symbolic Effect(s): Challenges • Little focus on actual change • Can lead to misguided resource allocation (if successes lead to continued activity at the expense of other activities) • People’s impressions can be misleading Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(iv) System-wide Evaluation • Evaluation of the response by the whole system (higher order impact) • Generally aggregate findings from a series of evaluations • Focus on system as a whole • Effective if most outcomes and impacts met? • Needs to be informed by clear theoretically founded ”story” or ”angle” in order to interpret findings Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

(iv) System-wide Evaluation: Challenges • Complex and must be undertaken as a joint evaluation • Actors have different timeframes, methodologies and methods, measures of effectiveness and benchmarks • Which actors to include? • How do we weight different or contradictory perspectives? • How do we take into account the selfinterest of other evaluators? Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

3. Methods: When, How and What do we Evaluate? Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

When? Before • Baseline study • Ex-ante evaluation During • Real-Time Evaluation (RTE): peer-review of fast evolving operation undertaken at an early phase • Formative evaluation • Mid-term evaluation After • Summative evaluation • Ex-post evaluation (Source OECD 2002) Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

What Criteria should We use? • DAC Criteria for Evaluating Development Assistance 1. Relevance 2. 3. 4. 5. Norwegian Institute of International Affairs Efficiency Effectiveness Impact Sustainability Norsk Utenrikspolitisk Institutt • 1. 2. 3. 4. 5. 6. 7. ALNAP/OECD adaptation for conflict prevention and peacebuilding Relevance/ Appropriatedness Connectedness Coherence Coverage Efficiency Effectiveness Impact

Considerations of Method • Overall initial understanding of context and starting point central, as getting it wrong will make it more dificult to evaluate • Evaluation must be planned at early stage • Risk of being ad hoc, and ’going through the motions’ • Relevance of evaluation: Early focus on benchmarks central. • Benchmarks must relate to ’angle’, perspective or theory and context Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

What to evaluate? • Process evaluation: An evaluation of the internal dynamics of implementing organizations • Programme evaluation: Evaluation of a set of time bound interventions, marshalled to attain specific objectives • Cluster evaluation: an evaluation of a set of related activities, projects and/or programmes • System-wide evaluation: Evaluation of the response by the whole system to a particular disaster event or complex emergency, or of the system-wide impact of activities (Source OECD) Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

4. Challenges to the Evaluation of Peace Operations Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Some technical challenges • • • Norwegian Institute of International Affairs Which methods should one apply? How does one operationalize? Difficult and expensive to gather data How to understand effectiveness? How to set goals? How does one ensure learning? Norsk Utenrikspolitisk Institutt

Evaluating in complex and volatile settings • • Norwegian Institute of International Affairs Baseline? Methods? Scope? Indicators, evaluation criteria? Purpose? Follow-up, Purpose, Lessons-Learned? Are wide-ranging quantitative methods borowed from the development field possible? Norsk Utenrikspolitisk Institutt

Setting a baseline • • Norwegian Institute of International Affairs Is information about status ex-ante available? Compatibility of information from other actors? Is there time to conduct a baseline study? Does the conflict environment allow for a wideranging study? Formal vs. informal processes? Services and economy often informal during conflicts Is data available when provision of services has broken down? Eg. infant mortality, SGBV Existing figures often outdated Methods Norsk Utenrikspolitisk Institutt

Methodological Issues • Quantitative vs. Qualitative Methods • Can we evaluate Impacts (eg. Democracy) or should we stick to Outcomes? • Are the underlying ”theories of change” robust enough? • Can we expect evaluators to understand the conflict and context? Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Do we evaluate too much? • Focus on evaluation can come at the expense of activities and processes: • UNICEF: 2 -5% of Country Program Funds • OCHA: goal of at least 1% of funding • UNHCR: Almost 1% of tota lbudget Do we evaluate too little? • Do we have too little information about the effectiveness of activities? : • UN DPKO: 0, 1 % of total budget • UN DPA: 0, 03% Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

What happens to evaluations? • To what extent are they useful? • To what extent are they used to improve process? • Are they used to legitimize decisions already taken? • Are they used as a bulwark against policymakers and donors Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt

Broader Considerations • Many tools for evaluating impact come from the development field. Are these adequate for activities which often are more medium-term? • Evaluation of peacebuilding must rely on theories of change, which at best are educated guesses about peaceful development, and often wrong. • Peacebuilding is a young field. Do we have enought comparative material? Do we set our expectations too high? Norwegian Institute of International Affairs Norsk Utenrikspolitisk Institutt