World Bank Experience with Country-Level Monitoring and Evaluation

World Bank Experience with Country-Level Monitoring and Evaluation Ajay Chhibber Director Independent Evaluation Group World Bank

Four Main Topics 1. Growing trend towards results-based management 2. World Bank’s evaluation system ― role of independent evaluation 3. Examples of countries with good-practice monitoring and evaluation (M&E) systems 4. How the Bank’s Independent Evaluation Group helps countries strengthen their M&E systems 2

1. Results-Based Management ― The Power of Measuring Results ► If you do not measure results, you can not tell success from failure ► If you can not see success, you can not reward it ► If you can not reward success, you are probably rewarding failure ► If you can not see success, you can not learn from it ► If you can not recognize failure, you can not correct it ► If you can demonstrate results, you can win public support 3

Traditional Management ► Traditional management focuses on implementation • This involves tracking inputs ($$, resources, strategies), activities (what actually took place) and some outputs (the products or services produced) • This approach focuses on monitoring how well a policy, program or project is being implemented • Often used to assess compliance with plans and budgets 4

Results-Based Management Emphasizes Implementation and Results ► Results-based management involves the regular collection of information on how effective government (or any organization) is performing ► Results-based management demonstrates whether a policy, program or policy is achieving its stated goals, or outcomes ► and it investigates why these goals are, or are not, being achieved 5

Results-based Management Implementation 6 ► Long-term, widespread improvement in society Outcomes ► Intermediate effects of outputs on clients Outputs Results Goal (Impacts) ► Products and services produced Activities ► Task personnel undertake to transform inputs and outputs Inputs ► Financial, human, and material resources Binnendijk, 2000

Results-Based Management Is A Powerful Tool To Support The Transformation Of The Public Sector 1. Shifts from an input-activity-output focus to a focus on the outputs and outcomes of public sector policies, programs and projects 2. Emphasizes effective resource allocation for planning and budgeting 3. Helps countries assess if they are using their scarce resources most cost-effectively 7

A Powerful Tool (Continued) 4. Responds to elected officials and the public’s demands for accountability 5. Managers are judged by their programs’ performance, not be their control of inputs 6. Stresses knowledge and learning through continuous improvements 7. But recognize this is a political process with technical dimensions – not vice versa 8

2. The World Bank’s Evaluation System: (i) Self-Evaluation ► Self-evaluation is the responsibility of Management and Operational staff ► Project teams are responsible for periodic reporting on project progress and a final selfevaluation of the project ► Country teams report periodically on the progress of a Country Assistance Strategy and evaluate the Strategy after its conclusion 9

The World Bank’s Evaluation System: (ii) Independent Evaluation ► The responsibility of the Independent Evaluation Group (IEG) ► Reports directly to Board of the Bank ► Conducts quick desk reviews of all completed Bank projects ― compares these with the Self. Evaluations, highlights discrepancies ► More in-depth reviews of 25% of completed Bank projects ► Conducts sector, thematic, global reviews ► Evaluates Bank’s country assistance strategies, and reviews Bank’s own progress reviews 10

How Independent Evaluation Improves the Performance of Bank Operations? 1. Supports accountability by independently and publicly reporting on the results achieved by Bank Operations • exposing good and poor performance provides a necessary incentive for the Bank to achieve results 2. Helps the Bank to learn by distilling the Bank’s operational experience into knowledge of “what works” and “why” ► The potential tension between accountability and learning needs to be managed carefully 11

Three Examples of Influential Evaluations (1) IEG’s evaluation of Bank support for China’s national forest policy ► Helped legitimize debate on forest policy and the logging ban ► Strengthened support for improved M&E and for in-depth research and policy analysis ► Led to invitation to IEG to be an active member of China’s Taskforce on Forests and Grasslands 12

Three Examples of Influential Evaluations (2) IEG’s Evaluation of the Bank’s Approach to Global Programs ► Increased oversight and streamlining of the Bank’s global program portfolio ► Attracted strong interest from development partners for new funding and governance mechanisms of global programs ► Served as catalyst for the adoption of global evaluation standards 13

Three Examples of Influential Evaluations (3) World Bank / Government Evaluation of Delivery of Primary Education in Uganda ► Public expenditure tracking survey traced and publicized ‘leakage’ of earmarked funds ► Study strengthened parents’ ability to monitor the distribution of public spending at local level ► Regular publication of education funding transfers helped ensure that 90% of earmarked funds reached schools, up from 10% before 14

3. Two Countries with Good-Practice M&E Systems: Colombia, Chile Colombia’s M&E System: Architecture ► System is managed by the Department of National Planning (DNP), with strong support from the President’s Office ► On-line sub-system for monitoring and reporting government progress against Presidential Goals ► Ambitious agenda of impact evaluations (∑ 15 underway) ► DNP provides technical assistance for M&E to some ministries/agencies and municipalities 15

Two Countries with Good-Practice M&E Systems: Colombia, Chile Colombia’s M&E System: Strengths ► Very high utilization of system by President for oversight of ministers and ministries ― via performance targets ― and for accountability ► Rigorous impact evaluations conducted externally, and have high credibility ► Collaborative approach between DNP and sector ministries/ agencies, and with municipalities ► Performance budget reports; efforts to further strengthen performance budgeting ► Efforts to engage with civil society 16

Two Countries with Good-Practice M&E Systems: Colombia, Chile Colombia’s M&E System: Challenges ► High reliance on donor funding for M&E system ― low level of government own funding ► Insufficient link of M&E information to support national planning and budget decision-making ― this may now be changing ► Weak coordination of M&E roles / functions within DNP, with MOF and sector ministries ► Monitoring system’s data quality perceived as low 17

Two Countries with Good-Practice M&E Systems: Colombia, Chile’s M&E System: Architecture ► Designed, managed and used by MOF ► Developed incrementally, over past decade ► Performance indicators (∑ 1, 600) for all government programs ► Government program evaluations (∑ 160) these are desk reviews ► Rigorous impact evaluations (∑ 14) ► Comprehensive Spending Reviews -- desk reviews of all programs in a functional area 18 ―

Two Countries with Good-Practice M&E Systems: Colombia, Chile’s M&E System: Strengths (1) ► Evaluations conducted externally, in fully transparent process, and are highly credible ► All M&E findings reported publicly and sent to Congress ► M&E system closely linked to the information needs of MOF, especially for budget process ► Performance information used to set performance targets for ministries ― these are largely met 19

Two Countries with Good-Practice M&E Systems: Colombia, Chile’s M&E System: Strengths (2) ► High utilization of M&E findings by MOF in the budget process and to impose management improvements on ministries / agencies (see Table) Utilization of government evaluations by MOF Minor adjustment, modification, relocation of program Institutional relocation of program 25% 20 Change in program design or management process 39% 5% Substantial redesign of program or of internal management processes 21% Program termination 10% TOTAL 100%

Two Countries with Good-Practice M&E Systems: Colombia, Chile’s M&E System: Challenges ► Unevenness in quality of evaluations ― due to cost and time constraints ► Chile is probably not spending enough on evaluations ► Low utilization ― low ‘ownership’ ― of MOF’s evaluations by sector ministries 21

Building Country M&E Systems ― Lessons from International Experience 1. Key role of powerful champion of M&E 2. Opportunistic development of M&E systems Continuous review and modification Non-linear development of the systems 3. Centrally-driven, by capable ministry 4. Avoid competing systems ― such as different systems for planning and finance ministries 5. Incentives are key ― a process of cultural change to strengthen demand achieve high utilization of M&E information 22

International Lessons (Continued) 6. Build reliable ministry data systems 7. Role of structural arrangements to ensure M&E objectivity and quality 8. Long-haul effort, requiring patience 9. Limitations of relying on laws, decrees, regulations 10. M&E systems can be built and operated at relatively low cost u e. g. Colombia’s system costs about $2 m p. a. u e. g. Chile’s M&E system cost $0. 75 m p. a. 23

4. How Does IEG Support Countries’ M&E Efforts? (1) ► Part of IEG’s formal mandate ► Multi-pronged strategy of support ► Includes direct support to a few countries, done jointly with Bank’s Country Teams ● Recent focus countries: Brazil, Colombia, Chile, Egypt, South Africa, Uganda ► Support for country diagnoses ► Preparation of resource materials, on lessons learned: www. worldbank. org/ieg/ecd/ 24

How Does IEG Support Countries’ M&E Efforts? (2) ► Capacity building and training ● Since 2001, >1, 000 participants in International Program for Development Evaluation Training (IPDET) with Carleton University (4 week course) ● IPDET alumni actively engaged in international evaluation network and frequently assume senior positions in their country ● Regional training courses with core government agencies and universities ● Application and on-line course material: www. ipdet. org ► Support for South-South learning, e. g. conferences 25