Evaluation of the Paris Declaration Challenges for the

Evaluation of the Paris Declaration Challenges for the Evaluators AEA 2011 Niels Dabelstein Head, PD Evaluation Secretariat Bernard Wood Synthesis Team Leader

Evaluation components

The Key Evaluation Questions 1. “What are the important factors that have affected the relevance and implementation of the Paris Declaration and its potential effects on aid effectiveness and development results? ” (The Paris Declaration in context) 2. “To what extent and how has the implementation of the Paris Declaration led to an improvement in the efficiency of aid delivery, the management and use of aid and better partnerships? ” (Outcomes for aid effectiveness) 3. “Has the implementation of the Paris Declaration strengthened the contribution of aid to sustainable development results? How? ” (Development outcomes)

The Central Messages • The Paris Declaration has contributed to change of behaviour – but unevenly so. Partner countries have moved further and faster than donors. Some donors more than others and some very little. • The Paris Declaration has contributed to improve aid effectiveness – but much remains to be done. • The Paris Declaration has contributed to better development results – but not across the board. • The PD and AAA “campaign” remains relevant and has gained momentum – but needs nurturing to continue. 4

Challenges & lessons for complex, comparative evaluations 1. A fully participatory and independent approach is possible, but demanding, and not cheap. It hinges on good governance arrangements, synchronization, communications and support 2. Context is not just “background” – it is of the essence and should be systematically analysed and featured 3. Finding the basic programme theory can be especially critical for very complex evaluation objects – and it may be hidden in plain sight 4. Key common questions can be developed to reflect most cases, then supplemented by special priorities for individual cases 5. Make the common evaluation framework feasible for all —a chain is only as strong as its weakest link 6. Where new approaches are required, anticipate capacity differences and needs for technical support to teams

Challenges & lessons ll 7. Contribution analysis, properly applied, is indispensable for evaluating many complex change processes 8. The pressure for rapid assessment and the necessary time to see outcomes have to be balanced - evaluators must state the limits 9. Delays in practical arrangements for different participants’ mandates, structures, terms of reference, contracting, and approval and release of reports can disrupt or derail the whole joint effort. Clear expectations, peer pressures and deadlines ultimately need to be backed by a readiness to move ahead without stragglers 10. For a multi-site evaluation, the development of To. Rs and procurement of teams should be concurrent and the Synthesis process needs to be clarified from the start, in tandem with the common framework 11. Visibly ensuring independence is critical, particularly where potentially contentious findings are likely to emerge. This calls for strong procedures and standards, and clear governance

Key limitations • Evaluating the effects of a political Declaration traditional ‘linear’ approaches were not relevant • Limited time elapsed since 2005 • No comprehensive data from country studies on multilaterals and donors • Different methodology for donors (carried over from Phase 1) • Self-selection of participating countries / agencies – some gaps but still a reasonably representative “sample”

Strengths of Phase 2 • Evaluation Matrix developed for country studies, applying 11 outcomes of PD and AAA agreements. • Integrated evaluation quality assurance and peer review • Recognised the limits of aid in development and applied “contribution analysis. ” • A targeted process of guidance & support, recognising the primary importance of country studies. • Good governance of the Evaluation at national and international levels ( with 52 member International Reference Group) ensured joint process and independence and validated the framework and findings at key stages.

Full reports and supporting materials All documents from the Evaluation, including the full country evaluations and donor studies, can be found - in English, French and Spanish - on www. busanhlf 4. org and www. oecd. org/dac/evaluationnetwork/pde Tools, templates and guidance materials (in the Technical Annex to the Report) are also available on the above DAC site Thank you for your attention