Can Participatory Evaluation Approach Trigger an Increased Demand

Can Participatory Evaluation Approach Trigger an Increased Demand For Results of Relevant Stakeholders in 2014+ Programming Period? Krunoslav Karlovcec, Ministry for Economic Development and Technology, Slovenija

Ownership Stakeholders Empowerment participation learning negotiation flexibility Participatory evaluation approach Competencies Standards

Key starting points 1. Stakeholder identification process is an ex-ante process of determining individuals/organisation affected by the intervention. 2. Active involvement of stakeholders in the evaluation process is crucial as it increases the demand for results and volumes up the ownership and empowerment. 3. Competencies and standards in evaluation can contribute to effective conveyance of results to relevant stakeholders.

• Stakeholders are organisations/individuals who are affected by intervention during its lifetime or in subsequent years (Rist, R and Morra Imas, L. , 2009). • It is important to include as stakeholders those who would not typically be asked to participate in evaluation. • “Stakeholder analysis” is a method for stakeholder identification management. • Who are the stakeholders in European Cohesion policy? • Direct (legislature – the institutional framework of ECP) Ø Policy-makers, members of (EU) Parliament Ø EU Citizen are affected by the EU Cohesion policy

• Usually, evaluations are conducted by top-down approach, • That’s how the ECP evaluations are predominantely elaborated; • Participatory evaluation approach started as a response to traditional top-down approach; • Utilisation of evaluation findings and recommendations – is it more effective in top-down or in PE?

Participatory evaluation (1) • Evaluation should and can be used to empower the citizens to analyse and solve their own problems and address the needs; • According to the literature, participatory evaluation approach is more appropriate formative (mid-term, on-going, process) evaluation than summative (ex-post, final);

Participatory Evaluation (2) What are the pros? What are the cons? • Useful information for the program managers, • Enhances utilisation of evaluation findings, • Ownership of the evaluation report, • Mutual responsibility • Flexibility and loosened rigidity of the evaluation process, • Empowerment and active participation of (EU) citizen • • • Time- and resource-consuming, The conflicts among stakeholders easily emerge, One of the important issues is unequal powers of stakeholders Too much flexibility can distort the evaluation process, Too much time, money and other resources spent on stakeholder analysis which hinders evaluation process

How EC encourages the use of PEA in Evaluation • Reference in DG REGIO’s Draft Guidance on monitoring & evaluation, page 7, paragraph 2 • The principle of partnership embedded in the legislature, also in the 2014 -2020 (Article 5) • Multi-level governance – from local to national/European • Technical Assistance – resources, trainings, networking, • Still challenges: ownership and empowerment of the relevant stakeholders

Active participation and principle of partnership in the 2014+ Social and economic cohesion – two basic pillars of ECP; A new pillar was added – teritorrial cohesion, New instrument Integrated Teritorrial Investments, Place-based programming and implementation enhances the active participation of EU citizen in monitoring and delivery of results, • Partnership Agreement, Performance Framework – one of the new instruments to enhance better performance and evaluation utilisation… • … but how to position the evaluation on the place it belongs • •

The competencies and standards in the field of evaluation • Two examples of competencies/standards: Ø IDEAS (International Development Evaluation Association – Competencies for Development Evaluation Evaluators, Managers and Commissioners, adopted in 2012, Ø Canadian Evaluation Society’s Program Evaluation Standards, • Definition of competency: “…the demonstrated background, knowledge and skills necessary to practice (development) evaluation or to appraise its practice by others…” (IDEAS, 2012).

IDEAS • According to IDEAS, the DG’s Network representatives are Evaluation Managers; • My assumption is that the first step in increasing the importance of evaluation managers is setting up a framework of competencies as a self-assessment tool; • Utilisation of Evaluation findings and recommendation: Ø Communicating evaluation findings, Ø Promoting a Culture of Learning from Evaluation.

IDEAS (2) • Communicating evaluation findings • Raises awareness and use of evaluations through effecitve communication in each stage of evaluation, promoting transparency of evaluation, managing the dissemination; • Include the key stakeholder response in the final report, as relevant; • Speak truth to power in demonstrating strong evidence-based findings; • Promoting a Culture of learning from evaluation • Supports follow-up tracking of recommendation, • Seeks to build the evaluation capacity of others

Canadian Evaluation Society • Canadian Evaluation Society's Program Evaluation Standards, developed in 2012; • In the terms of utilisation of the evaluation findings - a section "Utility Standards" enhances the setting up of evaluation standards of evaluator credibility, attention to stakeholders, utilisation of relevant information and above all timely and appropriate communicating & reporting and concern for consequences and influence

Canadian Evaluation Society (2) • U 1 Evaluator Credibility Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context. • U 2 Attention to Stakeholders Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation. • U 3 Negotiated Purposes Evaluation purposes should be identified and continually negotiated based on the needs of stakeholders. • U 4 Explicit Values Evaluations should clarify and specify the individual and cultural values underpinning purposes, processes, and judgments.

• U 5 Relevant Information Evaluation information should serve the identified and emergent needs of stakeholders. • U 6 Meaningful Processes and Products Evaluations should construct activities, descriptions, and judgments in ways that encourage participants to rediscover, reinterpret, or revise their understandings and behaviors. • U 7 Timely and Appropriate Communicating and Reporting Evaluations should attend to the continuing information needs of their multiple audiences (key stakeholders). • U 8 Concern for Consequences and Influence Evaluations should promote responsible and adaptive use while guarding against unintended negative consequences and misuse.

SLOVENIA • Activities in the field of evaluation began in 2005 (2004 -2006 period) • Inter-ministerial steering group was established in 2008 (2007 -2013 period), • Negotiation process on the evaluation scope was extensive but evaluations were contracted out to external evaluators; • Evaluations used mainly descriptive and normative evaluation questions, designs & approaches, • No causal (cause & effect) question and design was ever used

SLOVENIA (2) • 2014 -2020 programming period: • Regional development programmes 2014 -2020; • Monitoring & Evaluation of the Regional development programmes 2014 -2020 – a Handbook with the promotion and the utilisation of PEA; • Theory of change will be linked to stakeholder identification/analysis; • A special annex of the Handbook will present the competencies/standards checklist and their utilisation. • Final version September 2013.

Conclusions • Stakeholder analysis – identify relevant stakeholders, not all of them. • Participative evaluation approach enhances empowerment and ownership of the relevant stakeholders – delivery of results is more effective. • Build up a set of competencies/standards as a selfassessment checklist-based tool, not an extensive document.

Literature: • International Development Evaluation Association (IDEAS): Competencies for Development Evaluation Evaluators, Managers and Commissioners; 2012; • Canadian Evaluation Society - Evaluation Standards; http: //www. evaluationcanada. ca/site. cgi? s=6&ss=10&_lang=en • 3. Rist, Ray C. and Morra Imas, L. G. : The Road to Results, The World Bank, 2009; • 4. Zukoski, A. and Luluquisen, M. : Participatory Evaluation; What is it? Why do it? What are the challenges? , Community-based Public Health Policy & Practice, 2002

Thank you for your attention! Krunoslav. karlovcec@gov. si