Скачать презентацию Innovation Programme Evaluation in the Context of Structural Скачать презентацию Innovation Programme Evaluation in the Context of Structural

bad9e7c9f3a247b044076d590a1ba9c1.ppt

  • Количество слайдов: 19

Innovation Programme Evaluation in the Context of Structural Funds • Presenter: Michael Dinges, • Innovation Programme Evaluation in the Context of Structural Funds • Presenter: Michael Dinges, • Joanneum Research • Authors: Effie Amanatidou, Ioanna Garefi, • ATLANTIS Consulting S. A. EUNIP International Workshop, 5 -6 May 2011, Florence

Contents • • Methodological Approach SF evaluation requirements The principle hypotheses Results EUNIP International Contents • • Methodological Approach SF evaluation requirements The principle hypotheses Results EUNIP International Workshop, 5 -6 May 2011, Florence

Methodological Approach SF literature review Requirements in terms of evaluation design, ways of execution, Methodological Approach SF literature review Requirements in terms of evaluation design, ways of execution, QA criteria Evaluation-related context in three countries (institutional settings, procedures, culture) Uptake of SF evaluation-related requirements in specific contexts; Relevance, feasibility and appropriateness of SF requirements; Impacts from implementation INNO-Appraisal results for SF-type evaluations cf. evaluation topics, data methods and quality issues INNO-Appraisal results cf. deviations across SF-type and non SF-type evaluations Char’s of quality evaluations Deviations between SF-type vs. non SFtype evaluations Contribution of SF provisions to high-quality evaluations. Impacts of SF regulations on structures, practices, culture and learning Literature review Interviews Survey results Analysis/conclusions EUNIP International Workshop, 5 -6 May 2011, Florence

SF evaluation requirements (1/2) • Structures: Authorities for Management, Certification, Auditing and a Follow-up SF evaluation requirements (1/2) • Structures: Authorities for Management, Certification, Auditing and a Follow-up Committee. • Evaluation design: request in the documents for negotiations for description of monitoring & evaluation system, info on appropriations and resources needed and ex-ante evaluation. • Evaluation execution: Community structural assistance is subject to ex-ante, mid-term and ex-post evaluation. • SF provisions specify the aim, type, and nature of evaluations and provide suggestions on how the different types should be carried out. Reference to existing evaluation guides is made. EUNIP International Workshop, 5 -6 May 2011, Florence

SF evaluation requirements (2/2) • Evaluators’ selection: explicit preference for independent bodies or experts SF evaluation requirements (2/2) • Evaluators’ selection: explicit preference for independent bodies or experts selected via a competitive tendering process. • Publicity & Dissemination: refers not only to activities supported under SF but also evaluation results; good practice to make public the entire evaluation report. • Quality assurance: – Evaluation report: Meeting needs; relevant scope; open process; defensible design; reliable data; sound analysis; credible results; impartial conclusions; clear report useful recommendations – Evaluation process: Coherent objectives; adequate To. R; tender selection; effective dialogue feedback; adequate info; good management; effective dissemination EUNIP International Workshop, 5 -6 May 2011, Florence

The principal hypotheses… 1. SF requirements may lead to specific characteristics in delivery & The principal hypotheses… 1. SF requirements may lead to specific characteristics in delivery & practice of evaluation 2. SF requirements may lead to higher quality evaluations 3. High quality SF evaluations may have greater impact in terms of usefulness and dissemination 4. SF regulations demand high standards on structures and processes that inevitably lead to some institutional learning and structure building EUNIP International Workshop, 5 -6 May 2011, Florence

1. SF requirements may lead to specific characteristics in delivery & practice of evaluation 1. SF requirements may lead to specific characteristics in delivery & practice of evaluation • Evaluation topics: less striking differences than expected across the two groups overall • More differences when looking at topics per evaluation type (ex-ante, interim, ex-post) within each group EUNIP International Workshop, 5 -6 May 2011, Florence

1. SF requirements may lead to specific characteristics in delivery & practice of evaluation 1. SF requirements may lead to specific characteristics in delivery & practice of evaluation ‘SF’ sample Evaluation topics Ex ante ‘non SF’ sample Acc Interim Ex post Ex ante Acc Interim Ex post External Consistency 33, 33% 16, 67% 27, 78% 5, 56% 4, 72% 11, 02% 36, 22% 22, 83% Internal Consistency 31, 43% 17, 14% 34, 29% 5, 71% 7, 03% 11, 72% 36, 72% 20, 31% Coherence/Complementarity 32, 35% 8, 82% 17, 65% 5, 88% 5, 04% 9, 24% 27, 73% 18, 49% 6, 25% 18, 75% 37, 50% 15, 63% 4, 80% 13, 60% 44, 00% 27, 20% 18, 42% 15, 79% 31, 58% 18, 42% 6, 35% 14, 29% 42, 86% 27, 78% Quality of Outputs 6, 67% 20, 00% 13, 33% 16, 67% 2, 52% 9, 24% 21, 85% Value for Money/Ro. I/Cost-Benefit 7, 69% 15, 38% 3, 85% 0, 00% 2, 59% 8, 62% 13, 79% 20, 69% 17, 24% 31, 03% 6, 90% 4, 80% 9, 60% 35, 20% 24, 00% 3, 23% 16, 13% 19, 35% 12, 90% 1, 68% 5, 04% 20, 17% 16, 81% Input Additionality 9, 38% 15, 63% 18, 75% 9, 38% 2, 59% 5, 17% 17, 24% 22, 41% Output Additionality 8, 82% 17, 65% 11, 76% 14, 71% 2, 61% 6, 96% 18, 26% 20, 00% Behavioural Additionality 0, 00% 13, 33% 6, 67% 3, 42% 8, 55% 21, 37% Policy/Strategy Development 25, 71% 14, 29% 25, 71% 11, 43% 7, 03% 11, 72% 36, 72% 18, 75% Gender issues 26, 47% 14, 71% 11, 76% 0, 00% 2, 59% 3, 45% 6, 90% 0, 86% Minority issues 19, 35% 0, 00% 0, 89% 1, 79% 0, 00% 0, 89% Goal Attainment/Effectiveness Outputs, Outcomes and Impacts Programme Implem. Efficiency Project Implementation Efficiency EUNIP International Workshop, 5 -6 May 2011, Florence

1. SF requirements may lead to specific characteristics in delivery & practice of evaluation 1. SF requirements may lead to specific characteristics in delivery & practice of evaluation • Data Analysis methods: not surprising differences across the two groups – not striking differences across the different evaluation types in either group • Data Collection methods: not surprising differences across the two groups – clearer differences across the evaluation types within each group ‘SF’ sample Data collection methods Existing Surveys/Databases Ex-ante ‘non SF’ sample Acc Interim Ex post Ex-ante Acc Interim Ex post 28, 95% 15, 79% 26, 32% 18, 42% 3, 17% 8, 73% 28, 57% 21, 43% Participant Surveys 5, 56% 8, 33% 30, 56% 2, 78% 1, 60% 10, 40% 32, 00% 25, 60% Non-participant Surveys 8, 00% 0, 00% 4, 00% 1, 68% 14, 29% 8, 40% Interviews 13, 89% 2, 78% 19, 44% 2, 78% 5, 34% 12, 98% 42, 75% 23, 66% Focus Groups/WS/Meetings 18, 75% 6, 25% 18, 75% 3, 13% 4, 76% 7, 94% 20, 63% 15, 87% Peer Reviews 4, 17% 0, 00% 8, 33% 0, 00% 1, 67% 10, 00% 6, 67% Technometrics / Bibliom. 0, 00% 0, 00% 0, 83% 1, 67% Document Search 29, 41% 2, 94% 32, 35% 14, 71% 5, 65% 7, 26% 27, 42% 17, 74% Monitoring Data 20, 59% 14, 71% 32, 35% 11, 76% 2, 61% 14, 78% 38, 26% 21, 74% EUNIP International Workshop, 5 -6 May 2011, Florence

1. SF requirements may lead to specific characteristics in delivery & practice of evaluation 1. SF requirements may lead to specific characteristics in delivery & practice of evaluation SF requirements do make a difference in guiding the topics to cover and analysis & collection methods to use across the evaluation types (ex-ante, interim, expost) within the SF group but not across SF and non SF samples. This suggests that the SF requirements repeat more or less what is advisable by international practice and thus followed by the non SF sample as well. EUNIP International Workshop, 5 -6 May 2011, Florence

2. SF requirements may lead to higher quality evaluations • Compliance of SF evaluations 2. SF requirements may lead to higher quality evaluations • Compliance of SF evaluations to quality standards is less than non SF type evaluations • Certain SF standards are clearly ‘overlooked’: production of useful recommendations; discussion of results with government and stakeholders SF requirements do not necessarily lead to high quality appraisals Quality characteristic Degree of compliance (medians) (1: Not at all, 5: Yes, definitely) SF type Non SF type Suitability of methods chosen 4, 00 5, 00 Well documented information sources 4, 00 Address of To. R 4, 00 5, 00 Analysis based on data 4, 00 Design appropriate to objectives 4, 00 Conclusions based on analysis 4, 00 5, 00 Satisfactory application qualitative methods 4, 00 5, 00 Coverage of broader context 3, 50 3, 00 Satisfactory application quantitative methods 4, 00 5, 00 Discussion within government circles 2, 00 4, 00 Discussion with participants / stakeholders 2, 00 4, 00 Usefulness of recommendations (^) 2, 4 3, 1 EUNIP International Workshop, 5 -6 May 2011, Florence

3. High quality SF evaluations may have greater impact in terms of usefulness & 3. High quality SF evaluations may have greater impact in terms of usefulness & dissemination Sample: SF (N=’ 4 s’ or ‘ 5 s’ in quality) Usefulness of Recommendations (Median of 1 -5 scale) Pgm design Pgm implem. Future pgm Contemp. Pgm Dissemination Policy form. Discuss gov't Discuss stkhold. Address To. R 3, 00 4, 00 1, 00 2, 00* 3, 00 Eval. Design 3, 00 4, 00 1, 50* 2, 00* 3, 00* Methods 3, 00 4, 00 1, 00 2, 00* 3, 00* Qualitative 3, 00 3, 50 1, 00* 1, 50* 2, 50 2, 00* Quantitative 3, 00 4, 00 - 2, 50 2, 00 Information 3, 00* 4, 00 1, 00* 2, 00* Analysis 3, 00 2, 50 3, 50 1, 00* 2, 00* Context 2, 00* 2, 50 2, 00* 1, 50* 2, 00* 2, 50* 3, 00 2, 00 4, 00 1, 00* 1, 50* 2, 00* 2, 50* Conclusions In general high-quality SF evaluations are not considered particularly useful except in the case where they provide recommendations on future programmes. But this appears also in the case of non-SF evaluations. EUNIP International Workshop, 5 -6 May 2011, Florence

3. High quality SF evaluations may have greater impact in terms of usefulness & 3. High quality SF evaluations may have greater impact in terms of usefulness & dissemination Sample: Non SF (N=’ 4 s’ or ‘ 5 s’ in quality) Usefulness of Recommendations (Median of 1 -5 scale) Pgm design Pgm implem. Future pgm Contemp. Pgm Policy form. Dissemination Discuss gov't Discuss stkhold. Address To. R 3, 00 4, 00 2, 00 3, 00* 4, 00 Eval. Design 4, 00 3, 00* 4, 00* Methods 3, 00 4, 00 3, 00* 4, 00* Qualitative 3, 00 4, 00 2, 50* 3, 00* 4, 00* Quantitative 3, 50 4, 00 2, 50 3, 00 4, 00 Information 3, 00* 4, 00 2, 00* 3, 00* 4, 00* Analysis 3, 00 4, 00 2, 00* 3, 00* 4, 00* Context 4, 00* 3, 00* 4, 00* 3, 00 4, 00 2, 00* 3, 00* 4, 00* Conclusions Increased discussions with government and wider stakeholders are more caused by high quality non SF evaluations. High quality SF requirements do not necessarily lead to high impact in terms of usefulness and dissemination EUNIP International Workshop, 5 -6 May 2011, Florence

4. SF regulations demand high standards on structures and processes that inevitable need some 4. SF regulations demand high standards on structures and processes that inevitable need some institutional learning and structure building • Greece: policy learning has remained limited and the use of evidence-based techniques is marginal and limited to satisfying the obligations towards the EU. When policy implementation also suffers from deficient staffing, and unsound management, it is only possible to typically abide by the rules and regulations rather than let them establish an evaluation culture and effective evaluation system. Thus, the limited impact caused by SF regulations. EUNIP International Workshop, 5 -6 May 2011, Florence

4. SF regulations demand high standards on structures and processes that inevitable need some 4. SF regulations demand high standards on structures and processes that inevitable need some institutional learning and structure building • Malta is in a transition phase currently taking steps to respond to SF requirements and establish an evaluation system but finds SF regulations inadequate in enhancing real policy learning. • Poland is also a new MS with similar problems with Greece in terms of innovation governance. However, it has responded more effectively to the needs referring to an effective evaluation system (Evaluation Unit/groups, increasing staff, significant role of evaluation in decision-making) but too much eagerness resulted in too many evaluations in a still immature period. Case studies partly support above hypothesis EUNIP International Workshop, 5 -6 May 2011, Florence

Case studies also provide possible explanations for survey findings (1/2) • Greece: despite long-lasting Case studies also provide possible explanations for survey findings (1/2) • Greece: despite long-lasting experience SF impacts from structures and required coordination have not managed to penetrate the system and overcome fragmentation – SF evaluations seen more as ‘internal’ exercises – more typical application of procedures and satisfying obligations than essential institutional and policy learning – but significant capacity building limited discussions with government and stakeholders • Greece: conclusions too general while not necessarily addressing additionality and impact assessment, thus limiting the quality of inputs to new programmes and schemes limited usefulness of recommendations EUNIP International Workshop, 5 -6 May 2011, Florence

Case studies also provide possible explanations for survey findings (2/2) • SF quality standards Case studies also provide possible explanations for survey findings (2/2) • SF quality standards suggested than imposed + typical application of relevant regulations such as in Greece suggested quality criteria may not be applied in practice • SF regulations too much focused on financial aspects and correctness of Structural Funding and implementation as noted in case of Malta too - lack of impact assessment and evaluation of qualitative achievements high quality SF evaluations may not lead to high impacts in terms of usefulness and dissemination to national stakeholders EUNIP International Workshop, 5 -6 May 2011, Florence

Conclusions • SF regulations do provide guidance on how to conduct specific types of Conclusions • SF regulations do provide guidance on how to conduct specific types of evaluations but this does not differ from international practice so the differences across the SF and non SF group are not marked. • Despite that fact that certain quality criteria are suggested under SF regulations these are not necessarily followed in practice. • The specific SF-type quality criteria when followed do not necessarily lead to high quality evaluations while the high quality evaluations do not necessarily have the highest impact in terms of usefulness of recommendations or discussions of results with government and stakeholders. • The country cases provide possible explanations for these results depending on the way the SF regulations on evaluation are taken up. • Nevertheless, there are strong positive impacts from SF regulations on capacity and structure building. On the other hand, institutional and policy learning and the establishment of sound evaluation systems still remains limited. EUNIP International Workshop, 5 -6 May 2011, Florence

Thank you! For more information: Effie Amanatidou: amana@otenet. gr Ioanna Garefi: garefi@atlantisresearch. gr EUNIP Thank you! For more information: Effie Amanatidou: [email protected] gr Ioanna Garefi: [email protected] gr EUNIP International Workshop, 5 -6 May 2011, Florence