In-depth Evaluation of R D Programs how could

In-depth Evaluation of R&D Programs – how could it be accountable? Symposium on International Comparison of the Budget Cycle in Research Development and Innovation Policies Madrid (Spain), 3 -4 July 2008 OECD/GOV/PGC/SBO Seung Jun Yoo, Ph. D. R&D Evaluation Center KISTEP, KOREA 1

Contents 1. Overview of Current Evaluation 2. Architecture of In-depth Evaluation 3. Procedure of In-depth Evaluation 4. Evaluation with Accountability 5. Challenges and Discussion 2

Overview of Current Evaluation 1 - All R&D Programs (191) evaluated every year! - specific evaluation (mainly using checklists) : 27 programs - self/meta evaluation : 164 programs - in-depth evaluation (4 horizontal programs (pilot run)) : climate change related R&D programs : university centers of excellence R&D programs : infrastructure (facility/equipments) R&D programs : genome research R&D programs 3

Overview of Current Evaluation 2 - Efficiency & Effectiveness of Evaluation? - evaluating 191 programs every year? - efficiency & effectiveness of evaluation itself is questionable, considering characteristics of R&D programs. . - too much loads of evaluation to evaluators, program managers and researcher, etc. - not enough time to prepare and perform evaluation for all R&D programs and to communicate with stakeholders (→ might yield poor accountability? ) 4

Architecture of In-depth Evaluation 1 - Main Players NSTC MOSF KISTEP (Evaluators) Evaluation Supporting Groups R&D programs of each ministry MEST Decision maker for R&D Evaluation and Budget Allocation MOE MIK MIFAFF MW …… Agency *NSTC (National Science & Technology Council) Agency 5

Architecture of In-depth Evaluation 2 - Evaluation & Budget allocation Evaluation group formed R&D Budget Input for budget allocation Survey/Analysis Programs/Projects implemented Programs To (re)plan and/or improve program In-depth Evaluation Feedback 6

Architecture of In-depth Evaluation 3 - Budget Process 5 yr plan 1 st Budget Review Ministry Budget Ceiling 2 nd Budget Review with Evaluation Results Budget Committee of National Assembly (Dec. ) Program Budget Ministry of Strategy & Finance (MOSF) NSTC 7

Procedure of In-depth Evaluation 1 - 7 -month schedule (suggested!) - Selected by selection committee based on special issue, etc. (month 0) - In-depth evaluation procedure for selected program(s) - month 1 : form evaluation group, gather program(s) data, study target R&D program(s), find major evaluation points - month 2 : develop logic model (with system dynamics, etc. ) - month 3/4 : perform in-depth analysis (relevance, efficiency, effectiveness, program design & delivery, etc) 8

Procedure of In-depth Evaluation 2 - month 5 : interview (researchers, program managers, etc. ) - month 6 : report interim evaluation result (MOSF, department(s)) - month 7 : report final evaluation result & recommendations 9

Evaluation with Accountability 1 - Responsibility 1 balance between quantitative and qualitative evaluation is important - systematic approach for qualitative evaluation is challenging - program goals vs projects implemented vs output enough time for evaluation is essential (7 -month schedule) - to achieve the goal of evaluation with accountability - give enough time for PM to cope with evaluation process † suitable only for limited # of programs to evaluate 10

Evaluation with Accountability 2 - Responsibility 2 qualitative assessment is needed to achieve the purpose of evaluation - measuring simple # of publications and patents? - publications : impact factor (1 -2 yrs), citation index (more than 3 yrs) - patents : commercial purpose → technology value evaluation - selected projects with excellent performance : consistent funding is required irregardless of its program evaluation! 11

Evaluation with Accountability 3 - Acceptability 1 understand well characteristics of program, sharing with stakeholders - performance indicators are useful tools to get stakeholders’ agreement - researchers, program managers, MOSF, etc. - to set up an evaluation strategy and points! - important especially for acceptability and for improving program delivery 12

Evaluation with Accountability 4 - Acceptability 2 (Understand & Change!) communication with stakeholders - interview with stakeholders is important to increase accountability : researchers, program managers, MOSF : evaluation strategy would better to share at the beginning - number of interviews are also important : lack of understanding evaluation is key inhibitor for accountability! : interviews at major steps such as evaluation strategy, survey weak/strong point of the program, report interim evaluation results, etc. 13

Challenges and Discussion 1 - Understand & Change & Improve! - Stakeholders should understand their program(s) : otherwise, rigid and too much defensive for keeping unchanged - Systematic way to understand diverse aspects of programs : goal, contents, projects, design & delivery, etc. - Sharing of program information to change : change and improve (to all stakeholders) 14

Challenges and Discussion 2 - Scientific, Socio-economic Interest - Technology impact evaluation for socio-economic understanding - Results of technology level evaluation are also useful 15

Challenges and Discussion 3 - Communication → Consultation - Communication among stakeholders (Ministry/Agency, Researchers, MOSF, KISTEP, etc. ) - For better evaluation practices, communication should be transformed to a way of consultation 16

Muchas gracias! 17