Report about R D Evaluation Workshop in Japan Evaluation

Report about R&D Evaluation Workshop in Japan: Evaluation of Science and Technology Policy System in Japan: Country Case Study http: //www. mri. co. jp/PROJECT/20051011_ird 01. html Osamu NAKAMURA, Ph. D. Director for Technology Evaluation and Research Division, METI JAPAN

Evaluation of Science and Technology Policy System in Japan: Country Case Study Organizers: Japan Society for Science Policy and Research Management (JSSPRM) Ministry of Economy, Trade and Industry (METI) Ministry of Education, Culture, Sports, Science, and Technology (MEXT) National Institute of Advanced Industrial Science and Technology (AIST) National Institute of Science and Technology (NISTEP) … In cooperation with WREN! (Washington? /World? Research Evaluation Network) Date: June 2 -3, 2005 Place: Mita Public Hall, Tokyo, JAPAN

Policy Evaluation Exercises in Seoul / Tokyo In cooperation with WREN! Policy Evaluation Program Evaluation Project Evaluation General Discussion in Seoul (May 30 & 31, 2005) Country Case Study Exercise in Tokyo (June 2 & 3, 2005)

u Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 1: Science and Technology Policy System in Japan moderator: Prof. Kondoh (Yokohama Natl. Univ. ) commentator: Dr. Holland (White House, OSTP) Prof. Georghiou (Manchester Univ. ) u Session 2: Evaluation of Science and Technology Policy and Reflection of its Result in Japan moderator: Prof. Niwa (Natl. Graduate Ins. for Policy Studies) commentator: Dr. Reeve (EC) Dr. Shipp (NIST) u Session 3: Evaluation of Science and Technology Programs and Reflection of its Result in Japan moderator: Prof. Miyazaki (Tokyo Inst. of Technology) commentator: Dr. Oros (USDA) Mr. Teather (NRC of Canada) u Session 4: Evaluation of National Research Institutes moderator: Prof. Hayashi (Natl. Inst. for Academic Degrees and Univ. Evaluation) commentator: Dr. Jordan (SNL) Mr. Valdez (DOE)

Kick-Off: Prof. Vonotas (GWU) Evaluation of Science and Technology Policy System in Japan: Country Case Study

Progress of R&D Evaluation in Japan Policy Program Project Institute 2002　 Government Policy Evaluations Act Impact As. 2005 National Guideline on the Evaluation for Governmental Research and Development (III) Ex-ante Monitoring/ Ex-post 2001 National Guideline on the Evaluation for Governmental Research and Development (II) 1997　 National Guideline on the Evaluation for Governmental 2001 Research and Law on the General Rules Development (I) of Incorporated Administrative Agencies Ryo Hirasawa, 30/08/2005, rh@rhirasawa. info

Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 1: Science and Technology Policy System in Japanese Science and Technology Policy System and Evaluation Dr. Tomohiro IJICHI Associate Professor, Institute of Innovation Research, Hitotsubashi University Affiliated Fellow, National Institute of Science and Technology Policy, MEXT

Ministry of Education, Culture, Sports, Science, and Technology Ministry of Economy, Trade and Industry Ministry of Health, Labour, and Welfare Ministry of Agriculture, Forestry, and Fisheries

Evaluation of Science and Technology Policy System in Japan: Country Case Study Public R&D Budget in Japan Development 23% Investigation 6% 20 Billions US$ Applied 27% Basic 44%

Evaluation of Science and Technology Policy System in Japan: Country Case Study Comments from Commentators: Evaluation results should be reflected to the budget on the uniform basis across different types of researches. In active research fields, many researchers at different research units should compete to stimulate each other. It would produce the new discipline with high yield. So, the balance between the administrative efficiency and competition of researchers is very important. Japanese evaluation system has no definite framework of ex-ante evaluation, so few projects/programs have clear targets to be accomplished. An incentive system to R&D projects based on the results of evaluation should be carefully set in order to promote reasonable competition.

Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 2: Evaluation of Science and Technology Policy and Reflection of its Result in Japan Current Situation and Challenges for Policy Level Evaluation in Japan Dr. Jun SUZUKI Professor, Technological strategy, S&T policy, Graduate School of Engineering Management, Shibaura Institute of Technology

1. Backgrounds and track records for evaluation of science and technology policy Law on the Evaluation of Science and Technology Policy Conducted by the Government 　　　　　（Government Policy Evaluation Act, April 2004） METI Basic Plan for Policy Evaluations achievement 　・ex-ante evaluation for 113 policy measures, an integration of 840 projects since FY 2002 　・of which 75 policy measures (42 are R&D) subject to ex-post evaluation Case：solar power generation MEXT Basic Plan for Policy Evaluations achievement ・ 3 evaluation procedures 　ex-ante, ex-post, and synthesized evaluation ・ ex-ante and ex-post evaluation 　 ex-ante: 78 projects ex-post: 8 projects ・achievement evaluation: 42 policy measures Case：HIMAC He stressed the importance to employ the Balanced Score card and Behavioral Additionality to assess the effects of project performance.

Comments from Evaluation of Science and Technology Policy System in Japan: Country Case Study Commentators: It is very important to evaluate not only intended effects in the mid-term and long-term but also secondary effects. Besides objectives and goals, management of the performance is also very important element of program management. Project budget must be managed in a consistent manner throughout the project period. It is unclear in Japan who is responsible for the evaluation results. The evaluation system in EU sets importance in the balance of: accountability transparency justification learning trust in evaluation

Evaluation of Science and Technology Policy System in Japan: Country Case Study

Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 4: Evaluation of National Research Institutes Revised Evaluation System to Reflect the Future A model＠National Institute of Advanced Industrial Science and Technology (AIST) ・Independent administrative institution under METI ・To perform innovation-oriented basic researches Dr. Osamu NAKAMURA Senior Researcher, Evaluation Department, AIST

Sustainable Development by Full Research Outcomes AIST Society Public resources AIST places its highest priority on the pursuit of complete research, "Full Research", engaging Type-I Basic Research to the development of “products” by conducting intensive Type-II Basic Research. People’s Interest (applause) dream Type-I basic research Universities nightmare reality Type-II basic research Industrial development Industries AIST Full Research

Evaluation from the view point of OUTCOMES Procedure of Evaluation external reviewers: internal reviewers: Specialists or Intellectuals Strategy of AIST Principal Reviewers to make sure the research ＊Roadmap Evaluation strategy　＊Output Evaluation performances to know the level of ＊Roadmap for outcomes ＊Output indexes achievements ＊Management and ＊Managementresearchers to perform Full Research 　 to encourage Evaluation Research Units 　　Evaluation: Rating with comments n tio ca me llo on m A Co ce izati r ou gan s Re eor R nts Evaluation Department Nation President of AIST Evaluation Committee

Notes: Evaluation is for Encouraging with Love. u Evaluation is for Communication. u Evaluation is for Strategy. u Evaluation is for Creation. u Evaluation is to be fed back. u Evaluation reflects the Future. u Evaluation is like a Compass for Voyage. u “Revised Evaluation System to Reflect the Future” (O. Nakamura, AIST)

Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 4: Evaluation of National Research Institutes Evaluation System of RIKEN ・Independent administrative institution under MEXT ・To perform comprehensive researches for natural science Dr. Masayuki UCHIDA Evaluation Section, Policy Planning Division RIKEN

2 -1 Advisory Council System of RIKEN Brain Science Institute Director，BSI RIKEN Bio. Resource Center Director，BRC RIKEN Genomic Sciences Center Director，GSC AC RIKEN Plant Science Center Director，PSC AC RIKEN SNP Research Center Director，SNP AC The Board of Executive Directors RIKEN Frontier Research System Director，FRS AC President AC AC Recommendation AC AC RIKEN Advisory Council RIKEN Discovery Research Institute & Harima Institute Directors, DRI & HI RIKEN Research Center for Allergy and Immunology Director，RCAI AC RIKEN Center for Developmental Biology Director，CDB Review Report Advice Response

Comments Evaluation of Science and Technology Policy System in Japan: Country Case Study from Commentators: It is very good to perform evaluation of a research institute based on its strategic roadmap, outputs, and research management. If you want to improve the research environment, you should set a target and monitor indicators across years. It must be emphasized to motivate researchers by setting appropriate criteria for evaluation. Important elements to create a good business model of a research institute: 1. Core competency 2. Initiative and products (deliverables) from it 3. Leadership based on a rational criteria 4. Risk management (monetary risks, market risks, scientific risks, non-technical risks, etc. )

Evaluation of Science and Technology Policy System in Japan: Country Case Study Day 2 Session 1: Applicability of logic model to policy evaluation Dr. Gretchen Jordan (SNL) Mr. George Teather (NRC of Canada) Session 2: Application of US PART for Japanese public RTD evaluation Mr. Bill Valdez (DOE) Dr. Mike Holland (White House, OSTP) Session 3: Application of EU FW 6 Evaluation system for Japanese public RTD Prof. Luke Georghiou (UM) Dr. Neville Reeve (EC) Session 4: Current program evaluation methods Dr. Stephanie Shipp (NIST) Session 5: Training program for policy evaluation/R&D evaluation Prof. Nicholas Vonortas (GEU) Dr. Cheryl Oros (USDA) Session 6: Newest evaluation tools Dr. Tom Fiddaman (Ventana Systems)

Summary: Evaluation of Science and Technology Policy System in Japan: Country Case Study We conducted extensive discussions on methodologies/ focuses on policy/program/project evaluations on Japanese science and technology policy system as a country case study with the guidance of WREN– for the first time ever. Next Steps for Japan (METI): We are trying to introduce logic models into planning and evaluation of METI projects and programs. We consider that the some criteria in PART (US) or FP (EU), such as those for management, must be incorporated into Japanese S&T policy evaluation system. We should get into the network of the world evaluation community in order to apply global top-level evaluation methodologies to Japanese S&T policy evaluation system.

Evaluation of Science and Technology Policy System in Japan: Country Case Study

Thank you ! Evaluation of Science and Technology Policy System in Japan: Country Case Study

Evaluation of Science and Technology Policy System in Japan: Country Case Study Appendix

Evaluation of Science and Technology Policy System in Japan: Country Case Study R&D Budget of Japanese Ministries (including expenses for labors and facilities, 2005) MIC MLIT 2% 2% MAFF 3% MHLW 4% others 3% JDA 4% 36 Billions US$ METI 17% MEXT 65%

Chiba (HIMAC) (3) Heavy Iron Medical Accelerator in Chiba (HIMAC) ◆Background to development ・　Construction of Heavy Iron Medical Accelerator in Chiba (HIMAC) began in 1984 as the world’s first heavy particle accelerator for medical treatment, based on the Comprehensive 10 -Year Strategy for Cancer Control. ・　Clinical experiments began in 1994. ・　The device has been used in 2, 184 people up to February 2005. ・　HIMAC was provided to researchers both in Japan and overseas as equipment for utilization in biological experiments that would provide the basis for medical treatment, as well as in physical and engineering experiments. Over 500 outside researchers use the device each year. ・　Highly-advanced medical treatment began in November 2003, and 341 patients have undergone treatment. ◆Special features：・　Causes no pain, placing less burden on patients ・　Induces little or no side effects ・　Can treat lung and liver cancer within a short period (less than 1 week) ・　Can treat refractory cancers (e. g. , that of the bone and soft tissues) that respond poorly to other treatment methods