ARIMNet 2 Governing Board Meeting 27 -28 October

ARIMNet 2 Governing Board Meeting (27 -28 October 2016, Rome, Italy) Workpackage 3 Anabel de la Peña (anaisabel. delapena@inia. es) INIA (SPAIN) 1

Workpackage 3 Monitoring, follow-up and impact assessment of calls and funded research projects 2

Objectives § Analyse and evaluate the ARIMNet 1 and ARIMNet 2 (both calls) § Optimize the call procedures § Develop common guidelines for the monitoring to assess the impact of the calls § Lessons learnt (pros and cons) for conclusions looking for best options for the future 3

Tas Task 3. 1 s Evaluation of calls, calls procedures and impact assessment Ø INIA (M 1– 48) Task 3. 2 Monitoring the progress within the research projects continuously Ø FCT (M 1– 48) Task 3. 3 Assessment of research project results at mid-term and at the end (IRESA) Ø (IRESA (M 1– 48) 4

Survey Task 3. 1 4 questionnaires addressed to: ØCall Office (CO) ØCall Board (CB) (funding bodies) ØApplicants (coordinators) ØEvaluators (EC) Questionnaires § Introduction: Brief explanation of ARIMNet 1 call § Sections according with the phases of the call: questions with multiple answers D 5

ARIMNet 1 in brief Ø Ø Ø Calendar of the call (May 2011 - February 2012) Consortiums of 4 – 6 partners: no limits of partners Mo. U signed by 14 organizations (11 countries) 3 core topics and 18 subtopics (broad scope) 86 “letter of intent” (non-compulsory), 79 proposals eligible Eligibility: ARIMNet criteria and national eligibility Evaluators: External referees + Evaluation Committee (EC) 2 EC members/proposal + 4 referees/proposal 2 Call Board meetings A submission website at Call Office website 10 projects selected 6

ARIMNet call 2011 THE CALL TIMING • • • ) Pre- announcement (6 th May 2011) Launching and guidelines on the web (20 th June 2011) Letter of intent (non compulsory) (12 th June 2011) Full proposals (25 th September 2011) Eligibility (26 – 28 th September 2011) 1 st Evaluation Committee (5 th October 2011) Scientific Peer Review (5 th – 15 th October 2011) 2 nd Evaluation Committee (19 – 20 th October 2011) 1 st Call Board and funding decision (21 st December 2011) 2 nd Call Board (14 th February 2012) Contract negotiations (first quarter of 2012) 8

ARIMNet Call 2011 Call Board (funding bodies) Country Algeria INRAA Cyprus ARI Egypt ARC France INRA, CIRAD Greece Organization DEMETER Israel MOARD Italy MIPAAF Morocco IAV Spain INIA Tunisia IRESA Turkey NAGREF 8

Questionnaire s Call Office 23 questions (7 sections) 1. The preparation process: Focused on the process of preparing the ARIMNet Call 2011. 2. Application phase: Focused on impressions of the operational processes put in place by call 2011 3. Eligibility phase: Focused on eligibility process of proposals under the call 2011 4. Scientific evaluation phase: Focused on the scientific evaluation process 5. Selection and funding decisions: Focused on the outcomes of the funding decisions and destination of funds 6. Negotiation phase: Focused on the process of negotiation for national contracts 7. General issues: Focused on the general process of the call 9

Questionnair es Call Board 27 questions (5 sections) 1. General Vision: Focused on the general process of ARIMNET Call 2011 2. Preparatory phase: Focused on the process of preparing ARIMNET Call 2011 3. Application phase: Focused on the process of implementation under the ARIMNET Call 2011 for proposals 4. Selection phase: Focused on the process of project selection under the ARIMNET Call 2011 5. General strategic issues: Focused on the general strategic process of ARIMNET Call 2011 10

Questionnair es Applicants (coordinators) 15 questions (2 sections) 1. Call procedure: Focused on the process of applying for funding under the ARIMNET Call 2011 for proposals. 2. Communication: Focused on the process of communication between the applicants and the Call Office and Call Board when applying for funding under the ARIMNET Call 2011 for proposals List of applicants : Only the coordinators of the proposals will answer this questionnaire in order to avoid imbalance between the number of respondents per project. 11

Questionnaire s Evaluators (Evaluation Committee) 13 questions (1 section) 1. Evaluation procedure: Focused on the process of evaluation of proposals under the ARIMNet Call 2011 This questionnaire is addressed to the Evaluation Committee and is based on the Guidelines for Evaluation. In the case of the External reviewers, other questions should have been included. 12

Task 3. 1 Timeline (work done) Tasks Responsible partner INIA contributors 2014 FCT, IAV April INIA ALL April INIA FCT, IAV April Identification of target stakeholders by country for the implementation of the each questionnaire Preparation and insertion of an online survey On-line survey pre-test phase Launch of the on-line survey to implement the 4 questionnaires Analysis INIA ALL May FCT FCT INIA, IAV ALL May June INIA FCT, IAV Summary report INIA FCT, IAV D 3. 1 Call procedure reports INIA July- September. October Preparation of different questionnaires ( call office, call board, applicants (coordinators) and evaluation committee), and circulate them between contributors Circulate them to ARIMNet 2 for comments Consolidation of the 4 questionnaires final version Final version of the 4 questionnaires to be sent to FCT 13

Evaluators survey Mediterranean countries South ; [VALOR] North; [VALOR] 14

Evaluators conclusions Positive § § § § § Good internal relationship among them Good support from Call Office Adequate classification of proposals (most) Relevant the scientific and technical quality of proposals (originality, methodology, etc) Very satisfied with the global impact of the proposals (60%) Very relevant the quality of consortium (most) Relevant the project management and feasibility (all) Enough time available for the evaluation (all) Preferably physical meetings for final decisions (most) 15

Evaluators conclusions Negative § Only 7 respondents § Not good distribution of proposals: § Most of evaluators assessed proposals on topic 1 “production systems” (80%) § A few assessed proposals on topic 2 “food chain” (20%) § Too much work: 17 proposals assessed by each expert (average) § 20% judged not good the quality of the consortium § 20% considered not relevant the added value to the research community 16

Call Board survey Mediterranean countries South / East; [VALOR] North; [VALOR] 17

Call Board conclusions Positive § Suitability of thematic research areas and topics to national priorities (all) § Adequate of information requested to applicants (most) § Distribution of proposals (4 referees per proposal) very good considered § Enough timing for letter of intent and funding decision (most) § Nomination of experts by the Call Board (most) § Funded projects as response to their organization´s policy needs (all considered) § Most considered “cash” as funding mechanism and only a few wanted “in kind” § All considered the call as extra value to Mediterranean basin 18

Call Board results Negative § § § § § Poor communication with Call Office (30%) Main difficulty: time and lack of alignment for the national contracts Second difficulty: availability for funding in some countries Procedure to nominate experts: long and tedious NCPs had no evidence of proposals (letter of intent non compulsory) A lot of complaints from applicants in the application phase The submission tool not useful (half) Only a ranking list Poor scientific information from experts to Call Board Dissemination results not enough 19

Applicants survey Mediterranean countries East; [VALOR] South; [VALOR] North; [VALOR] 20

Applicants conclusions Positive § § § Enough time to prepare the proposal (most) Easily to set up the consortium (half) The electronic submission tool good assessed The ARIMNet website well considered Support from NCPs really well appreciated (most) Timing for different steps: good/poor (fifty/fifty) 21

Applicants conclusions Negative § Main difficult issue: financial aspects (most) § Promotion of the call thorough national websites: useless (more than the half) § Notification letters to coordinators too generic (a quarter) § Letter of intent not clear (half) 22

Call Office survey Timing for different steps 23

Call Office conclusions Positive § § § Pre-announcement good and clear Lo. I useful to indicate number of proposals Good communication with applicants, evaluators Call Office solved all the difficulties (in general) Inputs from Call Office in different phases (application and evaluation): very important 24

Call Office conclusions Negative § Desirable more experience in the preparatory phase § Call documents more clear: ü national eligibility criteria and prioritization, ü identity of applicants (natural person or legal entity) ü role of the experts (write a report per proposal) § Searching of referees: time consuming § Lack of alignment in contract signatures § Difficult communication with some funders 25

Lessons learnt for future calls § FUNDING DECISIONS: consideration of scientific evaluation, much more than national priorities § TOPICS: restricted and specific research topics § CONSORTIA: limited number of partners (minimum and maximum) § DISSEMINATION OF RESULTS: improved and shared with stakeholders § CALL STEPS: Two (PP and FP) for avoiding big number of proposals § TIMELINE: more time for eligibility and evaluation 26

Lessons learnt for future calls § EVALUATION: § Guidelines for referees/experts with clear instructions about the content of the evaluation report § Distribution of reviews from referees to experts in advance, for objections and/or comments § REFEREES: Searching them in advance § NATIONAL CONTRACTS: alignment for signature § NEGOTIATON PHASE: foresee fall back procedures in the Mo. U 27

ARIMNet 2 call 1 THE CALL TIMING • • ) Pre- announcement (30 th June 2014) Launching and guidelines on the web (15 th September 2014) Pre-proposals(1 st December 2014) Full proposals (11 th May 2015) Decision for funding (30 th October 2015) Contract negotiations ( From November 2015) Start of the projects (January – April 2016) 8

ARIMNet 2 Call 1 Timeline (work done) Tasks Responsible partner Contributors 2016 Identification of target stakeholders by country for the implementation of the each questionnaire Preparation and insertion of an online survey INIA ALL May FCT September On-line survey pre-test phase FCT INIA September Launch of the on-line survey to implement the 4 questionnaires Analysis FCT ALL 9 th September INIA FCT October. November Summary report INIA FCT December D 3. 1 Call procedure reports INIA 29

Anabel de la Peña anaisabel. delapena@inia. es Thanks for your attention 30