History and design of the European Community Innovation

History and design of the European Community Innovation Survey (CIS) questionnaire Prof. Anthony Arundel Australian Innovation Research Centre, University of Tasmania, Australia and UNU-MERIT, Maastricht, the Netherlands

1. Introduction

Responses to the CIS from over 200, 000 enterprises Conducted every two years (7 surveys so far) CIS 2014 under development

Uses of the CIS-1 • Policy relevant indicators used in the 2013 European Innovation Union Scoreboard: 1. 2. 3. 4. 5. Non-R&D innovation expenditures Percent of SMEs that innovate in-house Percent of SMEs that collaborate on innovation Percent of SMEs with product or process innovations Percent of SMEs with organisational/marketing innovations 6. The share of total sales from innovative products

National performance on the Innovation Union Scoreboard, 2013

Benchmarking Policy uses of innovation surveys: Benchmarking motivates, but indepth analysis is required to know what to do. In-depth analyses

Number of academic papers using Community Innovation Survey data (1994 -2012) 500 100 451 450 428 Cumulative Number of studies 400 367 Annual Count 350 316 300 261 250 211 200 165 182 140 150 81 100 50 4 9 14 20 27 34 90 80 70 60 50 40 30 100 20 50 10 0 0 1994 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Publication year

Research themes over time 40 35 Who innovates Innovation intensity Number of papers 30 Innovation strategies 25 Effects on performance 20 15 10 5 0 1994 1996 1997 1998 1999 2000 2001 2002 2003 2004 Publication year 2005 2006 2007 2008 2009 2010 2011 2012

Effects on performance 1. CIS output indicators - Change in annual turnover or employment over three years - Innovation sales share - Crude estimate of productivity 2. Data linkage to administrative data for profits, productivity, production etc. 3. Panel data: can examine effect over time on innovation sales share

2. R&D versus innovation

The origins – R&D surveys • First attempts to measure R&D: – 1917: First R&D survey, with further experimentation up until the late 1930 s. – 1953: First large American R&D survey. – 1963: First international R&D survey. – 1981: OECD considers R&D data quality and international comparability acceptable.

R&D focus of early innovation research • Are innovation activities a supplement to R&D? (perspective of Frascati Manual) or • Is innovation is a unique activity that may or may not involve R&D?

13

June 2008 14

What is going on? • There is a lot more than R&D – other factors contribute to economic development and productivity improvements. – Technology purchases (adoption as in the fishing sector) – Organisational innovation – Innovative activities that do not require R&D 16

Ways of innovating without R&D • Technology adoption • Minor modifications or incremental changes, including use of engineering knowledge • Imitation including reverse engineering • Combining existing knowledge in new ways

Percent of firms innovating without R&D 45. 4% of innovative firms in this sample did not perform R&D Of R&D performing firms, 63% reported innovations that did not require R&D

Method of innovation (from least to most advanced in-house capabilities) for product/process innovation and for organisational innovation

R&D versus innovation • R&D is largely an activity of manufacturing firms – much rarer in services – Innovation data are a better measure of innovation activities in services • Definition of R&D (requirement for scientific or technological novelty) is confusing for firms in services – What about development without research?

Definition of Innovation • An innovation is the implementation of a new or significantly improved product (good or service), process, new marketing method, or new organisational method. Source: 3 rd revision, Oslo Manual, OECD – Innovation is NOT invention. – It does NOT require creative effort – adoption of new manufacturing technology is an innovation. – It does NOT need to be commercially successful. 22

Percent of innovative firms in Europe that donot perform R&D • Innobarometer 2007: 52. 5% of innovative firms did not perform R&D. • CIS-3 (2002 - 2004): 50% did not perform R&D

Percentage of Innovative and R&D Performing Firms by Size: CIS-1 Estimates for 8 Countries Combined 100 90 80 70 60 50 Innovators R&D performers 40 30 20 10 0 < 20 20 -99 100 -249 250 -499 500 -999 1000 -1999 Number of Employees Countries: Germany, Italy, Belgium, Netherlands, Luxembourg, Ireland, Denmark, Norway > 2000

Innovative status of an enterprise in the CIS • An innovative firm has introduced one or more types of innovations within a defined period of time (3 years). • Innovative status can depend on the maximum degree of novelty of at least one of the firm’s innovations: – ‘World first’ – often based on in-house creative effort. – New to the firm’s market: could be based on creative effort (reverse engineering, engineering improvements) or bought from another firm. – Only new to the firm: purchased off the shelf.

Innovation as a creative versus diffusion activity Maximum Creative effort New product/process resulting from large investment in in-house R&D Purchased technology is adapted to firm’s needs Minimum creative effort New technology bought ‘off the shelf’

R&D status by change in turnover 2004 to 2006 Source: Arundel, Bordoy & Kanerva, 2008 Results confirmed in an econometric model that controls for size, sector, country, total innovation expenditures, innovative capabilities.

3. History of the Community Innovation Survey (CIS)

First innovation surveys • Object based: collect data on specific innovations identified in trade journal advertisements or other sources. – Townsend, 1981, Kleinknect (various studies) • De Bresson & Murray (1984): stratified random survey of all Canadian firms (not just R&D performers), asked for descriptions of three most important innovations. – Defined innovation as “any new or improved product which has withstood the trial of the market and generated a return on investment, or a new or improved process for commercial production. By new we mean new to Canada”.

Subject-based innovation surveys • The focus is on the firm or enterprise, with questions about its innovation activities. • First surveys in the 1970 s in Canada, limited to R&D performing firms. • Advantages over an object approach: – Can collect information on all types of innovations – Can cover process and other types of innovations.

Experimental innovation surveys of the 1980 s • MIT (United States) & Fraunhofer (Germany) surveys only sent to R&D performing firms. • Scandinavian survey: conflicting definitions, product innovations defined as deriving from ‘R&D projects that resulted in marketable new products”. • Ifo (Germany) referred to new or improved products and processes. • Dutch, French and Italian surveys asked if the firm had introduced an innovation that was new to the firm or its local region.

First Oslo Manual (1992) • “The core task is to integrate an understanding of the R&D contribution with an account of the non-R&D inputs to the innovation process” – Innovation is a supplement to R&D • What differentiates a change from an innovation are “elements of novelty and significance” – (similar wording to the Frascati Manual) • But, definition of an incremental innovation is ambiguous about the need for R&D: “an incremental product innovation is an existing product whose performance has been significantly enhanced or upgraded.

CIS-1 (implemented in 1993) • CIS-1 definition of a product innovation • “A significant innovation is a newly-marketed product whose intended use, performance, characteristics, technical construction, design, or use of materials and components is new or substantially changed. ” • “An incremental innovation is an existing product whose technical characteristics have been enhanced or upgraded. ” – Whether or not R&D is required is left undefined.

CIS-2 to CIS-4: removing ambiguity over R&D • CIS-2 (implemented in 1997): – Asked respondents ‘who developed’ their innovations and included the option ‘mainly other enterprises or organisations’ • CIS-3 (implemented in 2001): – Changed definition of product and process innovations to state that they only needed to be ‘new to your enterprise’. • CIS-4 (implemented in 2005): – Requirement for R&D for collaboration removed, with definition of collaboration changed to ‘joint R&D and other collaboration projects’

Survey CIS-1 CIS-2 Observation Main changes or additions compared to the previous survey period 1990 -1992 Added questions on who developed product and process innovations. 1994 -1996 Cooperation question extended to cover more than R&D. Deleted questions on sources of new technology, technology transfer outside the enterprise, appropriation methods, and product life cycles. CIS-3 1998 -2000 Changed question on the innovation sales share to the share of sales from unchanged, new-to-firm and new-to-market products. CIS-4 2002 -2004 Added questions on three types of organizational innovation and two types of marketing innovation, plus questions on the effects of organizational innovation. Under product innovation included separate questions for goods and services. Asked about three types of process innovations. CIS-2006 2004 -2006 Frequency increased to every two years. Implemented cognitive testing for all question changes and additions. Increased coverage of organizational and marketing innovation. One-page module for questions of high policy interest, with the first module on environmental innovation. Separate question on expenditures for design. Module on creativity and skills. Module on strategies and obstacles to growth. Reintroduced a modified version of the CIS-1 appropriation question. New questions on public procurement and innovation CIS-2008 2006 -2008 CIS-2010 2008 -2010 CIS-2012 2010 -2012

But, the forces for R&D might be back, with a proposal to combine the CIS and R&D surveys!

Countries combining the CIS 2010 with the R&D survey 76% No Yes, but only for some respondents 3% Yes, for all respondents 14% 7% Not relevant 0% 10% 20% 30% 40% 50% 60% 70% 80%

4. The CIS today

• The European CIS is a ‘general’ innovation survey that covers a large number of topics, none of them in particular depth. – Exception: knowledge sourcing

Limitations of the CIS • Cross-sectional survey –only a few countries have panel data: – Norway, Germany, Tasmania (not a CIS survey but has some similar questions) • As a general rule, does not include questions on rare activities – These questions collect little information of value but add to the response burden

Who answers the CIS? Ideally, the CEO or R&D manager

Who might use the CIS? Policy analysts Businesses Academics

Three different users • Business managers – Need timely data for benchmarking against other firms A general failure of large surveys, but feasible for smaller surveys. • Academics – Need access to micro data for testing innovation theory (many data access limitations) • Policy community (main target) – Need results that are directly relevant to policy issues (rarely provided by academics); often rely on indicators and descriptive analyses but econometrics are useful if properly explained.

Definition of innovative enterprises • At least one product, process, marketing or organisational innovation in the preceding three years. • Only needs to be ‘new to the firm’ – includes simple technology adoption. – Criticized for being too broad

Indicator for ‘how’ enterprises innovate: Spain EU average

Finland versus Portugal

Main players in the CIS • Eurostat (statistical agency of the European Union) coordinates the design of the CIS questionnaire and makes recommendations for the survey methodology. • CIS Working Group: All European countries that implement the CIS are members. OECD and the European Commission also attend. They assess the revised questionnaire from the Task Force and can request changes. Decision to accept based on consensus. • National Statistical Offices: Responsible for implementing the CIS in their country. By EU law, must provide specific indicators to Eurostat; can voluntarily provide full data for the Eurostat Safe Centre and anonymized data.

CIS Task Force

Topics covered in the CIS • Firm’s main markets • Product and process innovation – – – – • • Who developed Innovation sales share (output variable) Novelty (world-first, Europe first, country first) Innovation activities & expenditures Public support Sources of information Types of collaboration partners Organisational and marketing innovations Barriers to innovation (drivers in CIS-2014) Innovation objectives (not asked in every CIS) Module of questions of policy interest

Percent of UK firms reporting cost factors as of high importance as barriers to innovation (CIS-4) Source: D’Este et al, 2012

Proposed new question on barriers for non-innovative enterprises

Importance of innovation drivers

Modules • CIS 2008: – Environmental innovation • CIS 2010: – Creativity and skills (including how the firm encourages creativity) • CIS 2012: – Strategies and obstacles for growth (meeting enterprise goals) – Appropriation methods (also used in CIS-1) • CIS 2014: – Improved questions on environmental innovation, including importance of different drivers

Required & optional CIS questions • Many questions required by EU law (EC regulation 1450/2004) • Optional questions include: – Novelty (world first) of product and process innovations – Innovation module – Questions on public procurement, appropriation • National statistical offices can and do add other questions of high relevance to their own policy issues.

Keeping the CIS relevant CIS questions used in benchmarking need to stay the same, but new questions and topics are needed every year to attract academic researchers and evidence for topical policy questions. In response, since CIS 2008 the questionnaire includes a module of one-off questions that can change in every CIS.

System for continuous improvement of the CIS 1. Cognitive testing of all new questions & major changes 2. Quality reports for each CIS survey -non-response rates for specific questions - Experiences of National Statistical Offices 3. Research by NSOs on methodology - Effect of combining R&D and innovation surveys - How respondents understand key concepts 4. Bi-annual surveys of NSOs - Survey method (online, mailed, face-to-face) - Post survey methods for improving data quality 5. Surveys of policy users - What innovation survey data do they use 6. Database of academic papers that use the CIS - Which questions do they use and do not use - New questions that academics would like for their research

Cognitive testing • Face-to-face interviews with firm managers – Goal to ensure that all questions are understood as intended by all respondents, and – Respondents can provide accurate responses • Introduced on a consistent basis in 2004 (recommendation of report on CIS-4).

Logical error

Logical error due to question placement

Accuracy of responses (not all problems can be solved)

Two questions in one

Item non-response rates: use of quality reports to track progress • Improvements to question design: – Decreased item non-response rates for the innovation sales share question from approximately 25% to less than 3% by CIS-4 – Decreased item non response rates for the question on innovation expenditures from 36% to approximately 15% by CIS-4.

Research by National Statistical Offices • Norway tested the effect of combining the R&D and CIS surveys – Found small declines in number of innovative firms. • Netherlands compared responses to online and mailed surveys. – Found much higher share of innovators to online surveys (70%) compared to mailed surveys (50%)

NSO survey • For every CIS, survey NSOs on a range of relevant questions • Topics include: – issues with new questions in the previous survey – survey methods, particularly the use of online surveys – New questions that may have tried and which might be of interest to all NSOs

NSO survey results Main survey method for CIS 2010 60% 55% 50% 40% 38% 30% 20% 7% 10% 0% 0% Mail Online Telephone interview 0% Face to face Other

Combinations of survey methods for CIS 2010 Mailed with online as secondary method 14% Online with mail as secondary method 45% Mailed with telephone interview as secondary method 17% Online with telephone interview as secondary method 17% Other combinations 7% 0% 10% 20% 30% 40% 50%

Software for online surveys 25 20 20 15 10 5 3 1 0 0 Created online Survey monkey survey internally Lime survey 24 countries provided an online option (80%) Other commercial software

Possible questions for removal from CIS 2014 (% of countries) First choice for removal Second choice for removal Last choice for removal 31. 0% 34. 5% Cooperation and type of cooperation (Q 6. 2 – Q 6. 4) 13. 8% 24. 1% 62. 1% Public sector procurement and innovation (Q 10) 48. 3% 31. 0% 20. 7% Sources of information (Q 6. 1)

Number of NSOs that added a non-standard question to their national 2012 CIS survey 18 16 16 14 14 12 10 8 8 6 4 2 0 Yes No Useful for future CIS surveys

Surveys of policy users • Infrequent because expensive – Last large scale survey in 2005 – New survey planned for 2014 • Interested in the types of indicators that policy makers find useful • If CIS results have influenced policy

2005 Policy survey results R&D indicators CIS indicators

Academic users • Small-scale surveys of 10 to 20 leading academics, based on number of publications using CIS data. • Ask which types of new questions they would like, which questions perform poorly in analyses, etc. • Maintain database of publications using the CIS.

Academic use of CIS data Results for 2012 incomplete

Results for 2012 incomplete

5. Conclusions: Key lessons for a useful innovation survey

10 recommendations: not all of which have been fully implemented in Europe 1. Collect data on question quality, including item non-response rates, ‘not relevant’ response rates, number of respondents who contact you for better explanations, etc. 2. Cognitively test all new questions and major changes to existing questions (50 interviews) 3. The working group to design innovation questionnaires should include representatives from both statistical offices and user communities.

4. The addition or deletion of CIS questions should be backed by empirical data on data quality and on the relevance of the question to users. 5. Allow room for experimentation with new questions – perhaps by province? 6. System for continuous improvement. 7. Add new questions in each CIS – keep it interesting for respondents and for users.

Recommendations that have not been fully implemented in Europe 8. Provide academics with timely access to the data. 9. In return for data access, insist that academics provide a ten page evaluation of the relevance of their research for policy. 10. Develop a strong interface between the policy community and the National Statistical Office to encourage the uptake of CIS results for policy uses.

And two more…. • Create a panel data set (possibly a subsample) • Provide data linkage to other data, especially for outcome analyses

Thank-you! For questions or more information a. arundel@utas. edu. au Detailed CIS reports available for last 5 CIS surveys.