Results of the Assessment of the Defense Experimental

Results of the Assessment of the Defense Experimental Program to Stimulate Competitive Research (DEPSCo. R) Brian Zuckerman Presented to COSEPUP Evaluation of the National Science Foundation's Experimental Program to Stimulate Competitive Research (EPSCo. R) and Similar Programs in Other Federal Agencies December 17 th, 2012 Slide 1

Study Origin and Timeline • FY 2008 Defense Authorization Act (Public Law 110 -181), Section 241, instructed the Secretary of Defense to utilize a defense Federally Funded Research and Development Center (FFRDC) to carry out an assessment of the DEPSCo. R program. • Institute for Defense Analyses was tasked to conduct study February 2008 • Results briefed to Senate and House Armed Services Committee staff (SASC/HASC) November 2008 • Study results cleared for public distribution January 2009 Slide 2

Legislative Mandate for Study 1. Tangible results and progress toward the objectives of the program a. Applications used by, or supportive of, operational users b. Expanded national research infrastructure 2. Activities consistent with statute 3. Assessment of program elements 4. Assessment of activities of state committees 5. Advantages and disadvantages of institution & state based formulas 6. Mechanisms for improving the management and implementation of the program Slide 3

Descriptive Statistics • Between 1993 and 2008, 729 total DEPSCo. R awards – 546 individual Principal Investigators (PIs) » 121 PIs with multiple awards » 42 with three or more » 1 PI with eight awards – 22% of PIs have won 42% of awards • 1993 -2008 funding of $243 million – Decline after 2000 peak partially reversed in 2008 competition • 27 states and territories (states) have been eligible for at least one year since program authorized in current form in 1995, plus Missouri (eligible in 1993) – All eligible states except for the Virgin Islands have won awards • 19% (5) of eligible states have won 35% of awards – Montana, Alabama, Oklahoma, Nebraska, South Carolina • 7% (5) of institutions have won 28% of awards – Montana State, U. Nebraska Lincoln, U. Wyoming, West Virginia U. , U. Arkansas Slide 4

DEPSCo. R Program Objectives 1. To enhance the capabilities of institutions of higher education in eligible states to develop, plan, and execute science and engineering [S&E] research that is competitive under the peer review systems used for awarding federal research assistance 2. To increase the probability of long term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government for science and engineering research Slide 5

The DEPSCo. R State Share of DOD S&E Increased Source: National Science Foundation (NSF) Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCo. R program database of funding totals by state and year Note: Graph includes all states ever involved in DEPSCo. R program 1995 -2008 Note: The dotted lines represent linear regression models applied to the data Slide 6

DEPSCo. R Funding As a Percentage of DOD Funding in DEPSCo. R States Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCo. R program database of funding totals by state and year • DEPSCo. R has declined in importance as a source of funding for eligible states since 2000 Slide 7

“Success” Varied Among States Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCo. R database, and the DEPSCo. R program database of funding totals by state and year • DEPSCo. R-eligible states fell into four groups: – 6 states Near or above threshold (AL, HI, LA, MS, NM, SC) – 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV, ND, SD) – 6 states Middle (AR, DE, KS, OK, RI, TN) – 4 states + 2 territories Lagging (NH, PR, VT, VI, WV, WY) Slide 8

Average DEPSCo. R Funding as a Fraction of DOD Funding by State Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCo. R database, and the DEPSCo. R program database of funding totals by state and year Note: Average calculated as total DEPSCo. R funds during eligible years divided by total DOD funds during eligible years • DEPSCo. R share of university S&E R&D funding varies substantially by state • More than 60% for VT, WY • “Rising fast” states’ (AK, ID, KY, ME, MT, NE, NV, ND, SD) DEPSCo. R shares decline 2001 -2005 compared with 1993 -2000 Slide 9

Legislative Mandate for Study 1. Tangible results and progress toward the objectives of the program a. Applications used by, or supportive of, operational users b. Expanded national research infrastructure 2. Activities consistent with statute 3. Assessment of program elements 4. Assessment of activities of state committees 5. Advantages and disadvantages of institution & state based formulas 6. Mechanisms for improving the management and implementation of the program Slide 10

Expanded National Research Infrastructure (1) • Involving new investigators: – Recent (2006 2008) cohorts had about 60% new PIs – Most Army funded DEPSCo. R awardees (82%) had not previously received funding from Army Research Office (ARO) – 56% of PIs had been funded by the NSF either previous to or within the same year of their first DEPSCo. R award • Training graduate students and postdoctoral fellows: – ARO and Office of Naval Research (ONR) data suggest that awards fund about 1 Ph. D, 1 Master’s degree, 2 postdocs • Building physical infrastructure: – DEPSCo. R awards have supported purchase and maintenance of equipment but data not collected systematically by services Slide 11

Expanded National Research Infrastructure (2) • Leveraging new funding for defense-related research is limited: – 8% of non DEPSCo. R ARO awardees in DEPSCo. R states received a DEPSCo. R award before (or in the same year as) their first non DEPSCo. R ARO award – 4 DEPSCo. R awardees (less than 1%) won a DOD Multidisciplinary University Research Initiative (MURI) award after or in the same year as their first DEPSCo. R award » 2 of these investigators received non DEPSCo. R DOD funding before their first DEPSCo. R award • Leveraging other funding: – 63 DEPSCo. R awardees (12%) received their first NSF funding subsequent to their first DEPSCo. R funding Slide 12

Legislative Mandate for Study 1. Tangible results and progress toward the objectives of the program a. Applications used by, or supportive of, operational users b. Expand national research infrastructure 2. Activities consistent with statute • Activities were found to be consistent with statute 3. Assessment of program elements • Available data on DEPSCo. R program activities and outcomes are insufficient for monitoring and evaluation purposes 4. Assessment of activities of state committees • State committees prioritized proposals that met state infrastructure development goals and reflected the mission/research needs of DOD • Committee processes varied widely from state to state and limited and variable data prevented detailed assessment 5. Advantages and disadvantages of institution & state based formulas 6. Mechanisms for improving the management and implementation of the program Slide 13

Legislative Mandate for Study 1. Tangible results and progress toward the objectives of the program a. Applications used by, or supportive of, operational users b. Expanded national research infrastructure 2. Activities consistent with statute 3. Assessment of program elements 4. Assessment of activities of state committees 5. Advantages and disadvantages of institution & state based formulas 6. Mechanisms for improving the management and implementation of the program Slide 14

Approach Taken • Assessment focused on comparing the current state-based formula with an institution-based criterion based on a maximum threshold for DOD S&E research funding • As directed in the study legislative mandate, particular emphasis was given to supporting defense missions and expanding the nation's defense research infrastructure – Since the legislative charge for the assessment does not specify how “expanding the nation’s defense research infrastructure” should be interpreted, the assessment considered advantages and disadvantages using a variety of possible interpretations • Assessment also considered alternative criteria: – Indicators of state level S&T capacity (SEI, Milken Institute) – Normalization by state population (Census) • State-based and institution-based criteria can be combined – PI level criteria are also possible Slide 15

Effect of Institution Based Funding Threshold Maximum DOD Funding Threshold All Universities With Non -Zero DOD Research Funding in 2005 Eligible for the 2008 Competition Potentially Eligible for the 2009 Competition No limit 360 77 114 $10 million 316 75 109 $5 million 269 65 90 $3 million 231 55 76 $1 million 157 36 52 • 77 institutions in 2008 DEPSCo. R-eligible states (360 total) received nonzero research funding from DOD in 2005 – 38 Carnegie “Very High” or “High” research institutions in DEPSCo. R states • $5 million threshold would make 269 institutions eligible – Twelve of the 77 institutions in currently eligible jurisdictions (e. g. , University of Delaware, University of Nevada, Brown, Clemson, Vanderbilt, University of Nebraska) would become ineligible • Considering only Carnegie “Very High” or “High” research universities, shift would increase number of eligible universities from 38 to 121 Slide 16

Definition of “Expanding National Research Infrastructure” • If interpreted as increasing equity in funding among states or achieving state -level infrastructure goals, a state-based formula would be advantageous – Current state based formula for eligibility harnesses the state EPSCo. R committees to coordinate infrastructure and capacity building at the state level – If intent is to increase the competitiveness of historically underrepresented states, eligibility can easily be determined at a state level • If interpreted as involving new investigators or institutions in defense-related research, an institution-based formula would be advantageous – Allows targeting of programmatic resources toward investigators at institutions that have not historically built relationships with DOD – Approach taken by late 1980 s/early 1990 s DOD Research Initiation Program – While a state based approach includes the flexibility to channel DEPSCo. R proposals toward historically underrepresented universities or new investigators within an eligible state, the institution based approach allows greater flexibility to target underrepresented universities and investigators throughout the entire country • Could not be determined whether state-based or institution-based approach would elicit more qualified applications to support defense missions – Larger number of eligible institutions implies more proposals, but quality indeterminate Slide 17

Effect of State Population Normalization ’ 05 DOD S&E R&D Funding, by State • Green states were eligible in 2008, red states are “graduates”, blue states were never eligible • Comparing top chart with bottom shows dramatic difference in order ’ 05 Per capita DOD S&E R&D Funding, by State • DEPSCo. R graduates among highest per capita recipients • Several DEPSCo. Religible states (e. g. , AK, DE, ND, RI, MT, SD) above average in funding per capita Slide 18

Other Capacity Indicators • Some DEPSCo. R-eligible states (e. g. , NH, RI, DE, KS) above average in Milken Institute state S&T index – Some “graduated” states below average • Some DEPSCo. R-eligible states (e. g. , DE, KS, NE, NH, RI, VT) in the top or second quartiles for more than half of Science and Engineering Indicators’ seven R&D output measures Slide 19

Legislative Mandate for Study 1. Tangible results and progress toward the objectives of the program a. Applications used by, or supportive of, operational users b. Expanded national research infrastructure 2. Activities consistent with statute 3. Assessment of program elements 4. Assessment of activities of state committees 5. Advantages and disadvantages of institution & state based formulas 6. Mechanisms for improving the management and implementation of the program Slide 20

IDA Recommendations • DOD should change the current process for review of proposals to focus more heavily on investigators’ future potential to conduct research rather than on their current research capabilities • DOD program managers should be formally encouraged to serve as mentors and facilitators for DEPSCo. R investigators seeking to engage in further defense-related research • DOD should create data systems that will allow systematic tracking of DEPSCo. R activities and outcomes • Congress should re-examine and consider clarifying ambiguities in the DEPSCo. R legislative mandate • Once the DEPSCo. R objectives have been clarified, redesign the program with a strategy for enhancing competitiveness at relevant level (e. g. , individual, institution, state) Slide 21

Some DEPSCOR Legislative Language Is Inconsistent • Objectives specify that the research institution is the level at which competitiveness is to be enhanced but authorizing legislation also specifies that eligibility for DEPSCo. R be determined at the state level – 2008 change to eligibility criteria leaves ambiguous whether Congress intends DEPSCo. R to target competitiveness at the institution or state level • Objectives specify that goal is to increase probability of longterm growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government but eligibility determined based on DOD funding – Statute does not discuss whether Congress intends DEPSCo. R should focus narrowly on [6. 1] research that is relevant to DOD missions and priorities although this is how program is run Slide 22

Recommendations for Legislative Clarification • To ensure that the program is implemented in a manner that meets current legislative priorities, Congress should clarify whether – The program is intended to increase competitiveness for federal research funding in general or for particular types of research funding – The program is intended to fund primarily basic research, primarily applied research, or a combination – The primary unit at which competitiveness should be enhanced is the institution, state, or other (e. g. , individual investigator) • Eligibility criteria, funded activities, and other program elements should be structured in accordance with the program’s objectives Slide 23

Competitiveness can be Enhanced at Different Levels • DEPSCo. R supports individual or small-group research projects and can therefore be understood to primarily target capacity-building at the level of the individual – Though supporting training and purchase of equipment target capacity building at level of the institution as well • DEPSCo. R also operates at state level (e. g. , involvement of EPSCo. R committees, state-based eligibility criteria) • While it might be argued that institutional competitiveness depends on individual competitiveness and state competitiveness depends on institutional competitiveness, these dependencies are neither straightforward nor selfevident Slide 24

Structure Program to Enhance at the Desired Organizational Level • Assessment identified other programs that might serve as potential models for DEPSCo. R – State level: NSF EPSCo. R/Research Infrastructure Improvement (RII) Awards » One per state; integrated into state S&T plan; EPSCo. R committee involvement; capacity building rather than research – Centers: NIH IDe. A/Centers of Biomedical Research Excellence (COBRE) » Large team research, equipment, mentoring; milestones for transition to support by standard NIH mechanisms – Investigator level: DOD Research Initiation Program (RIP) or NIH/IDe. A Academic Research Enhancement Award (AREA) Program » Institution based eligibility criteria; small research awards to individual investigators; build relationships/expertise to allow for transition to support by standard research mechanisms Slide 25

Backup Slides Slide 26

State Eligibility over Time Source: IDA analysis of DEPSCo. R BAAs and DOD DEPSCo. R press releases Note: Red cells denote years in which jurisdictions were not eligible, and blue cells denote years in which states were eligible. Note: Missouri, which was eligible only in the 1993 -4 competition, was not included in the table. Slide 27

No Obvious Correlation Number of DEPSCo. R awards and program funding Linear regression model of DEPSCo. R state share of DOD university funding Slide 28

DEPSCo. R Emphasis is on Basic Research • Research oriented towards developing operationally useful devices or components is not a legislatively-mandated program goal • DEPSCo. R projects mostly fund basic research – Not a statutory requirement – Administered by basic research organizations within Air Force and Army, plus Office of Naval Research (ONR) » Interviews with DEPSCo. R program officers suggest that more applied research does not perform well in review – DEPSCo. R program officers tend to track transitions to 6. 2 or 6. 3 research, but not further • Two transitions to operational use were found Slide 29

Two Transitions to Operational Use Were Found • Ronald De. Vore, University of South Carolina: Wavelet mathematics for image compression for tactical applications – De. Vore and colleagues collaborate with program managers at Naval Air Warfare Center NAWC at China Lake to deliver wavelet based image processing platform – Charles Creussere of NAWC implements wavelet based image processing system for navigation in the Tomahawk Block II program • Michael Pursley, Clemson University: wireless, mobile, distributed, multimedia communication networks – Pursley and colleagues working since 1970 s with ITT on tactical radio development – Group used DEPSco. R funding to support research that provided better anti jam communications and greater multiple access capability – Research led to the Soldier Level Integrated Communications Environment (SLICE) wide band networking waveforms that have been integrated into the SINCGARS radio Slide 30