Evaluating NSF Programs Dr. Jennifer Giancola Carney, Abt Associates September 18, 2008
Agenda • Two NSF program evaluations (IGERT & CAREER) – Design & findings – Rationale for methods used – Limitations of methods used – Lessons learned • Q&A discussion Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Integrative Graduate Education and Research Traineeships (IGERT) Program • Since 1998, Ph. D training program (DGE) • Grants to universities who develop new IGERT-related programs (most $$ student traineeships) • Give Ph. D students interdisciplinary research experiences and enhanced professional skills & perspectives • Three phases of evaluation • Implementation study (1999 -2002) • Impact study (2003 -2005) • Follow-up study of graduates (2006 -present) Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
IGERT Evaluation: Began with a Logic Model Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
IGERT Implementation Study • Annual Monitoring: “Who? What? When? ” research questions – Who participates and why? What activities are conducted? – Annual web survey of program participants (PIs & trainees) – Describe the program recruitment strategies, training activities, faculty involvement • Site Visits: “How? Why? ” research questions – What challenges have projects encountered? How have they overcome them? – Interviews with faculty, students, chairs, administrators – Identify common challenges and solutions project management, faculty engagement, implementing interdisciplinary education within universities Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
IGERT Implementation Study (cont. ) • Examine implementation across projects and over time – Mixed methods (quantitative & qualitative data) • Data used for GPRA reporting, program management, revisions to solicitations, sharing common solutions with IGERT PIs • Limitations – Little information on longer-term effects of IGERT or broader program impacts on faculty and the university – No comparison to non-IGERT experiences to take into account overall trends in graduate education Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
2003 IGERT Impact Study • Impact study: So what? ” research questions – What have been the outcomes for participating IGERT faculty and students as compared to non-participating faculty and students? – Has there been any institutional impact of IGERT funding? • IGERT participants (PIs, dept chairs, faculty, students, administrators) and Non-IGERT participants (comparison group) Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
IGERT Comparison Group • Provide a counterfactual for what would have been had IGERT not existed. Needs to control for “academic quality” and variations among STEM disciplines • Matched each IGERT department to a non-IGERT department with whom they compete for graduate students – Vulnerable to selection bias: Outcomes may be due to preexisting characteristics of IGERT students, not to IGERT program • Examples of reported findings: – Can say: “IGERT trainees engage in more interdisciplinary activities as graduate students than non-IGERT students. ” – Cannot say: “IGERT causes students to engage in more interdisciplinary activities. ” (Maybe these students would have sought out i/d activities regardless. ) Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
IGERT Impact Study (cont. ) • Benefits – Examine value of IGERT for: • Departmental recruitment • Student preparation • Faculty interdisciplinary involvement • Institutional offerings and support for interdisciplinary education – Assess against counterfactual of “traditional” graduate ed. • Limitations – Focused on current participants – Tested lots of outcomes – hypothesis generating (not confirming) – No data on longer term outcomes for graduates Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
2006 IGERT Follow-up Study • Graduate study: “What? ” “So what? ” questions: – Where do IGERT graduates go and what do they do? Are they any different from non-IGERT graduates? Has IGERT helped prepare them for their chosen careers? – IGERT graduates and comparison group of non-IGERT graduates • Presenting detailed descriptive data on IGERT graduates • Limiting outcomes tested with comparison group to key outcomes (hypothesis confirming, though still selection bias) • Challenge: locating graduates – Monitoring system had info on point of contact – Easier to find those in academic positions versus non-academic positions. Introduces sample bias into results – will conduct nonresponse bias during analysis. Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Faculty Early Career Development (CAREER) Program • NSF’s primary support mechanism for junior faculty members since 1995. – Grants to individual faculty members • Support the research and early career advancement of junior researchers • Promote the integration of research and education: – Individual awardees – Changing university culture • Most recent evaluation: 2005 -2008 Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
CAREER: Research Questions • Descriptive questions re: perceptions of CAREER – How do stakeholders at NSF perceive the CAREER program and its relationship to the mission of NSF? – How do faculty members in departments that host CAREER awardee(s) view the CAREER program and its relationship to their research and educational missions? • Impact questions – What is the impact of CAREER on the research activities and career advancement of awardees? – What is the impact of CAREER on the integration of research and education by faculty members? Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
CAREER: Methodology • Descriptive Study – Interviews with NSF Program Officers – Survey of 700 department chairs – Site visits 22 departments – Samples representative of population in question (but no comparison group) Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
CAREER: Methodology (cont. ) • Quasi-experimental evaluation of impact on awardees – CAREER Awardees – Comparison group of Non-Awardees (same research potential and interest in integration of research and education) • Unsuccessful CAREER applicants who won another NSF grant as PI w/in 5 years of CAREER application • Matched using propensity scores (reduces selection bias) • Limited outcomes tested (confirming hypotheses) Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
CAREER: Findings • Description of how program goals are interpreted within and outside of NSF Inform program management • Description of characteristics of awarded PIs NSF program reporting (GPRA, etc. ) • Assessment of grant’s impact on awardees (“Receipt of a CAREER award increases the likelihood of receiving tenure”) Inform decisions about program continuation or modification Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Lessons learned • Know thy program: Until you understand the intervention, you cannot assess outcomes – Logic model, articulate goals – Develop indicators / measures of program success • Clearly define your research questions – Prioritize - you cannot evaluate everything – Identify data needs for reporting, decision-making – Be realistic (ask questions that can be answered about indicators that can be measured) • CAREER: “impact on institutional culture” Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Lessons learned (cont. ) • Identifying appropriate comparison groups – What’s the right counterfactual? Each comparison option allows you to answer different questions. Choose the option which best addresses the research questions. • IGERT - other interdisciplinary programs? Same or different institutions? All STEM students nationwide? – Choose right level of rigor (developing or testing hypotheses? ) – Consider risk of selection bias – Change over time (longitudinal studies; pre/post) – Take advantage of available data available • National datasets • CAREER – data available to do PS matching 9/18/2008 Abt Associates AGEP Capacity Building Meeting Presentation
Lessons learned (cont. ) • Ground each subsequent phase in findings from previous work: – Work from exploratory / descriptive evaluation to more summative / confirmatory evaluation. Each phase can answer questions raised (or not answered) in previous phases. • IGERT: Implementation Impact Graduate Followup • Take advantage of different levels of data collection – Qualitative versus quantitative; single versus cross-site • IGERT: Richness of single site visits enabled future studies Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Lessons learned (cont. ) • Think long term – Begin evaluation when program begins – Plan now for information you will need in the future • IGERT: tracking graduates • New study (GK-12) – building comparison group today for work in future Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Questions? Abt Associates AGEP Capacity Building Meeting Presentation 9/18/2008
Скачать презентацию Evaluating NSF Programs Dr Jennifer Giancola Carney Abt
2136ccf290a117d6c3c6265a34dc686a.ppt