Reengineering the SIPP The Evolution of a Phoenix

Reengineering the SIPP: The Evolution of a Phoenix Joint Statistical Meetings July 30, 2007 David Johnson 1

Reengineering the SIPP 2

The Unique Value of SIPP • To provide a nationally representative sample for evaluating: – annual and sub-annual dynamics of income – movements into and out of government transfer programs – family and social context of individuals and households – interactions between these items 3

Purpose of SIPP • Recommendation 2 -1: The two primary goals for SIPP should be to provide improved information on the distribution of income and other economic resources for people and families and on eligibility for and participation in government assistance programs. Within these two goals, most attention should be paid to improving the information for people who are economically at risk. A third important but subordinate goal is for SIPP to have a capability to respond to current policy needs for data in topical areas that are related to the core subjects of SIPP. – The Future of the Survey of Income and Program Participation, NAS, 1993 4

Current Uses of SIPP • Department of Agriculture – Model food stamp eligibility and measure food stamp participation • Department of Health and Human Services – Measure the economic effect of disabling conditions on children and adults, and determine "triggers" that cause people to go on or to go off programs. • Social Security Administration – Model SSI benefits, and the restructuring of Social Security such as age threshold changes. • Congressional Budget Office and Congressional Research Service – Use micro-simulation to measure participation in major government programs 5

6

Reengineering Goals • Cost reduction Ø Use of annual data collection • Improved accuracy Ø Lower attrition rates Ø Focused content selection Ø Integrated administrative data • Improved timeliness and accessibility • Improved relevance Ø Improved documentation Ø Improved processing system Ø Ongoing content determination process; use of “hooks” Ø Use of ACS 7

Main Components of the Reengineering Process • Improved Processing System and Collection Instrument • Development of Event History Calendar • Meetings with stakeholders, development of survey content, and use of reimbursable supplements • Use of ACS for sample • Use of administrative records data to supplement and evaluate survey data 8

Timeline of SIPP/DEWS/Re-engineered SIPP (The Order of the Phoenix) December 2005 Stopped work on converting existing SIPP to BLAISE and reprogramming post-data collection processing to SAS January 2006 Preparing for budget release, sessions with key federal stakeholders were held to discuss future SIPP plans February 2006 Publicly launched the “Program of Income, Wealth, and Health Insurance Measurement” and possible end of SIPP data collection in Sept 2006 March/May 2006 CEPR circulated “Save the SIPP” letter. Census presented progress to date, and suggested using linked CPS data and administrative records data June 2006 Brookings Workshop: Concerns expressed by stakeholders about possible data gap and usefulness of admin data. Stakeholder matrices sent to major users in order to identify critical items. It was determined that admin records would only assist in data quality evaluation. 9

Timeline of SIPP/DEWS/Re-engineered SIPP August 2006 First open stakeholder meeting held at Census. Publicly stated the Bureau was refocusing effort to use admin data to supplement data collection (not replace). New survey plans presented – EHC and ‘hooks’ to allow opportunities for reimbursable supplements. Announced plans to continue SIPP with reduced sample from 10/06 -5/07 Dec 2006/ Jan 2007 Another open stakeholder meeting at Census where more details were presented on content, structure, and plans for the new survey. Census presented plans for at CNSTAT Panel meeting. March 2006 With additional funding, announced plans to continue SIPP with reduced sample through Sept, 2007 April/May 2006 Topical content meetings held June/July 2006 Senate and House Appropriations Comm propose additional funding to continue SIPP collection in 2008 10

Current SIPP Status (subject to funding) 2004 2005 2006 2007 2008 SIPP ’ 04 Panel Collection 2/04 -1/08 (reduced sample 10/06 -1/08) 2009 2010 2011 2012 SIPP ’ 08 Panel Full Sample w/ TM Same as ’ 04 Panel Reengineering Planning 2006 – 2011 EHC Paper Test 3/08 Dress Rehearsal Test/Evaluation 9/09 Improved Panel Data Collection 2011/12 and on 11

Improving Processing System • Convert CAPI instrument to BLAISE • Document processing system – Convert from FORTRAN to SAS – Document edit and imputation procedures • Develop processing system in conjunction with survey for reengineered SIPP 12

Event History Calendars • An EHC interview is centered around a customized calendar that shows the reference period under investigation • The calendar contains timelines for different domains, for example, residence history, household composition, work history, and other domains relevant to the topic of study • Landmark events, such as holidays and birthdays can be used to aid the respondent’s memory 13

EHC Format • Survey Instrument – – Annual administration, Jan-April – Follow movers – Limited feedback • Calendar – – Improvement on other designs by integrating more closely with Blaise, the instrument programming language. • Simultaneous development of processing and instrument 14

2008 EHC Paper Test 2007 2008 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Contract with RTI to prepare paper EHC and Training Field Staff Training Prepare SOW for Instrument & Training Field EHC Evaluation Test in MD & IL Receive Admin. Records MD & IL Census development of procedures for administering the paper EHC Evaluation 2007 2008 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep SIPP 2004 Waves 10 - 12 Field EHC Evaluation Test Coding Reference period for paper EHC Evaluation Analysis EHC and SIPP 2004 Additional Validation with Admin. Records Administrative Records from MD & IL to validate SIPP - EHC 15

Event History Calendar Conference • "The Use of Event History Calendar Methods in Panel Surveys" December 5 -6, 2007 at the US Census Bureau • Agenda – Examining the Impact of Event History Calendar Interviewing on Data Quality from Disadvantaged Respondents - Belli – EHC in PSID – EHC in NLSY 97 – EHC in Netherlands – EHC in LA Family and Neighborhood Survey – EHC in British Panel survey – Seam Bias in SIPP 16

Stakeholders and Content Review • Responding users indicated a broad need for most of SIPP core content. • About 40 stakeholders completed matrix • Select areas were added based on lost topical module content. • Held five subject area meetings to discuss specific content – – – Health General income/Government programs Assets and wealth Labor force Demographics and other items 17

Use of ACS Data in Survey Design • Sampling Frame – American Community Survey (ACS) interviewed cases – Existing SIPP Primary Sampling Units (PSUs) – Goal is to use 2 -years worth of ACS sample • Oversample based on similar criteria as SIPP • Sample size similar to SIPP (budget permitting) 18

Use of Administrative Records • National-level prototype – SIPP linked with administrative records data at the person level • Creation of SIPP Synthetic Beta File matched with SSA data • State-level prototype – Many social programs administered at the statelevel – SIPP data linked to social program data from the states of Maryland Illinois 19

CNSTAT Panel • Goals of the panel are to evaluate: – the costs/benefits of various strategies for data linkage, – accessibility of relevant administrative records, – operational feasibility of linking administrative records and survey data, – quality and usefulness of linked data, and – strategies for providing public access to the linked data while protecting the confidentiality of individual respondents 20

Assessing Users’ Needs URL: http: //www. sipp. census. gov/sipp/dews Source: U. S. Census Bureau, Demographics Survey Division, Survey of Income and Program Participation branch Census Bureau Links: Home · Search · Subjects A-Z · FAQs · Data Tools · Catalog · Census 2000 · Quality · Privacy Policy · Contact Us 21

2009 Re-engineered SIPP Dress Rehearsal 2007 2008 2009 2010 2011 Jan --- May --- Sep --- Jan SIPP 2008 Panel – Waves 1 – 9 (planned) 2008 EHC Evaluation Eval. Analysis 2009 Re-engineered SIPP Dress Rehearsal Reference Period Field Activities 2009 SIPP Re-Engineering – Instrument Dev. Systems Tests/Preparation Processing and Evaluation 22

Matching SIPP responses with Social Security Administrative Data on Retirement Benefits NO Retirement Benefit on SSA file NO Retirement Benefit reported in SIPP 84. 8% 1. 3% Retirement Benefit reported in SIPP 2. 9% 10. 9% 23

Matching SIPP responses with Social Security Administrative Data on Retirement Benefits NO Retirement Benefit on SSA file NO Retirement Benefit reported in SIPP 84. 8% 1. 3% Retirement Benefit reported in SIPP 2. 9% 10. 9% 24

Matching SIPP responses with Social Security Administrative Data on Retirement Benefits NO Retirement Benefit on SSA file NO Retirement Benefit reported in SIPP 84. 8% 1. 3% Retirement Benefit reported in SIPP 2. 9% 10. 9% 25