A Comparison of Survey Reports Obtained via Standard

A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar * Priming Evaluation * Jeff Moore, Jason Fields, Gary Benedetto, Martha Stinson, Anna Chan, and Jerry Maples U. S. Census Bureau XXV International Methodology Symposium: Longitudinal Surveys: from Design to Analysis Symposium international sur les questions de méthodologie Statistics Canada October 27 -30 2009 Gatineau, Qc, Canada

SIPP Basics National panel survey – Since 1984 with sample size between about 11, 000 to 45, 000 interviewed households The duration of each panel varies from 2½ yrs to 4 yrs The SIPP sample is a multistage-stratified sample of the U. S. civilian noninstitutionalized population The survey uses a 4 -month recall period – 3 interviews / year The sample is divided into 4 rotation groups for monthly interviewing Interviews are conducted by personal visit and by decentralized telephone

The Save the SIPP Campaign New York Times Editorials – Discovering What Happens Next – Save our SIPP – The Continuing Saga of SIPP NY Representative, Carolyn Maloney stated: “I want to thank all of the policy groups and economists who worked so hard to help reverse the Administration’s original SIPP decision. Because of their dedication and hard work, the Administration came to understand how important SIPP is to creating and implementing good public policy. ”

SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12 -month reference pd. , event history calendar methods

EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between the occurrence and timing of events EHC Encourages Active Assistance to Rs - flexible approach to help elicit an autobiographical “story”

Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results BUT, Important Research Gaps - data quality for need-based programs? - comparison to 4 -month reference period?

Field Test Goals & Design Basic Goal: Can an EHC interview collect data of comparable quality to standard SIPP? - month-level data - one 12 -month ref pd interview vs. three 4 -month ref pd interviews - especially for need-based programs Basic Design: EHC re-interview of SIPP sample HHs

Stakeholder Concern About Test Design 1. Is the agreement shown in our primary re-interview results artificially inflated because of the initial interview experience – ‘priming’? In response - two groups - Cases having answered SIPP data in the 2004 Panel about CY 2007 (Primed) - 2004 Panel cases cut as part of a sample reduction after Wave 8, only EHC data for CY 2007 (Unprimed) 2. Field administration and training concerns - Field observation, Training, Interviewer and Respondent Debriefings. 3. Benchmarking to SIPP - An additional group will be created by the universe which matches to administrative records. As records are obtained for the federal (SSI, Medicare, Social Security Retirement) and state programs (TANF, Medicaid, Food Stamps, and WIC) we will continue with validation of the EHC and SIPP data for these programs.

Design Details (1) Main Sample: SIPP 2004 Panel Cases from Waves 10 -11 -12 - reported on CY-2007 via SIPP “Primed” Supplemental Sample: SIPP Wave 8 Sample Cut Cases - dropped from SIPP in 2006; “Unprimed” EHC Re-Interview in 2008, about CY-2007

Design Details (2) EHC Questionnaire - paper-and-pencil - 12 -month, CY-2007 reference period - selected SIPP topics (“domains”) - start with landmark events - within domains, anchor on “now” - month-level (at least) detail Field Period: Mid-April thru Late June 2008 Sample of Addresses, Not People - post-interview clerical match to SIPP

Control Card – 2008 Field Test Instrument

Page 1 of 5 – 2008 Field Test Instrument

Design Details (3) $40 Incentive, Non-Contingent and Non-FR-Discretionary Same Response Rules as SIPP - EHC interview for all adults (15+) - self-response preferred (proxy permitted) Field Staff: Census Bureau FRs - most with some interview experience - ~1/3 with SIPP experience - 3 -day training on EHC methods

Evaluation Plans Compare SIPP and EHC Survey Reports - Same People - Same Time Period - Same Characteristics Evaluation of “Priming” Effect Additional Evaluations: - Administrative Records Evaluation - Training Evaluation - Field Process Evaluation

Design Details (4) TOTAL Initial Sample Addresses Illinois Texas Total 2007 SIPP Respondents: SIPP Sample-Cut Cases: 1, 945 Outcomes: Household interviews 2007 SIPP Respondents: SIPP Sample-Cut Cases: 487 427 609 422 1, 096 849 935 692 Total: (91%) 1, 627 (91%) Individual Adult (15+) EHC interviews 2007 SIPP Respondents: 1, 922 SIPP Sample-Cut Cases: 1, 396 Total: (99%) 3, 318 (99%) EHC Adults matched to SIPP 2007 SIPP Respondents: SIPP Sample-Cut Cases: Total: (86%) (78%) 2, 756 (83%) FINAL ANALYSIS SAMPLE: 2007 SIPP Respondents: Non-movers 1, 420 Movers 200 SIPP Sample-Cut Cases: 1, 658 1, 098 Total 1, 620 Total 1, 090

What were the overall research questions? • Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? • Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?

What were the results? Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97 -98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92 -94% Disagreements are RARE EVENTS Results available: http: //www. census. gov/sipp/DEWS/moore-aapor 2009 slides-abbrev. ppt

Results Summary for Reinterview Evaluation 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); employment; school enrollment

The Movers The ‘Main Sample (Wave 11)’ of EHC respondents who reported for CY-2007 has an important difference from our ‘Supplemental Sample (Sample Cut). ’

The Movers The two subsamples of EHC respondents show different characteristics, and one primary cause are the Wave 11 respondents who moved between June 2006 and mid-2007.

The Movers

Assessment of Priming (1) SIPP Wave 10 -11 -12 Respondents Provide CY-2007 Data Twice - first SIPP, then EHC - results presented previously Are Their EHC Reports Biased by having responded to SIPP for CY-2007 already? - e. g. , more accurate EHC response - could bias field test interpretation Control Group: Wave-8 Sample Cut - last SIPP response in Jun-Sep 2006 - “Unprimed” re: CY-2007 (however still SIPP experienced)

“Priming”- SSI (Supplemental Security Income)

“Priming”- WIC (Women, Infants, and Children)

“Priming”- Work

“Priming”- Food Stamps

“Priming”- School Enrollment

Priming - Findings No evidence of priming with respect to having reported about CY-2007 in SIPP. - For each characteristic - Non-Movers from W 11 and Sample-Cut cases are not different - Interaction b/w these groups and months are not different - Results hold for both weighted and un-weighted models

Results Implications Levels of agreement from the reinterview comparison between SIPP and the EHC - are not artificially inflated due to experience as reinterview respondents re: CY-2007 - may still be biased because both the main reinterview sample and sample-cut respondents were long-time SIPP respondents.

Field Test Overall Summary Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform Next Test - larger, broader sample - “correct” timing of field period - automated questionnaire Specific Data Comparisons are Instructive

Comments: Jason Fields – Jason. M. Fields@Census. Gov Jeff Moore – Jeffrey. C. Moore@Census. Gov

Assessing Users’ Needs URL: http: //www. sipp. census. gov/sipp/dews

Page 2 – 2008 Field Test Instrument

Page 3 – 2008 Field Test Instrument

Page 4 – 2008 Field Test Instrument

Page 5 – 2008 Field Test Instrument