Скачать презентацию Understanding Society — Innovations in Panel Data Collection
f320908d253643378edf71a970441e36.ppt
- Количество слайдов: 124
Understanding Society - Innovations in Panel Data Collection and Analysis Michaela Benzeval, Peter Lynn, Tarek Al Baghal ISER, University of Essex 7 th ESRC Research Methods Festival 5 th July 2016 An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by Nat. Cen Social Research and TNS BMRB
What is Understanding Society? • Understanding Society: the UK Household Longitudinal Study : longitudinal household panel survey of all ages, designed to track all residents in households and analyse change at individual and household level by examining • what people are experiencing: health, circumstances and key events in their lives as they happen • attitudes, identity and subjective well-being • how individuals and households respond to policies and key events • linked administrative record • Incorporates British Household Panel Survey (started 1991) • Innovation Panel: sample for methodological research and experiments • Funded by ESRC, and number Gov Dept, publicly available data • Part of family UK longitudinal and worldwide panel studies Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Innovation in Understanding Society • Overall design • Innovation Panel: Testbed for experimentation - internal - external • Incorporating biomarkers and genetic information • Data linkage, new technologies and new data Today • introducing mixed mode data collection in a panel survey • improving measurement in a panel context • targeting the design of panel surveys Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Design (1) Design feature Main IP Sample size 32, 000 hh 1, 500 hh Sample design Stratified, clustered, equal probability sample of residential addresses in England, Scotland Wales, drawn from small users Postal Address File, random sample NI No NI sample BHPS At w 2 incorporated BHPS sample (8, 000 hh) x Following rules for W 1 hh - original sample members (OSM) All people who join OSM hh – temporary sample members Children born to OSM mothers (not fathers) only are OSMs Fathers of OSM babies become permanent sample members (PSM) if living with the baby at the interview after the birth OSMs and PSMs followed if they move within the UK, including into institutions Ethnic Minority Boost Sample(s) Main 5 groups at w 1, new Immigrant & ethnic minority boost w 6 X Refreshment samples X 500 hh at w 4, w 7, w 10 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Design (2) Design feature Main IP Annual interview - Fieldwork Each wave lasts 2 years, randomised allocation to 24 monthly samples Fieldwork May. September Questionnaire (multi-topic) - household grid & questions (10 mins) Annual and rotating content – wide range topics Annual content plus experiments - individual CAPI & CASI (16+)(39. 5 mins) - Youth questions (10 -15) (10 mins) https: //www. understandingsociety. ac. uk/documentat ion/mainstage/long-term-content-plan Geographic linkage Data linkage Range of consents asked Until IP 8 only asked for experiments Biomarkers and genetic data collection Wave 2 /3 nurse led interview X Mode of interview CAPI; CATI mop up from w 3 CAPI & CATI at Web/CAPI/CATI mixed w 2, web & CAPI mode from wave 7 from wave 5 plus CATI from w 6 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
How can researchers get involved in the IP? • Annual “IP Competition” for content of the questionnaire: o Small-scale experiments/methodological evaluations o Use longitudinal aspect of the survey o Data collection free for successful proposers • “Associated Studies” o Data collection separate from IP interview, sub-sample o Using mixed methods, e. g. qualitative interviews o Data collection funded and undertaken by proposer Ø https: //www. understandingsociety. ac. uk/about/innovation-panel Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Accessing the data and findings Study documentation: • Protocols, fieldwork reports, questionnaires, variable level metadata • User guides, Quality Profile, technical reports • Working Papers and publications from methodological research See www. understandingsociety. ac. uk Data are available at the UK Data Service, University of Essex: • Wave 1 - 5 data, including biomarkers; wave 6 and immigrant & ethnic minority boost sample to be deposited November 2016 • Innovation Panel data waves 1 -7 (wave 8 to be deposited July 2016) • Linked education data, geographical identifiers available by special licence or via secure lab See UK Data Service website: http: //ukdataservice. ac. uk/ Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Introducing mixed mode data collection in a panel survey Peter Lynn ISER, University of Essex An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by Nat. Cen Social Research and TNS BMRB
Survey Modes • Face-to-face interviewing (“CAPI”) • Telephone interviewing (“CATI”) • Self-completion on paper • Self-completion online (“web survey”) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Survey Tasks • Contacting the sample member • Persuading the sample member to take part • Sample member answers the questions (“data collection”) Contacting and data collection need not necessarily use the same mode However, it is efficient (in terms of costs and logistics) if they do Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Relative Merits of Modes • Probability/ease of contact: Mail, Email > CATI > CAPI • Probability/ease of co-operation: CAPI > CATI > Paper SC > Web • Quality of answers: Completeness/detail: CAPI > CATI > Paper/Web Honesty: Paper / Web > CATI > CAPI Effects of visual vs. aural presentation Effects of interviewer presence • Costs: Web > Paper > CATI > CAPI • Constraints: sampling frames Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Understanding Society Context I Solely CAPI initially: • To get best possible initial response rates; • To get respondent commitment; • To collect complex and sensitive data But with intention to move towards including web: • To reduce data collection costs • To reflect respondent preferences/norms Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Context II ESRC ITT for waves 6 – 8: “There is a critical expectation that mixed modes of data collection will be introduced to the main study by wave 7. The proposal will include a SWOT analysis of the proposed new modes, plans for transition (including with fieldwork collection agencies) and identify potential cost savings in the short, medium and long term. ” Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Context III ISER Response: “proposed design uses modes sequentially…. sample members are initially invited to complete the survey by web; non-respondents to the web survey are issued to face-to-face interviewing; in a final stage non-contacted or refusal cases are issued to telephone interviews. The conditions for implementing the mixed modes strategy on the main survey are 1) that the approach used does not significantly raise overall attrition compared with using face-to-face interviewing only; 2) that the approach used leads to significant cost savings at least in medium term, and 3) that it maintains data quality comparable with face-to-face interviewing. ” Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Mixed Mode Experiments • UKHLS Innovation Panel has carried several experiments with mixed-mode data collection: • Wave 2 (2009): (CATI+CAPI) vs. CAPI • Wave 5 (2012): (Web+CAPI) vs. CAPI • Waves 6 -8 (2013 -15): (Web+CAPI+CATI) vs. (CAPI+Web) • (Waves 1, 3, 4 solely face-to-face) • The experiments were multiple purpose. Prime purposes were to study effects on non-response, attrition and costs. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 2 Experiment: CATI • Wave 2 sample randomly allocated to 3 equal-size groups (n ≈ 900 per group): • Face-to-face only (control group); • “Early transfer” mixed mode (treatment 1): CATI, but switch to CAPI as soon as it is clear that an interviewer visit to the household will be necessary; • “Late transfer” mixed mode (treatment 2): CATI, and try to interview as many household members by phone before switching to CAPI: Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
3 Key Research Questions What are the relative effects of the three mode protocols on: 1)Response rates and attrition? 2)Sample composition? 3)Survey costs? Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Some findings from…. I Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Mixed Modes and Attrition Response Rate Wave 2 Face-to-face 75. 8% Mixed Modes 67. 4% Difference Odds Ratio (P) -8. 4 0. 67 (0. 001) Base = respondents at wave 1 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Mixed Modes and Attrition Response Rate Wave 2 Wave 3 Face-to-face 75. 8% 66. 3% Mixed Modes 67. 4% 59. 9% -8. 4 -6. 4 0. 67 (0. 001) 0. 79 (0. 07) Difference Odds Ratio (P) Base = respondents at wave 1 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Mixed Modes and Attrition Response Rate Wave 2 Wave 3 Wave 4 Face-to-face 75. 8% 66. 3% 57. 3% Mixed Modes 67. 4% 59. 9% 53. 7% -8. 4 -6. 4 -3. 6 0. 67 (0. 001) 0. 79 (0. 07) 0. 88 (0. 29) Difference Odds Ratio (P) Base = respondents at wave 1 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Early Transfer and Attrition Response Rate Wave 2 Wave 3 Wave 4 Face-to-face 75. 8% 66. 3% 57. 3% Early trans MM 67. 6% 58. 1% 49. 9% -8. 2 -7. 4 0. 68 (0. 01) 0. 73 (0. 03) 0. 75 (0. 03) Difference Odds Ratio (P) Base = respondents at wave 1 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Late Transfer and Attrition Response Rate Wave 2 Wave 3 Wave 4 Face-to-face 75. 8% 66. 3% 57. 3% Late trans MM 67. 2% 61. 7% 57. 5% -8. 6 -4. 6 +0. 2 0. 66 (0. 002) 0. 86 (0. 27) 1. 02 (0. 90) Difference Odds Ratio (P) Base = respondents at wave 1 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Response Rate 80 75 70 65 F-to-F MM-Early 60 MM-Late 55 50 45 wave 2 wave 3 wave 4 • “Recovery” entirely restricted to “Late transfer” treatment Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Sample Composition Logistic regressions: • Around 10% of tested effects ‘significant’ at 0. 05 level • Little more than would be expected by chance • Not systematic R-indicators: R-Indicator Face-to-face Tele-light Tele-intens Fitted model (7 vbles) 0. 73 0. 67 Full model (19 vbles) 0. 66 0. 68 0. 65 Ø Differences between protocols in non-response bias appear minimal Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Survey Costs Wave 2: In MM sample, 52% of households required an interviewer visit Ø Significant cost saving compared to CAPI Ø The saving did not differ between the two MM protocols Waves 3, 4: Interviewer visits per responding household Face-to-face Tele-light Tele-intens Wave 3 6. 21 7. 36 6. 28 Wave 4 6. 02 7. 45 6. 14 Ø No continuing cost savings if revert to CAPI Lynn | Wollongong, 17 -10 -2013
CATI Conclusions I 1. Sequential MM (Tele>CAPI) damaged response rates, relative to CAPI-only • For some sample members, a telephone approach reduces propensity to co-operate with a subsequent face-to-face request • The mechanisms by which this happens are unclear 2. When subsequent waves are CAPI-only, the RR difference between groups erodes • Occasional implementation of a MM wave on a CAPI-panel may not do long-term damage to cumulative RR Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
CATI Conclusions II 3. The erosion of the RR differential occurred only with the telephone-intensive protocol • Specifically, those who responded at w 2 with tele-light were more likely to drop out at w 3 or w 4 than those who responded with tele-intensive • The effect was restricted to households with 2+ adults • Perhaps caused via intra-hhd communication and the salience of a face-to-face interview. Perceived inequity? Acceptability? Perceived lack of persistence? Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
CATI Conclusions III 4. Few differences in non-response bias between groups • Differential effects of MM may operate broadly across subgroups 5. Considerable cost savings are possible with MM, compared to face-to-face • But these do not continue at subsequent waves if mode reverts to face-to-face • And savings decline over waves even if mode remains CATI+CAPI, as proportion of interviewer time spent travelling and contacting reduces Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 5 Experiment: Web Two components: • Original sample, for whom this was the 5 th wave; • Refreshment sample, for whom this was the 2 nd wave. Households randomly assigned to one of two treatment groups: • CAPI (one-third of each sample); • Mixed mode (two-thirds of each sample). Mixed modes treatment: • Letter with URL and ID. Sent also by email where available; • Hhd grid (first respondent only), hhd qre (first householder), individual qre • 3 email reminders at 3 -day intervals; After 14 days, reminder letter + face-to-face visits began Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
3 Key Research Questions What are the relative effects of the three mode protocols on: 1)Survey costs? 2)Unit response rates? 3)Item response rates? Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Some findings from…. I Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Costs: Households Responding Completely by Web Proportion of households (n) Original sample Total sample 21. 5 Wave 4 respondent households 24. 7 Wave 4 nonrespondent households Refreshment sample Total sample 2. 3 23. 0 Analysis restricted to £ 10 incentive group ØOne-fifth to one-quarter of households do not require an interviewer visit: considerable cost savings if sample is large Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Households Responding Completely by Web are more likely to be: • With broadband access; • All adults in household use internet; • Not more than 2 adults in household; • In urban areas Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Household Response Rates Original sample IP 4 responding IP 4 non-responding F 2 F MM P Refreshment sample F 2 F MM P HH response rate 84. 1 81. 1 0. 29 32. 6 40. 4 0. 38 85. 1 82. 2 0. 49 Complete HHs 63. 9 55. 7 0. 02 14. 0 22. 2 0. 30 60. 1 66. 2 0. 25 Partial HHs 20. 3 25. 4 0. 07 18. 6 18. 2 0. 95 25. 0 15. 9 0. 01 Non-contact 5. 6 2. 9 0. 04 16. 3 13. 1 0. 63 5. 4 6. 4 0. 71 Refusal 8. 7 14. 1 0. 05 46. 5 40. 48 8. 3 9. 6 0. 68 N 322 621 43 99 168 315 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Household Response Rates Original sample IP 4 responding IP 4 non-responding F 2 F MM P Refreshment sample F 2 F MM P HH response rate 84. 1 81. 1 0. 29 32. 6 40. 4 0. 38 85. 1 82. 2 0. 49 Complete HHs 63. 9 55. 7 0. 02 14. 0 22. 2 0. 30 60. 1 66. 2 0. 25 Partial HHs 20. 3 25. 4 0. 07 18. 6 18. 2 0. 95 25. 0 15. 9 0. 01 Non-contact 5. 6 2. 9 0. 04 16. 3 13. 1 0. 63 5. 4 6. 4 0. 71 Refusal 8. 7 14. 1 0. 05 46. 5 40. 48 8. 3 9. 6 0. 68 N 322 621 43 99 168 315 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Household Response Rates Original sample IP 4 responding IP 4 non-responding F 2 F MM P Refreshment sample F 2 F MM P HH response rate 84. 1 81. 1 0. 29 32. 6 40. 4 0. 38 85. 1 82. 2 0. 49 Complete HHs 63. 9 55. 7 0. 02 14. 0 22. 2 0. 30 60. 1 66. 2 0. 25 Partial HHs 20. 3 25. 4 0. 07 18. 6 18. 2 0. 95 25. 0 15. 9 0. 01 Non-contact 5. 6 2. 9 0. 04 16. 3 13. 1 0. 63 5. 4 6. 4 0. 71 Refusal 8. 7 14. 1 0. 05 46. 5 40. 48 8. 3 9. 6 0. 68 N 322 621 43 99 168 315 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Household Response Rates Original sample IP 4 responding IP 4 non-responding F 2 F MM P Refreshment sample F 2 F MM P HH response rate 84. 1 81. 1 0. 29 32. 6 40. 4 0. 38 85. 1 82. 2 0. 49 Complete HHs 63. 9 55. 7 0. 02 14. 0 22. 2 0. 30 60. 1 66. 2 0. 25 Partial HHs 20. 3 25. 4 0. 07 18. 6 18. 2 0. 95 25. 0 15. 9 0. 01 Non-contact 5. 6 2. 9 0. 04 16. 3 13. 1 0. 63 5. 4 6. 4 0. 71 Refusal 8. 7 14. 1 0. 05 46. 5 40. 48 8. 3 9. 6 0. 68 N 322 621 43 99 168 315 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Individual Response Rates Original sample IP 4 responding IP 4 non-responding F 2 F MM P Full interview Refreshment sample F 2 F MM P 82. 8 75. 8 0. 04 18. 5 24. 8 0. 11 69. 8 72. 0 0. 60 Proxy interview 2. 6 2. 0 0. 50 20. 2 9. 6 0. 00 5. 5 1. 9 0. 00 Partial interview 0. 0 1. 2 0. 01 0. 0 0. 5 0. 34 0. 0 1. 4 0. 03 Non-contact 3. 4 2. 9 0. 65 18. 0 10. 6 0. 05 5. 7 6. 8 0. 68 Refusal 8. 5 14. 1 0. 08 41. 0 43. 0 0. 66 12. 9 11. 1 0. 60 Other NR 2. 6 4. 0 0. 34 2. 2 11. 4 0. 00 6. 9 0. 62 N 495 934 178 395 348 592 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Individual Response Rates Original sample IP 4 responding IP 4 non-responding F 2 F MM P Full interview Refreshment sample F 2 F MM P 82. 8 75. 8 0. 04 18. 5 24. 8 0. 11 69. 8 72. 0 0. 60 Proxy interview 2. 6 2. 0 0. 50 20. 2 9. 6 0. 00 5. 5 1. 9 0. 00 Partial interview 0. 0 1. 2 0. 01 0. 0 0. 5 0. 34 0. 0 1. 4 0. 03 Non-contact 3. 4 2. 9 0. 65 18. 0 10. 6 0. 05 5. 7 6. 8 0. 68 Refusal 8. 5 14. 1 0. 08 41. 0 43. 0 0. 66 12. 9 11. 1 0. 60 Other NR 2. 6 4. 0 0. 34 2. 2 11. 4 0. 00 6. 9 0. 62 N 495 934 178 395 348 592 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Effects of Incentives Refreshment sample £ 10 incentive F 2 F MM P £ 20 incentive F 2 F MM P £ 30 incentive F 2 F MM P HH response rate 87. 5 74. 7 0. 11 87. 0 81. 9 0. 45 90. 3 92. 8 0. 59 Complete HHs 57. 5 52. 9 0. 66 61. 1 68. 6 0. 40 64. 5 78. 4 0. 08 Partial HHs 30. 0 21. 8 0. 39 25. 9 13. 3 0. 08 25. 8 14. 4 0. 10 Non-contact 0. 0 5. 7 0. 11 3. 8 0. 97 3. 2 4. 5 0. 68 12. 5 18. 4 0. 38 9. 3 12. 4 0. 59 3. 2 0. 9 0. 27 0. 0 1. 1 0. 50 0. 31 3. 2 1. 8 0. 55 Refusal Other NR N 40 87 3. 7 54 1. 9 105 62 111 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Effects of Incentives Refreshment sample £ 10 incentive F 2 F MM P £ 20 incentive F 2 F MM P £ 30 incentive F 2 F MM P HH response rate 87. 5 74. 7 0. 11 87. 0 81. 9 0. 45 90. 3 92. 8 0. 59 Complete HHs 57. 5 52. 9 0. 66 61. 1 68. 6 0. 40 64. 5 78. 4 0. 08 Partial HHs 30. 0 21. 8 0. 39 25. 9 13. 3 0. 08 25. 8 14. 4 0. 10 Non-contact 0. 0 5. 7 0. 11 3. 8 0. 97 3. 2 4. 5 0. 68 12. 5 18. 4 0. 38 9. 3 12. 4 0. 59 3. 2 0. 9 0. 27 0. 0 1. 1 0. 50 0. 31 3. 2 1. 8 0. 55 Refusal Other NR N 40 87 3. 7 54 1. 9 105 62 111 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Mixed Modes over Multiple Waves Response rates (based on all issued at wave 5). Odds ratios: Original sample: w 4 resps Original sample: w 4 non-resps Refreshment sample wave 5 0. 651* 1. 453 1. 113 wave 6 0. 911 1. 791* 1. 179 wave 7 1. 156 1. 678* 1. 257 wave 8 1. 510** 2. 315** 1. 393 Source: Wave 5: Calculations based on Jackle, A. et al (2015) Survey Research Methods; Waves 6 -8: From Gaia, A. (2016) ‘The effect of a switch to a mixed-mode design on panel attrition: Evidence from an experiment of the Understanding Society Innovation Panel, ’ paper presented at International Panel Survey Methods Workshop, Berlin, June Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Mixed Modes over Multiple Waves Percentage of completed interviews completed by web: Household interview Individual interview wave 5 31. 3 30. 0 wave 6 36. 8 38. 3 wave 7 42. 8 43. 6 wave 8 44. 8 45. 0 Analysis restricted to original and IP 4 refreshment samples, as IP 7 refreshment sample was treated web-first for the first time at IP 9. Source: Own calculations Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Item Non-Response Original sample Refreshment sample F 2 F Mean Item non-response (%) N Gross pay Item non-response (%) N MM P F 2 F MM P 0. 13 0. 19 0. 014 0. 17 0. 137 459 10. 0 229 856 17. 5 243 0. 022 411 8. 0 112 435 18. 0 0. 027 205 P: P-values from Wald tests of means adjusted for sample design Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 6 Experiment: Reducing Item Non -Response Three treatments. Variation is in what appears on screen following an attempt to skip a question: • Control: question repeated with DK/PNS options now included; • Reactive motivational: question repeated, with motivational statement: “If possible, please provide an answer to this question, as this is one of the key questions in this study. Please be assured that the information you give us will be treated confidentially. ” • Follow-up motivational: As control, but if any DK/PNS answers, these questions repeated at end of interview: “Thank you for taking part in “Understanding Society” this year. Earlier in the interview there were some questions that you did not answer. As you know, you do not have to answer any question you do not want to. However, X of these questions are important to researchers and we would be grateful if you could try your best to answer them. ” Treatments applied only to 6 questions with high item NR risk Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Some findings from…. I Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Measurement Issues • We commissioned Nat. Cen to code all questions proposed for waves 7 – 10 with respect to 13 characteristics associated with increased risk of measurement differences between modes • Summary measure of ‘risk of mode effect’ derived for all 1, 486 questions • Two kinds of empirical analysis of modes effects in IP data: - ‘Intention to treat’ analysis – essentially comparing mixed mode with CAPI; - Instrumental variable selection models to identify the ‘effect of the treatment on the treated’ – i. e. mode effects amongst those who respond by web Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Measurement Issues II • Many of the more sensitive questions (health, attitudes, satisfaction) are already asked by self-completion within CAPI; mode effects likely to be negligible for these • Other questions most at risk of mode effects are: - Questions about harrassment, prejudice etc, asked as part of the ethnic minority research strand; - Questions that benefit from interviewer encouragement/ support to answer fully and accurately, including income. • Current research programme to identify best analysis methods for mixed mode longitudinal data and to provide guidance to users • Includes a Ph. D studentship, starting October 2016 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Web Conclusions • Initial response rate lower when switching to web-first, but recovers after a couple of waves • Previous wave non-respondents are more likely to respond with web-first design than with CAPI-only • Cost savings should soon accrue as around 25% of responding households respond fully by web • Percent responding by web appears to increase over waves • May be possible to avoid potential increases in item response rates • Measurement error issues remain a concern; our focus is to minimise impact and provide analysis guidance Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Design of Main Survey • CATI was introduced from wave 3, only as a last resort for very difficult to contact sample members (1. 5% of individual interviews at w 3, 2. 3% at w 4, 2. 4% at w 5) • Web-first protocol introduced for first time at wave 7 – only for wave 6 non-responding households (8% of sample) • Web-first introduced on large-scale from w 8 (40% of households). Proportion will increase at w 9 (50%) and w 10 (60%) • Web-first is targeted at households in which: - No person is estimated to have a greatly lower response propensity with the mixed mode design (for w 8, ΔP < 0. 14); and - The probability of the household fully responding by web is not too low (P > 0. 02) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by Nat. Cen Social Research and TNS BMRB
Improving Measurement in Panel Surveys Tarek Al Baghal ISER, University of Essex An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by Nat. Cen Social Research and TNS BMRB
The Measurement of Change • Panel (longitudinal) surveys allow a unique opportunity to measure change in status for myriad questions • Unique opportunity, unique challenges • Errors tend toward spurious change, although underreporting of change possible • Possibly recall, but also measurement, data processing, and interviewer errors Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Example Measurement Experiments in the IP • Consumption/Expenditure: o Varied how households were asked about how much they spend: using a single question, including cues or a showcard listing categories to include, or asking separate questions about each expenditure category • Branched and unbranched ratings: o Half the sample were asked attitude questions using a 5 point agree/disagree scale o The other half were first asked about the direction of their attitude (agree/disagree) and then the strength Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Example Measurement Experiments in the IP (2) • Measuring household energy use: o Respondents were asked to report meter readings for gas, electricity and care mileage o Varied what reading households were asked for and tested whether a mention in the advance letter would encourage provision of the readings • Item non-response: o Tested techniques to reduce item non-response in the web survey, by reminding the respondent of the importance of their answer after the question, or at the end of the interview. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Example Measurement Experiments in the IP (3) • Dependent Interviewing Wording: o Conducted at several waves, respondents were asked questions using differing Proactive Dependent Interviewing wordings • And many more measurement experiments o See the IP User Guide o https: //www. understandingsociety. ac. uk/ Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 6 Expenditure Experiment • Collecting expenditure in a longitudinal survey not frequent • But potentially useful • Most budget surveys have many burdensome questions • IP 6 experiment is focused on collecting total household expenditure • Either asked using one-question or a breakdown of several expenditures, then summed. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 6 Expenditure Experiment • The single question varied to ask about either last or usual month • Breakdown was of 12 categories of spending (e. g. food) plus an other category. • Also divided across web and face-to-face interviews • Mean Web-Breakdown: 1807 • Mean F 2 F-Breakdown: 1801 • Both close to expectation Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 6 Expenditure Experiment • Mean Web-Single/Usual: 1646 • Mean Web-Single/Last: 2206 • Mean F 2 F-Single/Usual: 1219 • Mean F 2 F-Single/Last: 1312 • Appears underreporting in face-to-face survey when asking just the single question • Perhaps use the breakdown question Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP 6 Expenditure Experiment • Follow-up question asked how respondent came up with answer to single question • In face-to-face 67% said they added up categories, compared to 35% in web • But 28% of web respondents said they checked records, compared to only 10% • Less time pressure on the web possibly Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
IP Dependent Interviewing • May improve change measurement in panels using dependent interviewing (DI) • Responses from previous wave used in the repeated question in current wave • Two main types of DI: • Reactive Dependent Interviewing (RDI) • Proactive Dependent Interviewing (PDI) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Dependent Interviewing • RDI useful in ensuring panel consistency, reduction of error for smaller number of items • PDI focuses on improvement and efficiency of the initial recall attempt • PDI has been the focus of research in the IP Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Proactive Dependent Interviewing • PDI uses past response in initial asking in question stem • “The last time we interviewed you on July 5, 2015 you said that you were selfemployed. Is that still the case? ” • Shifts task mainly to recognition v. recall Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Proactive Dependent Interviewing • Increases potential for satisficing, i. e. choosing default/explicit option • PDI wording and design features not well studied • Besides wording, mode, attitude v. factual, and whether question prone to change Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
PDI in the Innovation Panel • Wave 3, 3 PDI experimental questions. • Wave 5, an additional 6 questions • Wave 7, a total of 18 PDI questions • Analysis here covers PDI in Waves 3 and 5 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
PDI Methods • 2 questions attitudinal, 7 autobiographical • At Wave 5, responses either web or F 2 F • t-1 change between independent asking identified • Analysis using the first asking of PDI question • All responses pooled, stacked within respondents Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
PDI Wordings • Unbalanced, Still (Waves 3 and 5) The last time we interviewed you on [DATE] , you said that … was [PREVIOUS RESPONSE]. Is that still the case? • Unbalanced, Change (Waves 3 and 5) The last time we interviewed you on [DATE] , you said that … was [PREVIOUS RESPONSE]. Has that changed? • Balanced, Still/Change (Wave 5) The last time we interviewed you on [DATE] , you said that … was [PREVIOUS RESPONSE]. Is that still the case or has that changed? • Balanced, Change/Still (Wave 5) The last time we interviewed you on [DATE] , you said that … was [PREVIOUS RESPONSE]. Has that changed or is that still the case? Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
PDI Methods • If respondent indicated a change to any of the PDI version, was asked follow-up as to the new value • Could still provide the same, previous value • Initial change indicated by response to PDI • Final change indicated by if values differed across waves Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
PDI Results • 7, 174 PDI Responses, 1, 624 Respondents • 22. 7% change between 2 previous waves (independent) • 11. 4% change across all PDI versions, diff. p<. 0001 • 7. 7% indicate a change using final report Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Change in change • 266 initial change response, no change final response • 235 respondents, median = 1, mean = 1. 13 • Largely located in F 2 F, unbalanced, change versions • Interviewer ICC = 0. 071 Respondent ICC = 0. 160 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Total Initial and Final Change 20% 17. 5% 10% 8. 6% 8. 5% 7. 0% 9. 7% 8. 0% 7. 3% 8. 2% Balanced, Still/Change Initial Balanced, Change/Still/Change Final 5% 0% Unbalanced, Still Unbalanced, Change Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Initial Change By Mode 25% 20. 3% 20% 15% 10% 9. 4% 9. 2% 6. 4% 6. 8% F 2 F 10. 2% 8. 8% Web 5. 3% 5% 0% Unbalanced, Still Unbalanced, Change Balanced, Still/Change Balanced, Change/Still Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Initial Change By Content Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Initial Change By Prior Change Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Multilevel Analysis of Change • Two models, for 1) Initial and 2) Final Change Report • Question characteristics and interaction with wording • Demographics, controls for web selection • Age, education, int. observation of R understanding – proxies for cognitive ability • Life changes – sum(moves, change relationship, had child, new educ. qualifications, change in disability) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Model Results (1) Initial Change (2)Final Change Response Characteristics PDI Wording (Unbalanced, Still) Unbalanced, Change 1. 13* Balanced, Still/Change Balanced, Change/Still Web Response Attitude Question 0. 07 0. 17 -0. 17 -0. 14 0. 39 0. 29 0. 37 -0. 08 0. 05 Lagged Change 0. 90* 0. 94* -1. 04* -0. 97* -0. 53* -0. 01 0. 60* 0. 29 -0. 01* 0. 23* -0. 01* 0. 29* 0. 076 0. 075 Unbalanced, Change * Web Unbalanced, Change * Prior Change Balanced, Change/Still * Attitude Respondent Characteristics Age Life Changes ICC
Model Estimates, by Mode Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Model Estimates, by Prior Change Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Discussion • DI is a useful tool to reduce error in panel settings • Wording matters: avoid “Change” options (first) • Still much to learn, e. g. : • How to best measure HH finance • Gaining consent, linking, and using admin records as new data • The increasing use of mixed-mode also potentially complicates these issues. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Targeting the Design of Panel Surveys Peter Lynn, ISER, University of Essex An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by Nat. Cen Social Research and TNS BMRB
Standardised Survey Design • Standardisation is central concept to quantitative measurement • All aspects of measurement should be consistent (identical) across all study subjects: - Contextual environment - Question wording - Question presentation - Response options - Prompts and probes - Interviewer neutrality etc. • Relates to concepts of reliability, consistency, accuracy Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Standardised Survey Design • But standardisation is typically extended to all other parts of the survey process by default, e. g. : - Respondent communications - Respondent incentives - Contacting procedures - Call schedule algorithms - Field period - Persuasion methods - Data management and editing etc. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
• Particularly with regard to survey procedures intended to secure respondent participation, it is less clear (than with measurement) that: - a standardised stimulus should produce a standardised reaction; - a standardised reaction is desirable • We do not aim to measure the participation propensities of respondents; we aim to maximise them Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Securing Participation • Large body of research into effects of various design features on participation propensity • Outcomes are typically contact rates, co-operation rates, response rates • Studied features include: - form, value and conditionality of incentives - content and design of advance letters - postage class and envelope design (for mail surveys) - ostensible survey sponsorship etc. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
• Common feature of most of these studies is that effects are found to be heterogeneous across sample subgroups: - effect of incentives depends on income - effect of persuasion messages depends on level of education - effect of interviewer gender depends on respondent gender etc. • Standard stimulus does not produce standard response, when it comes to participation propensity • And in any case, we want a maximised response • Implication is that targeted stimuli may be more effective Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Targeted Design Features • A design feature is varied between subgroups of sample members • It is possible to vary more than one feature simultaneously • Aim is to beneficially affect the relationship between survey costs and survey errors • Variation(s) in design feature(s) are identified and planned in advance of data collection • No adaptations are made during field work (static design) • Requires advance knowledge of relevant subgroup membership (informative sampling frame; longitudinal surveys) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Aims As implemented on surveys to date, aim is usually to achieve one of the following: • Reduce one or more components of nonresponse error; • Reduce one or more components of measurement error; • Reduce survey costs Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Implementation • Identify subgroups to treat differently • Identify treatments for each subgroup • Implement! • Criteria for subgroups: - Manageable number of groups - Each group should have defining characteristics that lend themselves to targeted treatment - Groups should vary in terms of the cost of treatment and/or contribution to error (e. g. response propensities) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Treatments When the aim relates to nonresponse error, treatments (design variations) can include: • Differential incentives, monetary or otherwise; • Differential field time (i. e. prioritising certain types of cases) or call scheduling; • Differential modes / mode combinations / mode prioritisation • Differential methods to encourage keeping in touch / notifying changes of address/phone/email; • Differential communications (advance letters, information brochures, between-wave mailings, etc). Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Example 1: Modes Wave 8 objective is to allocate 40% of gross sample to web-first sequential mixed modes design; remainder to CAPI-first. We first selected a random 20% of all sample households to remain CAPIfirst (control group). The rest are allocated to one of the two mode protocols in a targeted way, so that the “most promising” half of the households (40% of total) are web-first. Definition of “most promising” is derived from modelling the IP experimental data. Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Modelling of Outcomes Statistical models were developed, based on the IP 5 experiment, to predict: • The probability of a household completing all components online (P 1); • The probability of an individual completing the individual questionnaire with the CAPI-only design (P 2); • The probability of an individual completing the individual questionnaire with the web-first mixed-mode design (P 3). Then, household is allocated to web-first for wave 8 if: • Non-respondent at wave 7; or • P 1 > 0. 02 and P 2 -P 3 < 0. 14 for all household members (In fact, only a random subset of households allocated web-first by this algorithm are actually treated web-first in first months, to allow assessment of the efficacy of the procedure. ) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Example 2: Leaflets Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
British Household Panel Survey (now part of Understanding Society): A report of findings is mailed to sample members between waves in the hope of emphasising the saliency and interest of the survey and hence providing motivation to co-operate at the next wave. Between waves 17 and 18, two randomised treatments: • “Standard” report of findings to all sample members; • Targeted report: - Version 1 (“Young”) if aged < 25 - Version 2 (“Busy”) if self-employed, long work hours or long commute - Version 3 (“Standard”) otherwise Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Standard Report Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Targeted report (young) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Targeted report (employment busy) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Response Rates: Young People Targeted (%) Standard (%) Full face-to-face interview 93. 2 91. 6* Full face-to-face interview or shorter phone interview 94. 1 94. 2 No interview 5. 9 5. 8 n 843 856 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Response Rates: Busy People Targeted (%) Standard (%) Full face-to-face interview 90. 3 90. 1 Full face-to-face interview or shorter phone interview 97. 5 96. 5* No interview 2. 5 3. 5* 1205 1157 n Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Response Rates: Total Sample Targeted (%) Standard (%) Full face-to-face interview 91. 4 91. 1 Full face-to-face interview or shorter phone interview 96. 8 No interview 3. 2 5942 5857 n Note: only 25% of sample members (young or busy) received a targeted report Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Example 3: Invitation Letters Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Outline of the Experiment with initial letters sent to sample members on a wave of a panel survey • Invitation letters for the web-first sub-sample • Advance letters for the CAPI-first sub-sample Prime purpose of the letters is to motivate co-operation Experiment incorporates randomisation not only of the targeting treatment but also (orthogonally) of two other design features that are potential moderators of the effect(s) Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Study Design • Understanding Society Innovation Panel, wave 6 (2013); • N = 2, 733 adults (16+) issued to field for wave 6 • Nationally-representative probability sample (Great Britain) • Randomly allocated to two groups: - Standard letter (same for all sample members); - Targeted letter (wording varies between subgroups) • Orthogonal experimental treatments: - Mode: CAPI single-mode vs. Web-CAPI seq. mixed mode; - Time in sample: 6 th wave vs. 3 rd wave Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Targeted Subgroups Group Definition Frequency Percentage Employment-busy Employed for at least 39 hours/week, or employed 30 to 38 hours with a commute of least 60 minutes 425 15. 6 With children Responsible for at least one child under 15 in the same household at the time of most recent interview 339 12. 4 Young Aged 16 to 29 at the time of wave 5 323 11. 8 London Resident in London or south east England at the time of most recent interview 358 13. 1 Pensionable Of pensionable age at the time of wave 5 (60 or over for women; 65 or over for men) 464 17. 0 Remainder None of the above 824 30. 1 Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Variants of the Initial Letter First paragraph of the Thank you so much for helping with the Understanding Society survey last year. letter (for previous. The survey helps researchers and policy makers understand the changes in the wave respondents): needs of the country across diverse subjects like
Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Results: Response Rates Sample subgroup n Response Rate Standard Targeted Letter % % 72. 0 73. 8 P Full sample 2, 733 Previous wave respondents (RESP) 1, 979 87. 4 85. 9 0. 34 Previous wave non-respondents (NRESP) 754 32. 4 41. 4 0. 01** Time in sample: 6 waves (TIME 6) 1, 853 72. 5 71. 6 0. 64 Time in sample: 3 waves (TIME 3) 880 70. 9 78. 8 0. 007** Single-mode CAPI (CAPI) 946 71. 4 71. 1 0. 92 Mixed mode web-CAPI (MMODE) 1, 787 72. 3 75. 3 0. 16 NRESP * CAPI 248 27. 5 29. 9 0. 67 NRESP * MMODE 506 35. 0 46. 5 0. 008** TIME 3 * CAPI 325 64. 9 78. 8 0. 005** TIME 3 * MMODE 555 74. 4 78. 9 0. 21 0. 28 Notes: ** indicates P<0. 01, * indicates 0. 01
Results: Response Rates Sample subgroup n Response Rate Standard Targeted Letter % % 72. 0 73. 8 P Full sample 2, 733 Previous wave respondents (RESP) 1, 979 87. 4 85. 9 0. 34 Previous wave non-respondents (NRESP) 754 32. 4 41. 4 0. 01** Time in sample: 6 waves (TIME 6) 1, 853 72. 5 71. 6 0. 64 Time in sample: 3 waves (TIME 3) 880 70. 9 78. 8 0. 007** Single-mode CAPI (CAPI) 946 71. 4 71. 1 0. 92 Mixed mode web-CAPI (MMODE) 1, 787 72. 3 75. 3 0. 16 NRESP * CAPI 248 27. 5 29. 9 0. 67 NRESP * MMODE 506 35. 0 46. 5 0. 008** TIME 3 * CAPI 325 64. 9 78. 8 0. 005** TIME 3 * MMODE 555 74. 4 78. 9 0. 21 0. 28 Notes: ** indicates P<0. 01, * indicates 0. 01
Results: Response Rates Sample subgroup n Response Rate Standard Targeted Letter % % 72. 0 73. 8 P Full sample 2, 733 Previous wave respondents (RESP) 1, 979 87. 4 85. 9 0. 34 Previous wave non-respondents (NRESP) 754 32. 4 41. 4 0. 01** Time in sample: 6 waves (TIME 6) 1, 853 72. 5 71. 6 0. 64 Time in sample: 3 waves (TIME 3) 880 70. 9 78. 8 0. 007** Single-mode CAPI (CAPI) 946 71. 4 71. 1 0. 92 Mixed mode web-CAPI (MMODE) 1, 787 72. 3 75. 3 0. 16 NRESP * CAPI 248 27. 5 29. 9 0. 67 NRESP * MMODE 506 35. 0 46. 5 0. 008** TIME 3 * CAPI 325 64. 9 78. 8 0. 005** TIME 3 * MMODE 555 74. 4 78. 9 0. 21 0. 28 Notes: ** indicates P<0. 01, * indicates 0. 01
Results: Response Rates Sample subgroup n Response Rate Standard Targeted Letter % % 72. 0 73. 8 P Full sample 2, 733 Previous wave respondents (RESP) 1, 979 87. 4 85. 9 0. 34 Previous wave non-respondents (NRESP) 754 32. 4 41. 4 0. 01** Time in sample: 6 waves (TIME 6) 1, 853 72. 5 71. 6 0. 64 Time in sample: 3 waves (TIME 3) 880 70. 9 78. 8 0. 007** Single-mode CAPI (CAPI) 946 71. 4 71. 1 0. 92 Mixed mode web-CAPI (MMODE) 1, 787 72. 3 75. 3 0. 16 NRESP * CAPI 248 27. 5 29. 9 0. 67 NRESP * MMODE 506 35. 0 46. 5 0. 008** TIME 3 * CAPI 325 64. 9 78. 8 0. 005** TIME 3 * MMODE 555 74. 4 78. 9 0. 21 0. 28 Notes: ** indicates P<0. 01, * indicates 0. 01
Results: Response Rates Sample subgroup n Response Rate Standard Targeted Letter % % 72. 0 73. 8 P Full sample 2, 733 Previous wave respondents (RESP) 1, 979 87. 4 85. 9 0. 34 Previous wave non-respondents (NRESP) 754 32. 4 41. 4 0. 01** Time in sample: 6 waves (TIME 6) 1, 853 72. 5 71. 6 0. 64 Time in sample: 3 waves (TIME 3) 880 70. 9 78. 8 0. 007** Single-mode CAPI (CAPI) 946 71. 4 71. 1 0. 92 Mixed mode web-CAPI (MMODE) 1, 787 72. 3 75. 3 0. 16 NRESP * CAPI 248 27. 5 29. 9 0. 67 NRESP * MMODE 506 35. 0 46. 5 0. 008** TIME 3 * CAPI 325 64. 9 78. 8 0. 005** TIME 3 * MMODE 555 74. 4 78. 9 0. 21 0. 28 Notes: ** indicates P<0. 01, * indicates 0. 01
Conclusions re. Letters • Response Rates increased for - Previous-wave non-respondents (in mixed mode) - Recent panel entrants (in CAPI mode) • A targeted initial letter can increase response rates; • Effects are uneven across survey design contexts and sample subgroups; • Important difference: initial letter acts only as prenotification in CAPI mode, but as an invitation letter in mixed mode: - CAPI: no immediate action that sample member can take; - Mixed mode: can immediately go online and fill out the survey • Positive effects on response rate are only observed for lowpropensity subgroups, so sample composition may be improved Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Future Prospects for Targeted Design Features • Targeted procedures to become a routine part of survey production? (May require survey organisations to modify their systems) • Treatments to become more sophisticated? • Aims to become more sophisticated (e. g. multiple error sources/ components? • Reflects increasing awareness of the ineffectiveness / inefficiency of standardised procedures • Consistent with efficiency / budget-cut agendas Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Accessing the data and findings Study documentation: • Protocols, fieldwork reports, questionnaires, variable level metadata • User guides, Quality Profile, technical reports • Working Papers and publications from methodological research See www. understandingsociety. ac. uk Data are available at the UK Data Service, University of Essex: • Wave 1 - 5 data, including biomarkers; wave 6 and immigrant & ethnic minority boost sample to be deposited November 2016 • Innovation Panel data waves 1 -7 (wave 8 to be deposited July 2016) • Linked education data, geographical identifiers available by special licence or via secure lab See UK Data Service website: http: //ukdataservice. ac. uk/ Understanding Society: The UK Household Longitudinal Study https: //www. understandingsociety. ac. uk/ Research Methods Festival, Bath, July 2016
Appendix Lynn | Lausanne, 15 February 2013
1 -Person Households Response Rate Wave 2 Wave 3 Wave 4 Early trans MM 98. 2% 75. 0% 71. 3% Late trans MM 99. 1% 71. 8% +0. 9 +4. 1 +0. 5 2. 09 (0. 31) 1. 26 (0. 47) 1. 03 (0. 92) Difference Odds Ratio (P) Base = persons in wave 2 responding 1 -person households Lynn | Lausanne, 15 February 2013
2+-Person Households Response Rate Wave 2 Wave 3 Wave 4 Early trans MM 88. 7% 67. 3% 57. 2% Late trans MM 88. 2% 74. 3% 68. 9% -0. 5 +7. 0 +11. 7 0. 95 (0. 81) 1. 46 (0. 08) 1. 70 (0. 004) Difference Odds Ratio (P) Base = persons in wave 2 responding households with 2+ persons Lynn | Lausanne, 15 February 2013
Conditional on W 2 Outcome Response Rate MM-Early 70. 5 61. 4 77. 2 71. 3 MM-Early 29. 9 23. 8 MM-Late W 2 Non-respondents Wave 4 MM-Late W 2 respondents Wave 3 28. 0 27. 4 > Effect driven by wave 2 respondents Lynn | Lausanne, 15 February 2013
Increasing Take-Up of Web Original sample IP 4 responding % No bonus 25. 8 Monday Friday 22. 6 0. 61 % 4. 2 298 6. 3 292 21. 0 0. 01 N 295 17. 7 Bonus P IP 4 non-responding 4. 0 301 6. 5 P 0. 57 % 35. 5 52 35. 1 56 0. 66 N 55 % = proportion of households fully responding by web Lynn | Lausanne, 15 February 2013 Refreshment sample 34. 7 51 35. 9 P N 152 0. 93 151 150 0. 80 153