Q 2014 Questionnaire Test Using Cognitive Approach

Q 2014 Questionnaire Test Using Cognitive Approach : Lessons Learned - Redesign of Time Use Survey Park, Hyun-Jung phj 88@korea. kr Questionnaire Design Lab, Statistics Research Institute

Contents § Background § Cognitive Approach § Cognitive Interview : TUS Redesign § Test Results § Conclusion How do you spend in every single day? 1 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Background § Survey Field Today: • Changed in its complex & diversity • Increased in use of mixed mode methods • Need to match the characteristics of survey instruments § Main goal of questionnaires testing using cognitive approach • Improve questionnaire to reduce response burden and to enhance the data quality 2 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Cognitive Approach (1/3) § Statistics Korea undermined the importance of questionnaire • Compared to the extensive work in other methodology research (Data collection, editing and imputation) § Based on the Cognitive Aspects of Survey Methodology • Established Questionnaire Design Lab (QDL) in 2013 • Conducted researches on major household surveys through the Lab § This paper focuses on the results of a recently tested Time Use Survey (TUS) 3 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Cognitive Approach (2/3) § The Lab includes two cognitive interview rooms, a usability test room, a FGI room, and an analysis room. Cognitive Interview Lab 2 Cognitive Interview Lab 1 Observation & Analysis Room Eye-tracking test Lab 3 4 Focus Group Interview room Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Cognitive Approach (3/3) Cognitive Interview Eye-tracking • Small group • Paid volunteers • Recording verbal data by in-depth interview • Recording respondents’ eye movements by eye-tracker • Detecting a wide range of potential sources of errors 5 • Identifies issues related to visual & verbal aspects Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Cognitive Interview(CI) Time Use Survey Period Response burden • Complex survey: gives respondents considerable amount of burden Questionnaire • Composed of two parts: 19 questions & time diary • Time diary: a sequential list of activities for reference days Main purpose 6 • Conduct every 5 -year survey since 1999 • Identify root causes of problems • Redesign a questionnaire prior to the upcoming field test Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

C I Method § In case of being absent from interviewers, self-administered by respondents • Instructions on how to complete a ‘time dairy’ will affect the fidelity of the content § Decided to experiment on the groups by controlling for the amount of instructions(long vs. short) on the time diary • The results will be compared on the differences in completion time and the number of activities 7 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Participants § Assess the accuracy of self-administered data and measure the completion time from two samples • First experiment: 8 staff from Statistics Korea • Second experiment: 20 participants recruited through homepages, a local newsletter, and by word of mouth. Gender Age group Men First (staff) Second Treatment Control 8 Women 3 4 2 2 5 16 8 8 Education Household member 20 -39 40 -59 College University 3 -4 + 3 4 1 7 8 2 16 5 15 16 1 8 3 7 9 1 8 2 8 7 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 Work 5+ Yes No 0 4 1 3 8 7 3 4 0 13 7 6

Interview Protocol § Prepare a list of potential cognitive issues to be investigated in the questionnaire § Scripted scenarios for appropriate probing questions to ask for each item Cognitive Interviewing Methods Think aloud 9 Probing Rating for response difficulty and confidence Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 Vignettes

Interview Process § Interviewer explains instructions on the time diary • Treatment group: short explanation (around 1 minute) • Control group: more detailed explanation (less than 5 minutes) § In-depth interview: 21 probing questions on selected items to collect thoughts and opinions of participants Welcome and introduction of task Responding to questionnaire In-depth interviewing (probing) Interviewer (Scenario) 10 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 Completion of cognitive interview

Analysis Results (Test time) § Longest completion time: 59. 4 mins. non-employed woman in age group 40 § Shortest completion time: 8. 4 mins. employed man in age group 30 80 Interviewer's explanation 70 60 34. 3 50 Completing questions 27. 4 27. 8 40 30 24. 9 26. 7 25. 2 Completion ‘time diary’ 29. 6 26. 9 18. 8 20 Probing 10 4. 1 11 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 tro l co n en t m tre at st af f d pl oy e ed N on -e m pl oy Em -5 9 40 -3 9 20 W om en en M To ta l 0

Analysis Results (Probing) Probing related to comprehension § About half of the participants(13/28) • Without difficulty • Found it difficult to compute bonuses including annual incentives and to distinguish pre-tax from post-tax income Example : Summary of probing results on ‘household income’ (Original question) Last year, how much was the monthly gross household income before the tax deduction? (Experiment results) About half of the participants understood the question Some did not understand ‘transferred income’ in income description (Suggested revision) Specify ‘transferred income’ with examples (Ex. pension, premium, living expenses form other household, etc. ) 12 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Analysis Results (Probing) Probing related to comprehension § More than half of the participants: • Have generally used references on three items • But rarely used on ‘unpaid family workers’: it is likely that regularly paid workers could easily respond to the question without the reference. Usage of Term Descriptions Head of household Household income Unpaid family workers Employment status 23/28 8/28 17/21* Reference * If only employed 13 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Analysis Results (Probing) Probing related to comprehension § Rating the degree of difficulty : • Normally easily understood • ‘Status on separated family and reasons’: more response burden than other items Level of difficulty on questions and descriptions Item Status on separated family and reasons Average monthly household income Unpaid family worker Satisfaction on time use in ‘time diary’ Probing questions · How hard was this question to answer? 3. 82 · How hard was this description to understand? 3. 24 · How hard was this description to understand? · How hard was ‘the day when you wrote the time diary’ to understand? 2. 57 * Rated on a 10 -point scale (1 = definitely easy, 10 = definitely difficult) 14 Score Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 2. 75

Analysis Results (Probing) Probing related to retrieval § Rating the confidence: “How certain are you of your answer? 7. 44 to 8. 81 on a 10 -point scale • Working hours: even the open-ended, scored the highest because majority respondent are regular workers • Total floor area of the house: identified problems such as “don’t know” and “unit conversion” • Household income: scored higher than expected Level of self-confidence on responses Item Total floor area of the house Household income Working hours Score 7. 44 8. 11 8. 81 Remarks Retrieval and unit conversion Response category use Most respondents were regular worker (1=least confident, 10=most confident) 15 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Analysis Results (time diary) § Reliability of responses in the ‘time diary’ through the number of activities • Number of activities: statistically significant differences in gender and employment status § Efficiency through completion time of the ‘time diary’. • Completion time: no statistical significance Total men women 20 -39 40 -59 Employed Non-employed 16 n 28 7 21 9 19 17 11 Number of Activities 31. 1 24. 1 33. 4 29. 2 32. 0 27. 1 38. 2 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 T p-value -4. 28 0. 000 -0. 82 0. 42 -3. 85 0. 002

Analysis Results (time diary) § In comparison to other groups • Completion time: statistically significant difference between ‘readers’ and ‘skimmers’ It is debatable whether ‘readers’ are less efficient than ‘skimmers’ • Number of activities: no statistical significance Group Staff Treatment Control Pressure No pressure Readers Skimmers n 8 10 10 16 12 8 10 Completion time 18. 77 25. 28 25. 98* 25. 29 21. 10 26. 99 18. 70 T(F) p-value Number of Activities 26. 12 33. 60 1. 76 0. 194 32. 50 31. 94 1. 21 0. 239 29. 92 31. 25 2. 30 0. 035 29. 50 * The data for the participant with longest completion time was excluded. 17 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014 T(F) p-value 2. 30 0. 121 0. 64 0. 527 0. 42 0. 678

Conclusion (1/3) § Cognitive response process of TUS • Comprehension: respondents found it less difficult to understand questions or term descriptions (lower than 5. 5 mid-point) • Retrieval: more confident about their responses (higher than 5. 5 mid-point) • Response: modified or deleted through probing on the easiness and vignette classifications on the validity of response categories § Average response time: about 64 minutes • Reading time for instructions(2 min. )+completion time for questions(12 min. )+time diary(25 min. per day) 18 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Conclusion (2/3) § Problems in the questionnaire • Difficulties understanding terms and where to write the answers • Instructions not read thoroughly on the ‘time diary’ → instructions need to be more user-friendly by using visualized symbols § Efficiency and integrity of the survey (other groups) • Completion time: significant difference in only readers/skimmers • Number of activities: in gender and employment status 19 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Conclusion (3/3) § Cognitive interviewing generally identifies problems • Not guarantee statistical significance of the results (Willis, 2005) • Cost effective method for collecting quality information with a small sample § Statistics Korea plans to review main questionnaire-based surveys one by one • Challenging task but a good opportunity to improve the quality of official statistics 20 Questionnaire Test Using Cognitive Approach: Redesign of Time Use Survey 3. June 2014

Thank You