Скачать презентацию A Strategy for Prioritising Non-response Follow-up to Reduce Скачать презентацию A Strategy for Prioritising Non-response Follow-up to Reduce

258da3e0a226f597f5c7382b14d41ee2.ppt

  • Количество слайдов: 24

A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National Statistics

Outline of presentation • Introduction response-chasing in ONS business surveys • Understanding non-response effects, Outline of presentation • Introduction response-chasing in ONS business surveys • Understanding non-response effects, patterns and reasons • Strategy for response-chasing scoring methods – current investigations and future strategies 2

Introduction • Non-response … the failure of a business to respond in part or Introduction • Non-response … the failure of a business to respond in part or full to a survey. Effect on: – bias and standard error, – perception of output quality, – business behaviour Improve response rates by: – better questionnaire design, sample rotation rates, … – response-chasing - necessary, but expensive • Quality improvements and efficiency targets – effective targeting needed 3

Current practice at ONS • Use of % targets (mainly counts, occasionally other variables) Current practice at ONS • Use of % targets (mainly counts, occasionally other variables) • Written reminders to all. Then targeted phone calls, … could lead to enforcement • Businesses identified as ‘key’ (by survey area) chased intensively first • After ‘keys’, principle to chase large-employment businesses next • Methods differ between surveys 4

Current practice at ONS • Areas for improvement: – Methods for ‘key’ businesses: make Current practice at ONS • Areas for improvement: – Methods for ‘key’ businesses: make more consistent, transparent, scientific – Effective use of response-chasing tools – Team structure and knowledge (Area undergoing restructure) • Efficiency initiatives – save resources: some changes already implemented – effects being monitored; evaluation needed 5

Efficiency initiatives – removal of second reminders 6 Efficiency initiatives – removal of second reminders 6

UNDERSTANDING NON-RESPONSE UNDERSTANDING NON-RESPONSE

Patterns of non-response • Industrial sector - identified those with lower response rates (e. Patterns of non-response • Industrial sector - identified those with lower response rates (e. g. catering, hotels) • High correlation between industry response rates at early and final results • Size of business – larger businesses take longer to respond. Chasing strategy ensures responses are received later though 8

Intensive Follow-Up (IFU) exercise • Dual aims: – to estimate non-response bias (work in Intensive Follow-Up (IFU) exercise • Dual aims: – to estimate non-response bias (work in progress – see final paper) – to establish reasons for non-response and (later) cost responsechasing • Used the Monthly Inquiry into the Distribution and Services Sector (MIDSS): – – dedicated team for the IFU contacted c. 600 non-responders per month in chosen industries businesses to receive up to 5 phone calls reason for initial non-response; nature of call; length of call 9

IFU results – returned data • c. 80% of all businesses selected for IFU IFU results – returned data • c. 80% of all businesses selected for IFU returned questionnaire, but • many businesses returned questionnaire just after deadline – no call needed! • Only c. 60% of those contacted returned questionnaire 10

IFU results – reasons for non-response Reason for initial nonresponse Number who gave a IFU results – reasons for non-response Reason for initial nonresponse Number who gave a reason Returned data after IFU calls Still didn’t return data after IFU calls Forgot, missed date 667 77% 23% Too busy, too low priority 361 67% 33% 67% Actively decided not to 11

BUILDING A RESPONSE-CHASING STRATEGY BUILDING A RESPONSE-CHASING STRATEGY

Dealing with businesses that don’t respond • Aim to make response-chasing more efficient • Dealing with businesses that don’t respond • Aim to make response-chasing more efficient • Create a scoring system to prioritise/categorise non- responders • Focus on reducing non-response bias 13

Estimation in ONS business surveys We impute/construct where there is non-response. Then estimate totals Estimation in ONS business surveys We impute/construct where there is non-response. Then estimate totals as where 14

Bias in ONS business surveys • Total potential non-response bias (= total imputation error) Bias in ONS business surveys • Total potential non-response bias (= total imputation error) given by • We will concentrate on (i. e. the absolute error of imputation for each business) 15

Scoring - principles • Reduce imputation error by attempting to predict (Large value means Scoring - principles • Reduce imputation error by attempting to predict (Large value means increased risk if business is imputed – therefore target these) • May also wish to score to encourage good response behaviour from businesses – e. g. new-to-sample • Need a system that is easy to use and justify. 16

Scoring methods • (Mc. Kenzie) Calculate imputation error from previous returns; then rank into Scoring methods • (Mc. Kenzie) Calculate imputation error from previous returns; then rank into deciles: 0, 1, …, 9. (Smallest – Largest) New-to-sample or long-term non-responders = 10 Tested on MIDSS in 2001 -2; implementation issues • (Daoust) Calculate weighted contribution to estimates – categorise into 3 groups for follow-up • New investigations with adapted methods 17

Current investigations in MIDSS • Predict imputation error in monthly turnover (= y) – Current investigations in MIDSS • Predict imputation error in monthly turnover (= y) – Various predictors available – Rank businesses then group – No imputation score? Use stratum average. • Assess actual error against predicted. 18

Results (5 groups) • Percentage of within each priority score group Actual Score Imputation Results (5 groups) • Percentage of within each priority score group Actual Score Imputation error 4 88 3 8 2 3 1 1 0 << 1 19

Results • Percentage of within each priority score group Actual Score Weighted prediction Imputation Results • Percentage of within each priority score group Actual Score Weighted prediction Imputation error Previous imp. error 4 88 73 3 8 12 2 3 10 1 1 3 0 << 1 2 19

Results • Percentage of within each priority score group Actual Score Weighted prediction Imputation Results • Percentage of within each priority score group Actual Score Weighted prediction Imputation error Previous imp. error Register turnover 4 88 73 68 3 8 12 15 2 3 10 8 1 1 3 5 0 << 1 2 4 19

Results • Percentage of within each priority score group Actual Score Weighted prediction Imputation Results • Percentage of within each priority score group Actual Score Weighted prediction Imputation error Previous imp. error Unweighted prediction Register turnover Register employment 4 88 73 68 42 40 3 8 12 15 20 15 2 3 10 8 11 12 1 1 3 5 9 18 0 << 1 2 4 18 15 19

Conclusions • Significant gains available in response chasing Future plans: • Refinements to scores: Conclusions • Significant gains available in response chasing Future plans: • Refinements to scores: – – optimum predictor individual adjustments (e. g. long-term non-responders) overall or by separate industry groups? multivariate surveys • Dynamic updating of scores • Live testing 20

References • Daoust, P. , (2006), 'Prioritizing Follow-Up of Non-respondents Using Scores for the References • Daoust, P. , (2006), 'Prioritizing Follow-Up of Non-respondents Using Scores for the Canadian Quarterly Survey of Financial Statistics for Enterprises', Conference of European Statisticians • Mc. Kenzie, R. , (2000) 'A Framework for Priority Contact of Non Respondents', Proceedings of the Second International Conference of Establishment Surveys 21