- Количество слайдов: 29
New Dimensions of Quality in Online Panels Jacqueline Lorch Vice President, Global Knowledge Management Survey Sampling International
The World Is Moving On…
Wh y. D oes It M a tter ? Questionnaire Design Fiel dwo Sampling rk Interpretation
What Are The Key Quality Elements? § A broadly-recruited, representative, well-managed pool of respondents § Respondents who respond honestly and conscientiously: Ø What guarantees are there to guard against bad data, i. e. respondent cheating or not concentrating/caring in their responses… ? § A well-designed survey instrument
How Do Respondents Fit In? § Google “Market Research” 464 million entries “Market Research Respondents” 2 million entries
What Is A ‘Professional Respondent’? § Some assumptions: Ø Quality of responses will be lower Ø Motivated to maximize incentives, not by intrinsic interest Ø Do too many surveys – become conditioned § But what definition? Ø Someone who gets paid?
Honesty Is The Issue “I am an IT Director with 7 wives…”
New Typology Of “Bad” Respondents § Hyperactive Respondents Ø Too many surveys, too many panels § Fraudulent Respondents Ø Misrepresent themselves § Inattentive Respondents Ø Don’t put thought into answers § Conditioned Respondents Ø Have learned from past surveys Smith and Hofma Brown, Harris Interactive
Hyperactive Respondents § Do busy panelists provide bad data? § Do the most responsive panelists take the surveys the fastest? § Are the fastest surveys the ‘worst’?
Hyperactive Respondents § US Survey. Spot panel, 3 month internal tracker Ø Two groups: – Survey. Spot only – One or more other panels § No difference in average time taken by each group § Similar answers on motivation for joining: Ø Motivated by chance to influence decisions – Survey. Spot-only: 50% – More than one panel membership: 47%
Fraudulent Respondents § How can you tell if someone is who they say they are? § How can you stop multiple panel memberships using different identities? § How can you tell if someone is just making up the answers?
Who Are You Today? Ask me something only I would/should know…. .
Catch The Cheat… § Respondents… Ø Who have 23 different ailments Ø Who report using non-existent brands Ø Whose education doesn’t match their profession
Inattentive Respondents § Fatigue leads them to skip questions § Don’t pay attention to instructions § May be inevitable after a certain length of survey
Inattentive Respondents § Related to interview length Length of interview, minutes 14 19 21 Items 15 5 15 Skipped all 5% 8% 11% § 5% baseline of inattention?
Respondents Speak Out § Repetitive questions “sorry this survey was just too long. ” § “Sometimes it becomes so repetitive you say, ‘to hell with it, I don’t need this. ’” § “You think you are about done and the same questions start all over again. ”
Inattentive Respondents § Time use > 7300 hours = 4 hours of sleep a night § Time use > 8760 hours = no sleep, ever. . more hours than in a year Length of interview, minutes 14 19 21 >7300 hours 5% 6% 6% >8760 hours 3% 4% 4% § 5% baseline of inattention? § A really tough question set?
Respondents Speak Out § “…absolutely ridiculous… [questions like] ‘if this pizza was a person. ’ ” § “Why is this bottled water like your favorite pet? ” § “In filling out this survey it asked when I would buy a new house. I said “never”. The next several questions were regarding my new house and it required an answer…So I quit the survey and didn’t finish it. ”
Conditioned Respondents § Only give the answers they do because of what they have learned from previous surveys § Or change behavior as a result of information from surveys § Have no redeeming features § But do such people exist…. . ?
TNS Experiment § 3 groups of respondents Ø High frequency – interviewed 5 times (1. 2. 3. 4. 5) Ø Medium frequency – interviewed 3 times (1. 3. 5) Ø Low frequency – interviewed 2 times (1. 5) § + Control group at wave 5 § Same questionnaire § Survey frequency not yet released by TNS § UK, France, Germany § n = 1202 (control = 1470)
Product Usage High Low
Product Purchase High Low
Brand Awareness High Low
Conditioned Respondents § Evidence is hard to find Ø Maybe surveys aren’t quite so important to respondents as they are to researchers! § Could we be over-reacting and losing good respondents?
Respondents Speak Out § “I would like to know why it is that practically every time I give my age I am refused the access to the survey? Is there something wrong with being 75 and in good health, mentally and physically? ” § Why do I spend 10 minutes answering questions on one of your surveys before I am given message ‘Sorry, you didn't qualify for this survey’. § “I have not been able to take several [most] surveys because I work in the grocery industry. I don’t think it is fair. I AM A CONSUMER TOO … It is not like I work for a company that makes or sells one brand. ”
Respondents Speak Out § “…ability of the human eyes to see fonts of microscopic size. What’s with this survey? The font size was like. 002. ” § “I said yes before; why are you going back and asking me the same question. ” § “Right to the point”…“if they say it’s 5 minutes, it’s 5 minutes. ” § “By and large ask sensible questions in a straightforward way. ”
What Needs To Happen? § Partnership…common terms and definitions § Avoid red herrings § We are not in the business of supplying bad panelists § Survey design is critical § We can’t do it alone