Survey and Questionnaire Design MAR 6648 Marketing Research

Survey and Questionnaire Design MAR 6648: Marketing Research January 20, 2010

Overview • How do you write survey questions? – What are some common traps survey writers fall into? – How do we avoid them? • What about special circumstances? – Embarrassing questions, etc. ?

Survey Research • Descriptive research • Survey modes and methods: – Telephone surveys – Mail surveys – Face-to-face surveys • At home • Mall intercepts – Web-based surveys

Advantages of Surveys • Can collect a great deal of data, including – Attitudes, interests, opinions, behaviors – Classification variables • Can be employed in virtually any setting • Can be inexpensive, depending on format

Disadvantages of Surveys • Effective implementation requires considerable judgment • Social interaction of interview may lead to errors • Subject to demand honesty concerns

First Big Point: Biases • Basic Tenet: Ask a question, and you’ll get a response! • Translation: Garbage in, garbage out!

Typical problems in wording questions

How questions create answers

How questions create answers

How questions create answers • A survey for Black Flag said: – “A roach disk… poisons a roach slowly. The dying roach returns to the nest and after it dies is eaten by other roaches. In turn, these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches? ” – Not surprisingly, 79% said effective! • A Gallup poll sponsored by the disposable-diaper industry asked: – “It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers? ” – 84% said no!

How questions shape answers • Some examples – Behavioral frequency questions • “How often do you…? ”

Behavioral Frequency Questions • How many hours of TV do you watch on an average week day? Please choose the category from the list below that best describes your behavior. Marketers Question: What % of class watches more than 2. 5 hrs of TV?

Behavioral Frequency Questions • Scale represents distribution in ‘real’ world – Values in middle reflect “average” / “usual” behaviors – Values at an end (or ends) reflect extreme behavior • Low Frequency Scale – Lower estimates of “average” – Avoid marking extreme high box • High Frequency Scale – Makes marking “>2. 5 hours” more acceptable • “Response Scale Effect”

How questions shape answers • How much money do you have in savings? – 1 -7; 1 = $0, 7 = $500 and over – 1 -7; 1 = $500 and under, 7 = $500, 000 and over • How wealthy are you?

How questions shape answers • Some examples – Behavioral frequency questions – Aided versus unaided questions

Aided and Unaided Questions • What do you consider to be the most important thing for children to prepare them for life? – – 1. ________ 2. ________ 3. ________ 4. ________ • What do you consider to be the most important thing for children to prepare them for life? – – – __Good education __Inheritance __Moral values __To think for themselves __Good friendships __Others (please specify)

How questions shape answers • Some examples – Behavioral frequency questions – Aided versus unaided questions – Framing of questions

Framing Effects • Question forms that lead respondents to give a certain response – Example: Allow versus forbid Yes No Q: Do you think the U. S. should forbid public speeches against democracy? 5% 95% Q: Do you think the U. S. should allow public speeches against democracy? 81% 19%

Framing Effects • Question form • Choice of example/source

Framing Effects • Question form • Choice of example/source – Surveyor power: Mispredicting political outcomes (Bischoping & Schuman 1992) – Perhaps partly motivational? • People who feel good about themselves show less of this bias (Cohen 2003)

How questions shape answers • Some examples – Behavioral frequency questions – Aided versus unaided questions – Framing of questions – Scales associated with questions • Odd versus even numbered scales • Semantic versus Stapel scales

How questions shape answers • Some examples – Behavioral frequency questions – Aided versus unaided questions – Framing of questions – Scales associated with questions – Sequence of questions

Sequence of Questions Q 1: General Colin Powell, the hero of the Gulf War, recently decided to become a member of a political party. Do you happen to know which party that is? Q 2: Now rate this party. Q 1: General Colin Powell, the hero of the Gulf War, recently decided to become a member of a political party, but decided not to run for the presidential elections. Do you happen to know which party that is? Q 2: Now rate this party. • Specific – General Questions – Happiness and dating (How many dates have you had recently? ) • Order 1 (Happiness then dates): Correlation = 0. 03 • Order 2 (Dates then happiness): Correlation = 0. 67

Sequence of Questions • Difficultly in answering one question can influence subsequent answers • List three (twelve) ways in which you are happy (unhappy) with your partner. • How happy are you with your partner?

Sensitive Questions • Respondents may not be willing to answer some questions, due to embarrassment, legal issues, etc. • This may result in – Item nonresponse – Termination – Distortion

Asking the Embarrassing Question • The casual approach – “Do you happen to have killed your wife? ” • The numbered card approach – “Would you please read off the number on this card which corresponds to what became of your wife? : • 1: Natural Death • 2: I killed her • 3: Other • The everybody approach (counter biasing method) – “As you know, many people have been killing their wives these days. Do you happen to have killed yours? ” • The other people approach – “Do you know any people who have murdered their wives? How about yourself? ” Barton, POQ, 1958

How to ask a sensitive question • Personal information – Build rapport, legitimize yourself • Embarrassing info – Downward bias – Counterbiasing statement • “Recent studies indicate that men often use their wives’ hair spray. Do you use your wife’s hair spray? ” – Projective techniques

Projective Techniques

Key Points • When wording questions… – Use simple words – Use clear words – Avoid leading questions – Avoid implicit alternatives – Avoid implicit assumptions – Avoid double-barreled questions – Consider frame of reference

Key Points • Don’t forget about question sequence – It can influence the nature of the respondent’s answers and can be the cause of serious error in the results • Use a simple and interesting opening question • Ask general questions first • Place uninteresting and difficult questions late in the sequence • Arrange questions in logical order

The Basic Steps of Questionnaire Design • Decide on question content • Decide how to ask each question – Response format – Question wording • Decide on question sequence – Funnel approach (general to specific) • Decide on physical characteristics • Carry out pretest, revise, and final draft

So… • What is a good survey question? • How are we to interpret survey question output?

A Managerial Problem In a few weeks, Verizon will start selling a version of the i. Phone compatible with its network. How likely is it that you will buy a Verizon i. Phone? Very Unlikely Somewhat Likely Very Likely • Temptation: – Report those numbers to your manager as is • Problem: 15% 35% 15% – Those numbers have no direct value • What does “somewhat likely” mean in real life?

A Managerial Problem In a few weeks, Verizon will start selling a version of the i. Phone compatible with its network. How likely is it that you will buy a Verizon i. Phone? Very Unlikely Somewhat Likely 35% – 1. Benchmark Very Likely 15% • Solutions: 15% • What do past surveys say? Actual Purchase Rate AT&T i. Phone 12% 28% 30% 9% VZ Droid Incredible 20% 25% 30% 25% 5%

A Managerial Problem In a few weeks, Verizon will start selling a version of the i. Phone compatible with its network. How likely is it that you will buy a Verizon i. Phone? Very Unlikely Somewhat Unlikely 15% 35% Somewhat Likely 35% Very Likely • Solutions: – 1. Benchmark • What do past surveys say? – 2. Focus on questions you can answer 15% • Relative responses on the same items Verizon Customers 35% 20% 10% AT&T customers 15% 25% 30%

A Managerial Problem In a few weeks, Verizon will start selling a version of the i. Phone compatible with its network. How likely is it that you will buy a Verizon i. Phone? Very Unlikely 15% Somewhat Unlikely 35% Somewhat Likely 35% Very Likely 15% • Solutions: – 1. Benchmark • What do past surveys say? – 2. Focus on questions you can answer • Relative responses on the same items – 3. Ask other questions/use other methods

Summary • Survey design is harder than it looks – Be sure to think about what the question suggests about the answer • Order and scale points matter, too – Be sure to keep respondents’ comfort in mind