Observation, Interviews, and Questionnaires a. k. a. How to watch and talk to your users
Agenda ¢ Questions? l l l ¢ ¢ Reminder: part 3 due NEXT WEEK Bring your prototype to class Highly recommended: read on Heuristic Evaluation and Cognitive Walkthrough, be prepared for them Observation Interview Questionnaire Evaluation plan discussion
Observing Users ¢ Not as easy as you think ¢ One of the best ways to gather feedback about your interface Watch, listen and learn as a person interacts with your system ¢ ¢ Usually what occurs during a “usability test”
Location ¢ Observations may be l In lab - Maybe a specially built usability lab • Easier to control • Can have user complete set of tasks l In field • Watch their everyday actions • More realistic • Harder to control other factors
Usability Lab Large viewing area in this oneway mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms. Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room. http: //www. surgeworks. com/services/observ ation_room 2. htm
Observation ¢ Direct l l l In same room Can be intrusive Users aware of your presence Only see it one time, relies on good note-taking May use 1 -way mirror to reduce intrusiveness ¢Indirect l. Video recording l. Software logging l. Reduces intrusiveness, but doesn’t eliminate it l. Gives archival record, but can spend a lot of time reviewing it
Engaging Users ¢ In simple observation, you see actions but not what is going on in their head ¢ Qualitative techniques l l Think-aloud - very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions • “What did you like best/least? ” • “How would you change. . ? ”
Think Aloud ¢ User describes verbally what s/he is thinking and doing • What they believe is happening • Why they take an action • What they are trying to do ¢ Very widely used, useful technique Better understand user’s thought processes ¢ Potential problems: ¢ l l Can be awkward for participant Thinking aloud can modify way user performs task
Cooperative approach ¢ Another technique: Co-discovery learning (Constructive iteration) l l ¢ Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert (coach) and one be novice More natural (like conversation) so removes some awkwardness of individual think aloud Variant: let coach be from design team (cooperative evaluation)
Alternative What if thinking aloud during session will be too disruptive? ¢ Can use post-event protocol ¢ User performs session, then watches video afterwards and describes what s/he was thinking l Sometimes difficult to recall l Opens up door of interpretation l
What if a user gets stuck? ¢ ¢ Determine in advance when and how you will offer help Use cooperative approaches: l l l “What are you trying to do. . ? ” “What made you think. . ? ” “How would you like to perform. . ? ” “What would make this easier to accomplish. . ? ” Maybe offer hints
Inputs ¢ Need operational prototype l ¢ could use Wizard of Oz or other simulation Need tasks and descriptions l l Reflect real tasks Avoid choosing only tasks your design best supports Minimize necessary background knowledge Pay attention to time and training required
Data ¢ Task based l l l ¢ Performance based l ¢ How do users approach the problem What problems do users have Need not be exhaustive, look for interesting cases Frequency and timing of actions, errors, task completion, etc. Analyzing data can be very time consuming!
Capturing a Session ¢ 1. Paper & pencil Is definitely cheap and easy l Can be slow l May miss things l Task 1 Time 10: 00 10: 03 10: 08 10: 22 Task 2 S e Task 3 S e …
Capturing a Session ¢ 2. Recording (audio and/or video) Good for think-aloud l Hard to tie to interface l Multiple cameras may be needed l Good, rich record of session l Can be intrusive l Can be painful to transcribe and analyze l
Capturing a Session ¢ 3. Software logging Modify software to log user actions l Can give time-stamped key press or mouse event l Two problems: l • Too low-level, want higher level events • Massive amount of data, need analysis tools
Example: Heather’s study ¢ ¢ Software: Meeting. Viewer interface fully functional Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing Resources – subjects were students in a research group, just me as evaluator, plenty of time Wanted completely authentic experience
Heather’s evaluation ¢ ¢ ¢ Task: answer questions from a recorded meeting, use my software as desired Think-aloud Video taped, software logs Also had post questionnaire Wrote my own code for log analysis Watched video and matched behavior to software logs
Example materials
Example logs 2303761098721869683|hrichter|1098722080134|MV|START|566 2303761098721869683|hrichter|1098722122205|MV|QUESTION|false|false|false| 2303761098721869683|hrichter|1098724978982|MV|TAB|AGENDA 2303761098721869683|hrichter|1098724981146|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098724985161|MV|SLIDECHANGE|5 2303761098721869683|hrichter|1098724986904|MV|SEEK|PRESENTATION-A|566|604189|0 2303761098721869683|hrichter|1098724996257|MV|SEEK|PRESENTATION-A|566|604189 2303761098721869683|hrichter|1098724998791|MV|SEEK|PRESENTATION-A|566|604189 2303761098721869683|hrichter|1098725002506|MV|TAB|AGENDA 2303761098721869683|hrichter|1098725003848|MV|SEEK|AGENDA|566|149613|604189 2303761098721869683|hrichter|1098725005981|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725007133|MV|SLIDECHANGE|3 2303761098721869683|hrichter|1098725009326|MV|SEEK|PRESENTATION|566|315796|149613 2303761098721869683|hrichter|1098725011569|MV|PLAY|566|315796 2303761098721869683|hrichter|1098725039850|MV|TAB|AV 2303761098721869683|hrichter|1098725054241|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725056053|MV|SLIDECHANGE|2 2303761098721869683|hrichter|1098725057365|MV|SEEK|PRESENTATION|566|271191|315796 2303761098721869683|hrichter|1098725064986|MV|TAB|AV 2303761098721869683|hrichter|1098725083373|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725084534|MV|TAB|AGENDA 2303761098721869683|hrichter|1098725085255|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725088690|MV|TAB|AV 2303761098721869683|hrichter|1098725130500|MV|TAB|AGENDA 2303761098721869683|hrichter|1098725139643|MV|TAB|AV 2303761098721869683|hrichter|1098726430039|MV|STOP|566|271191 2303761098721869683|hrichter|1098726432482|MV|END
Data analysis ¢ Basic data compiled: l l l ¢ Time to answer a question (or give up) Number of clicks on each type of item Number of times audio played Length of audio played User’s stated difficulty with task User’s suggestions for improvements More complicated: l l Overall patterns of behavior in using the interface User strategies for finding information
Data representation example
Data presentation
Some usability conclusions ¢ ¢ ¢ Need fast forward and reverse buttons (minor impact) Audio too slow to load (minor impact) Target labels are confusing, need something different that shows dynamics (medium impact) Need more labeling on timeline (medium impact) Need different place for notes vs. presentations (major impact)
Interviews & Questionnaires ¢ ¢ Subjective view of participants Quantitative – very structured l Questionnaires • often quantitative, but not entirely l Structured Interviews • Strict set of questions, deviation would compromise study ¢ Qualitative – less or no structure l Semi-structured interviews • Some deviation encouraged l Unstructured interviews • i. e. the ethnographic interview • Little guide, very explorative
Interviews Potentially lots of detail ¢ can vary questions as needed ¢ Inexpensive ¢ Time consuming to perform and analyze ¢ Some interpretation required ¢ Subject to interviewer biases ¢
Questionnaires Expensive to create ¢ …but cheap to administer ¢ Easier to get quantifiable results ¢ Can gather info from many more people ¢ Protects participant identity ¢ Only as good as the questions asked ¢
Structured Interviews More similar to questionnaires ¢ Require a lot of training for any hope at inter-interviewer reliability ¢ But that means that they tend to give much more repeatable results ¢
Unstructured Interviews Have a plan, but keep interview open to different directions ¢ Get participant to open up and express themselves in their terms and at own pace ¢ Create interpretations with users ¢ l ¢ Be sure to use their terminology Take lots of time, but learn a lot as well
Semi-Structured Interviews ¢ ¢ Predetermine data of interest - know why you are asking questions - don’t waste time Plan for effective question types • • l How do you perform task x? Why do you perform task x? Under what conditions do you perform task x? What do you do before you perform…? What information do you need to…? Whom do you need to communicate with to …? What do you use to…? What happens after you…? See Gordon & Gill, 1992; Graesser, Lang, & Elofson, 1987
Asking Questions Understand your goals ¢ Consider the ordering of the questions ¢ Avoid complex/long/multiple questions ¢ Avoid jargon; talk in participant’s language ¢ Be careful of stereotypes, biases ¢
Clarity is important Questions must be clear, succinct, and unambiguous How much time have you spent reading news on the Web recently? q q q Some A lot Every day Rarely Etc. q. None q 0 to 5 hours q 6 to 10 hours q 11 to 20 hours q. More than 20 hours
Avoid question bias Leading questions unnecessarily force certain answers. Do you think parking on campus can be made easier? What is your overall impression of… 1. Superb 2. Excellent 3. Great 4. Not so Great
Be aware of connotations Do you agree with the NFL owner’s decision to oppose the referee’s pay request? Do you agree with the NFL owner’s decision in regards to the referee’s pay demand? Do you agree with the NFL owner’s decision in regards to the referee’s suggested pay?
Leading questions ¢ ¢ ¢ People want to do well, give you what you are looking for Be aware of your own expectations before creating questions and while interviewing Use value neutral terms What do you like about this system? Vs. Tell me what you thought about this system.
Avoid hypotheticals ¢ Avoid gathering information on uninformed opinions ¢ Subjects should not be asked to consider something they’ve never thought about (or know or understand) Would a device aimed to make cooking easier help you?
Handle personal info carefully ¢ Ask questions subjects would not mind answering honestly. What is your age? l What is your waist size? l ¢ If subjects are uncomfortable, you will lose their trust ¢ Ask only what you really need to know
What’s wrong with this picture? How much easier is it to use this email client than Outlook? ¢ I see you choose to use your keyboard shortcuts more than the mouse. Is that faster for you? ¢ Your choice of red is different than any other user we saw. Why did you do that? ¢
Planning your interview: ¢ ¢ ¢ Introduction Warmup Main session Cool-off Closing Record everything exactly in your participants’ languages (don’t forget to test your recording equipment)
The warmup or “grand tour” question ¢ The first question helps set the tone for the interview l l ¢ Familiarize the participant to talking Encourage the participant that their true opinion does matter Question should be l l Easy to answer But not answered easily • More than just a “yes” or “no” response ¢ Examples: l l Tell me about the work you do? What made you buy the computer?
Prompts ¢ “Nudge” a participant in a direction, or to get additional response l l l Silent: remain silent until they say more Echo: repeat back and then ask “then what happens” etc. Make agreeing sounds: you say “uh huh” and the other person continues Tell Me More: could you tell me more about that? Clarifying: summarize and ask for confirmation or clarification, often leads to new discussion
Contents of a survey ¢ General/Background info Demographic data l Also functions as as a “warm up” l Correlate responses between groups l ¢ Objective questions ¢ Open-ended/subjective
Background examples ¢ Demographic data: Age, gender l Task expertise l • i. e. Have you ever worked in a restaurant? Motivation l Frequency of use l • How often do you… l Education/literacy • What training have you had in …?
Closed Format Restricting set of choices Quantifiable ¢ Advantages l l l Clarify alternatives Easily quantifiable Eliminate useless answer ¢ Disadvantages l l l Must cover whole range All should be equally likely Don’t get interesting, “different” reactions
Many forms of response Dichotomous ¢ Multiple Choice ¢ Multiple Response ¢ Rank/Match ¢ Likert ¢ Rating ¢
Questionnaire Styles Which word processing systems do you use? La. Te. X Word Frame. Maker Word. Perfect Rank from 1 - Very helpful 2 - Ambivalent 3 - Not helpful 0 - Unused ___ Tutorial ___ On-line help ___ Documentation
Likert-type scale Typical scale uses 5, 7 or 9 choices ¢ Above that is hard to discern ¢ Doing an odd number gives the neutral choice in the middle ¢ You may not want to give a neutral option ¢ Characters on screen were: hard to read 1 2 3 4 5 easy to read 6 7
What’s wrong with this picture? 2. What is your age? ________ 3. How long have you used the internet? <1 year 1 -3 years 3 -5 years >5 years 4. How do you get information about courses? Email Web site Flyers Registration booklet Advisor Other students 5. How useful is the Internet in getting information about courses? ___________________________________________________________
On line questionnaires Email or internet ¢ Change checkboxes into dropdowns, etc ¢ Take advantage of the technology – check input ¢ Ensure its as accessible as paper (browser and email client compatibility) ¢ Ensure confidentiality – how is this different from paper? ¢
Free Web Survey Tools ¢ Zoomerang l ¢ Survey Monkey l ¢ http: //www. zoomerang. com http: //www. surveymonkey. com php. ESP http: //phpesp. sourceforge. net l Open Source surveys using PHP. l
Analyzing your quantitative data ¢ “Code” open ended responses or interview questions to make quantitative l ¢ Categorize all responses Look for trends in the data l l Count, average, tabulate Make charts, etc Run statistical analysis Use lo-fi methods (post-its, affinity diagrams, etc)
Analyzing qualitative data Find interesting cases, responses ¢ Look for patterns of responses ¢ l Use post-its, affinity diagrams, etc. Look for any useful suggestions, improvements, explanations that help you improve your design ¢ Gather illustrative quotes from users that demonstrate your conclusions ¢
Evaluation discussion Someone else should be able to pick up your plan and execute it. ¢ Be as SPECIFIC as possible ¢ What criteria are important? l What tasks EXACTLY? l What data? How will you record? l What questions will you ask? l
Скачать презентацию Observation Interviews and Questionnaires a k a How
1554752913e3f130f15e765ab0605019.ppt