Usability Evaluation Issues in Commercial and Research Systems Laila Dybkjær, Niels Ole Bernsen NISLab, University of Southern Denmark Hans Dybkjær Speech. Logic™, Prolog Development Center A/S ASIDE 2005 -11 -10 COST Workshop, Ålborg University Slides available at www spokendialogue. dk/publications/2005 k/ASIDE-2005 -11 -10. ppt
NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic It’s all about design – usable design. . . Users, prompts, modalities, media, . . . Usability!
NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic Usability in academia and industry Academia: New and challenging? Focus on advanced systems and new knowledge Industry: Cost? ROI? Market? Customers? Focus on state-of-the-art and functionality But they have much to learn from each other · Lots of research results to streamline for industry · Lots of ”simple systems” questions open to research · Gap: EU research visions vs. industrial reality What can they learn from each other?
Three examples NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic System Traffic FAQ NICE HCA Task / domain Road traffic information Holiday allowance information H. C. Andersen’s life and fairytales edutainment Purpose Commercial Research I/O Speech Commercial Gov. support Speech Language Da 50 words Da 500 words En 2000 words Target All employees Children 10 -18 Car drivers Who built PDC it PDC, NISLab Speech, gesture, 3 D graphics NISLab and 4 other EU partners Wide range in purpose and complexity
NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic Cost and complexity Months Traffic 400 hours Simple FAQ 4000 hours Complex NICE HCA 40000 hours Very complex 0 2 9 13 23 36 F 2005 F 1 F 2 2002 P 1 P 2 2005 Academic focus: Prototypes – Industry focus: Final systems
NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic Usability evaluation criteria Difficult to select right criteria for given system • Is purpose to compare, to investigate, • or to define contract? To make proper selection, one must know the range and properties of criteria available Many usability criteria vaguely defined • E. g. “adequacy” or “sufficiency” of … Quantifiability often missing • Subjective or qualitative evaluation New system types require new criteria • Must be clearly defined and operationalised Standards may emerge, but new needs keep coming
Core usability evaluation criteria NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic System Criteria Traffic Interaction problems Correctness Transaction success FAQ NICE HCA Conversation success Naturalness Reasoning capabilities Ease of use Error handling Task and domain completeness Scope of user modelling Entertainment and education value User satisfaction Clearly different focus in academia and industry
NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic Usability evaluation methods Which one to choose depends e. g. on • Evaluation purpose • Resources (who, time, money) • Stage of development process Examples of methods • Walkthrough (early) • Focus groups (early, but ok any time) • Wizard-of-Oz (early-middle) • Field test (late) • Heuristic evaluation (best early but also ok later) • User interviews and questionnaires (any time) Many current practice methods!
Usability evaluation methods NISLab Methods Traffic Walkthrough (using Dialog. Designer) Semi-formal WOZ (using Dialog. Designer) In-house scenario-based test Expert evaluation of domain information FAQ Walkthrough (manually) In-house and external scenario-based test Questionnaire on web Monitored scenario-based lab tests Field data analysis Expert evaluation of domain information WOZ in schools and in museum Lab-test of first and second prototype Post-lab-test interviews & 2005 -11 -10 ASIDE 2005 Speech. Logic System NICE HCA Industrial systems need broad range of methods
Data and analysis NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic System Data Analysis Traffic Logfiles Problem identification via observation and feedback Log-based analysis of problems FAQ Logfiles; transcriptions; transaction annotations; questionnaires Logfiles; transcriptions; topic annotation; English evaluation; interviews Problem identification via observation, feedback from users and domain experts, and analysis of transcribed dialogues Analysis of WOZ and lab test data for design input; analysis of lab tests and interviews to get users’ opinion and develop new criteria NICE HCA Research more data and analysis needed
NISLab & Sketch, prompt design and recording, walkthrough, WOZ, document, test, formal properties (coherence, well-formedness, . . . ), . . . , you name it 2005 -11 -10 ASIDE 2005 Speech. Logic Electronic model IT tools possible A lot of knowledge and theory can be made operational
NISLab & 2005 -11 -10 ASIDE 2005 Speech. Logic Academia and industry do meet. . . Industrial actions and challenges • Optimise existing processes • Automation (transcription support, annotation, . . . ) • Use known results and theories • Unknown effects of new technology Academic challenges • Highly sophisticated technology • New factors to analyse, define, and measure • On-line adaptivity to users’ skills, expertise, … • Investigate troubles with ”simple” systems. . . even though they are also different beasts
Скачать презентацию Usability Evaluation Issues in Commercial and Research Systems
b084a8fca819b79c44a8b4d39f8f9572.ppt