Up Down Left Right A Usability Postmortem Tony

Up Down Left Right A Usability Postmortem Tony Rado – Executive Director Global Quality Assurance, SOE trado@soe. sony. com Paul Venuti – Director Development Services, SOE pvenuti@soe. sony. com www. soedevservices. com

Up, Down, Left, and Right. A Usability Postmortem Ø Ø Ø Why usability? Usability statistics Organizational relationships Benefits of in-house lab Challenges of in-house lab Usability lifecycle Ø 5 Phases Ø 18 month trial and error Ø Lessons learned Ø Vivox (EQ, EQ 2, SWG) Ø Free Realms Ø DCUO, Agency

A Usability Postmortem Why Usability? ØDesigners, Developers, Producers, and Executives watch target demographic play games ØIncrease time to market ØControlled environments ØWe didn’t have solution in place to extract critical information from target audience ØIssues with beta testing: Misinformation Critics of game Not targeting audience but accepting audience

A Usability Postmortem Usability Stats - May 08 to August 09 44 Usability Tests 1 Focus Test 2 Time-to-Task Data Collection Total of 473 Candidates Ø 60 Candidates – Competitive Analyses Ø 47 Candidates – Concurrent Vivox Ø 23 Candidates – Web Presence Ø 18 Candidates – The Agency Ø 177 Candidates – Free Realms Ø 7 Candidates – 3 rd Party Contract Ø 36 Candidates – Free Realms First Impression Focus Test Ø 71 Candidates – Free Realms Combat and Racing Data Gathering/Balance Ø 34 Candidates – DCUO

A Usability Postmortem Organizational Relationships: ØDevelopment Services ØQuality Assurance ØDevelopment ØMarketing

A Usability Postmortem Benefits of an In-House Usability Lab? Ø Quality equates to success Ø Educative Ø Quality buy-in from all vested parties Ø No limit on testing iterations Ø Real time monitoring Ø Immediate feedback / Immediate resolution Ø Ability to record A/V on site “Best practices call for between 8% and 13% of project budget be spent on Usability” *Jakob Nielsen, Alertbox, 7 January 03 Ø Not an ROI in classical sense ØProject cost measured in terms of money ØUsability measured in terms of increased usage, efficient use, high user satisfaction*

A Usability Postmortem Challenges of Building In-House Usability Lab? ØChanging Culture ØShrinking Budgets - $$$ ØInfrastructure ØTechnological Challenges ØDistance between lab and monitoring room: ØVideo ØAudio ØReal time synced camera and game play feed support for satellite studios

A Usability Postmortem [Embedded Video]

A Usability Postmortem Consultation Phase Initial and Ongoing Communication ØJob request ØPlanning meeting ØTechnological challenges ØDefine measurables Lesson Learned ISSUE: Requesting party disagreement: ØGoals ØData points ØCannot please and appease everyone RESOLUTION: Created one point of contact for which all communication funneled through

A Usability Postmortem Recruitment Phase Ø Generalized recruitment categories for both Internal and External candidates include: ØAge ØSex ØHr/Wk playing games (Online/PC/Console) ØFavorite games ØSpecialized – 3 rd party contract recruited an ethnic target for European/South American game

A Usability Postmortem Internal & External Candidates Internal Candidates Ø Cost Benefit – Recruitment and incentive costs Ø Initial use of internal candidates were employees then employee children Ø Recruited internal candidates with same requirements as if recruiting externally External Candidates ØFree Realms iterative – Created database of employee children and friends ØTreated employee’s children as external candidates ØRecruited Boy Scout Troops, local kids ØExhausted internal children pool and began external recruiting ØTime saved by utilization of external recruiting company

A Usability Postmortem Candidate Bias Internal Caroline Jarrett, in regards to internal candidate bias, states that “they may be your target users, in which case they are perfect” * **Direct observation doesn’t lie** External Ø Getting paid Ø Desire to please [Embedded Video] Ø On-site testing *Society of Technical Communications Usability Special Interest Group – March 28, 2009

A Usability Postmortem Recruitment Phase Lessons Learned - Internal ISSUE: Internal bias would taint data due to desire to please or be less than honest RESOLUTION: Ensured all candidates understood: ØSetting of quality bar equated to job security! ØPride in having input affect change ØOpinions would not affect job security! Lessons Learned - External ISSUE: Candidate no-shows! Sometimes extended testing cycle by one or two days… RESOLUTION: External recruiting company double confirmation including on testing day, Double book.

A Usability Postmortem Test Plan Creation Phase Foundation – Plan to Field Measurable Data Points ØContact Information ØScenario Questions ØDeliverables ØDefined Roles Ø User Profile Information (which ones were used) Ø Pre-test Questionnaires Ø Scenarios Lesson Learned ISSUE: The “You’re Doing It Wrong” syndrome RESOLUTION: Test plan approval

A Usability Postmortem Test Plan Example

A Usability Postmortem Testing Phase ØTechnicians – Ensure environment stability and smooth technological sailing ØModerators – Conduct testing, exhibit patience ØAnalysts – Observe, gather data, ensure test follows plan Lesson Learned Don’t give children sugar before a test… [Embedded Video]

A Usability Postmortem Testing Phase Lessons Learned ISSUE: Communication between monitoring room and Ulab RESOLUTION: Bluetooth headsets, IM, m. IRC, SMS ISSUE: Not working as intended (Mechanics, New Build) RESOLUTION: Full test plan walkthroughs before approval process and walkthroughs on test day Jakob Nielsen’s first rule of Usability: Don’t listen to users!!! [Embedded Video]

A Usability Postmortem Post Test Reporting Phase Largely Conducted by Analysts ØTop Ten Issues List – Executive Summary ØData gathering ØVideo highlight generation ØDesign suggestions ØPrioritized screenshots of Usability severity issues ØReport generation ØDeliverable ØPostmortem

A Usability Postmortem Post Test Reporting Phase [Sample Free Realms Mini-Game Report]

A Usability Postmortem What We Tested What We Learned from It

A Usability Postmortem Vivox Voice Chat Implementation [Embedded Video] Lessons Learned ISSUE: Spatial constraints became apparent. Candidates were talking to one another RESOLUTION: Volume in the headphones was increased ISSUE: Multiple stakeholders driving different agendas RESOLUTION: Communication conduit ensuring similar approach to static system across multiple platforms

A Usability Postmortem Free Realms [Embedded Video]

A Usability Postmortem Difference Between Testing Children and Testing Adults Ø Conducted research prior to beginning testing Ø External focus groups – observing and studying methods Ø Reviewed past GDC presentations about testing with children: ØCarla Engelbrecht, Teachers College, Columbia University Little Hands, Foul Moods, and Runny Noses: The Ups and Downs of Researching Games with Kids: GDC 2008

A Usability Postmortem Difference Between Testing Children and Testing Adults Ø Cognitive and physical limitations in children have been long ago mastered by adults* Ø Example: Free Realms Target Demographic 8 -14 ØChildren Aged 7 -12, Concrete Operational Stage ØUnderstand Time, Space, Another Persons Perspective ØStill rely on concrete objects and experiments to form ideas Ø Take off kid gloves *Carla Engelbrecht, Teachers College, Columbia University Little Hands, Foul Moods, and Runny Noses: The Ups and Downs of Researching Games with Kids: GDC 2008

A Usability Postmortem Lessons Learned ISSUE: Confusion among parties as to whom we were, and should, recruit, against target demographics RESOLUTION: Created User Profiles based upon 3 candidate profiles created in conjunction with Marketing and Development. [Embedded User Profile]

A Usability Postmortem [Embedded Video]

A Usability Postmortem Lessons Learned ISSUE: Build validation prior to testing RESOLUTION: Point of contact between studios, 3 rd party client, and technicians ISSUE: Geographic challenges RESOLUTION: Chat feature for remote users to speak with San Diego office, Remote viewing, Remote logging Result = Active participation from satellite studios

A Usability Postmortem Conclusion Greg Spence, Programmer EQ 2: “Obvious benefit was ability to identify issues early on that we wouldn’t have caught until we went live. Helped to see where players reacted and stumbled through feature sets. ” Clint Worley, Producer EQ: “Obvious that the way we were presenting it (Vivox) wasn’t compelling. We were looking at the trees instead of the forest. ” Ryan Wells, Assoc. Producer Free Realms: “It was great to be able to make tweaks each week, make predictions about how users would respond and see the results. Our assumptions were often wrong and we learned just how different our audience was from ourselves. ”