Quality of Service and Quality of Experience Correlations

Quality of Service and Quality of Experience Correlations in a Location -Based Mobile Multiplayer Role. Playing Game Maarten Wijnants 1, Wouter Vanmontfort 1, Jeroen Dierckx 1, Peter Quax 1, Wim Lamotte 1, Katrien De Moor 2, Jeroen Vanattenhoven 3 1 Hasselt University – Expertise Centre for Digital Media (EDM) - IBBT {firstname. lastname}@uhasselt. be 2 Ghent 3 KULeuven University – Media and ICT (MICT) - IBBT Katrien. R. De. Moor@Ugent. be - Centre for User Experience Research (CUO) - IBBT jeroen. vanattenhoven@soc. kuleuven. be

Outline • Introduction and Motivation • Research Context – Location-Based Role-Playing Game • Investigated Technological Factors • User Research – Practical Approach and Participant Demography – Test Procedure and Data Collection Methodology – Quantitative and Qualitative Findings • Conclusions 06/10/2011 ICEC 2011 2

Introduction and Motivation • Quality of Service (Qo. S) – Well-established concept in software and system engineering – Measure of technological performance • E. g. , performance guarantees in computer network – Technical dimensions Objectively measurable • Quality of Experience (Qo. E) – Fairly novel active research topic – Semantic variant of Qo. S • Denotes overall experience or satisfaction of end-user • Humane, multi-disciplinary and highly subjective metric and therefore hard to quantify objectively 06/10/2011 ICEC 2011 3

Introduction and Motivation • Qo. S and Qo. E are irrefutably related – Strong attention for technological factors remains explicit in user research • However, exact connection to date remains largely uninvestigated and therefore unclear – Such information is highly valuable for developers of (distributed) applications • Decent understanding of their mutual impact and interaction likely enables user experience optimization • Important asset in Customer Relationship Management • This paper unveils Qo. S/Qo. E correlations via a hybrid user research methodology 06/10/2011 ICEC 2011 4

Research Context • The field of mobile gaming was selected as context to conduct our research – Increasingly popular application domain • Focus on distributed multiplayer games – Network operation and performance might substantially influence user experience • Further scope reduction by concentrating on (outdoor) location-based games – Revolve around location determination: Position of players/objects in the physical world is tracked and plays prominent role in gameplay 06/10/2011 ICEC 2011 5

Research Context Location-Based Role-Playing Game • In-house developed mobile location-based multiplayer game with WW I setting • Played outdoors between two rivaling teams (Allied forces and German army) • Location-based game – Players walk around in the physical world while carrying a handheld device (netbook) – Player location tracking via GPS receiver • Data dissemination via 3 G connection • Game presents satellite view of the outdoor environment 06/10/2011 ICEC 2011 6

Research Context Location-Based Role-Playing Game • Interaction occurs by “scanning” QR tags – Each player carries (unique) QR code – Codes in playing area represent virtual items • Role-Playing Game (RPG) – Profile determines actions player can undertake, his objectives and type of collectable virtual items • Commander: Remains at a fixed indoor location; has a global overview of his team’s actions, permanently sees discovered objects and can instruct players on the field • Soldier: Collect weapons; engage enemies • Spy: Collect intelligence items; reveal enemy players • Medic: Collect medical supplies; heal injured teammates 06/10/2011 ICEC 2011 7

Research Context Location-Based Role-Playing Game 06/10/2011 ICEC 2011 8

Investigated Technological Factors • 3 technical/Qo. S-related parameters studied for their impact on player Qo. E – Each parameter acted as separate test condition • Parameter 1: Player Location Determination Precision – Investigate impact of imprecisions on player Qo. E – Position of players artificially and randomly varied in 10 meter radius around actual location • Parameter 2: Player Location Refresh Rate – Important design variable for location-based apps – Refresh player locations only per 6 seconds 06/10/2011 ICEC 2011 9

Investigated Technological Factors • Parameter 3: Voice communication delay – Communication is affected by network delay – ITU recommends 1 -way latency < 400 ms – 2 seconds of additional artificial delay • Test conditions are representative, but the used magnitudes might be exaggerated – To elicit pronounced and unambiguous reactions from test subjects 06/10/2011 ICEC 2011 10

User Research Practical Approach and Participant Demography • 4 separate user studies with 8 subjects each – Participants evenly divided in 2 teams – Each participant assumed different role (random) – Deliberately not informed about Qo. S manipulation • Test population demography – 12. 5 % was female; average 29 (SD = 9. 04) – Over 80 % had played video game in past month – Professional occupation: mostly students (40 %) and employees (50 %) • 7000 m 2 science park as playing area – 25 virtual items (QR codes) strategically dispersed 06/10/2011 ICEC 2011 11

User Research Test Procedure and Data Collection Methodology • Test session of approximately 2 hours, logically enveloping 3 consecutive phases • Pre-usage phase – Game objectives and mechanics verbally outlined – Paper-based questionnaire • Basic socio-demographical characteristics • Previous digital gaming experiences and expectations • During-usage phase – 40 minute duration; 4 intervals of equal length • Three intervals exclusively devoted to 1 test condition – Investigate influence of corresponding Qo. S parameter • 1 interval with “perfect” gameplay (reference scenario) 06/10/2011 ICEC 2011 12

User Research Test Procedure and Data Collection Methodology • In course of play session, 4 identical in-game surveys needed to be electronically filled in – At end of each conceptual interval of the session – Acquire detailed Qo. E data in each test condition • In-situ data sampling at the very moment of experience • Reduces risk of recall bias effect Accurate assessments • 11 statements measured on 5 -point Likert scale • Post-gaming phase – Second paper-based questionnaire • Inquiries about overall experiences, feelings, thoughts – Participant group conversation (focus group) • Share experiences, reflect on (non-)reported Qo. E factors 06/10/2011 ICEC 2011 13

User Research Quantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Global characteristics of non-instrumental dimensions • Users felt least ‘absorbed’ in role in interval 1; feeling monotonously increased as game progressed (same for ‘fun’) – Users needed to get used to the game? • Expectations adjusted in course of the gaming session – Pre-usage expectations were high; adapted according to context 06/10/2011 ICEC 2011 14

User Research Quantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Player location distortion test condition • Accuracy of location-related information concerning other players as well as objects was rated lowest • Influenced users’ perception of location update rate (both players and objects) Unjust! 06/10/2011 ICEC 2011 15

User Research Quantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Voice delay test condition • Participants felt most ‘frustrated’ + most ‘effort’ required • High rating for ‘experiencing technical problems in the communication with team members’ + lowest average score for ‘I can communicate smoothly with my team members’ 06/10/2011 ICEC 2011 16

User Research Quantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Normal gameplay test condition (no Qo. S tampering) • Highest degree of ‘absorption’ in the played role + maximal score for ‘fun’ • Highest rating for accuracy of player location information + for position refresh rate of both players and objects 06/10/2011 ICEC 2011 17

User Research Quantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Limited player location refresh test condition • Minimal ‘frustration’ and required ‘effort’ + high score for ‘fun’ • Technical manipulation no notable influence on Qo. E – Claim is supported by qualitative data from the post-gaming stage – Players were near the end rather centrally clustered (entire playing area already discovered) less reliance on the data displayed on their device? 06/10/2011 ICEC 2011 18

User Research Quantitative and Qualitative Findings • During-usage study succeeded in relating modifications in technical circumstances to the users’ (subjective) experiences • Example results from post-usage stage – Qo. S manipulation in 1 st and 2 nd playing condition were experienced as sources of frustration • Consistent with findings from in-game surveys – Quotes from focus groups • “The GPS update rate wasn't really a problem; most of the time they weren't running, so you could easily track the other players” • “As Commander I expect to know the precise location of my team; the faulty positions made this impossible” 06/10/2011 ICEC 2011 19

User Research Quantitative and Qualitative Findings Mean scores for in-game survey statements per player role Did player role influence Qo. E? • Statistically significant differences between roles for 5 aspects – – – ‘Absorption’: Soldiers felt notably more immersed than Commanders ‘Fun’: Soldiers had more pleasure than the Medic and Commander roles ‘Effort’: Significantly higher for Soldier profile (esp versus Commander) ‘Expectations’: Opposite of ‘effort’ (Commander > Soldier) Player location update rate: Clearly lowest for Commander (cfr post-usage) 06/10/2011 ICEC 2011 20

Conclusions • Two high-level conclusions – Modifications in 3 specific technical parameters held implications for quality of users’ experiences • E. g. , test condition without Qo. S impairments had highest average scores for ‘absorption’ and ‘fun’ – For some sub-dimensions, player reported Qo. E diverged depending on assumed role • E. g. , maximal ‘absorption’ and ‘fun’ for Soldier profile – Complementary qualitative and quantitative data 06/10/2011 ICEC 2011 21

Conclusions • Lower-level, specific findings illustrate extent of impact of Qo. S performance on Qo. E – E. g. , voice communication latency complicated coordination of players’ actions • Low ‘absorption’ + highest mean ‘frustration’ and ‘effort’ – Valuable information for software developers • How best to cope with Qo. S irregularities? • ‘Absorption’, ‘fun’ monotonously improved while playing + users adjusted expectations – Qo. E not merely determined by technical params • Qo. E exceeds Qo. S in terms of scope; importance and potential influence of non-instrumental aspects on Qo. E 06/10/2011 ICEC 2011 22

Questions 06/10/2011 ICEC 2011 23

User Research Quantitative and Qualitative Findings • Findings from pre-usage surveys – Covered (among other things) same 11 Qo. Erelated statements as the in-game surveys – Pre-usage responses indicated that test subjects attached most importance to • “Having fun” (Mean = 4. 58) • “Not having technical problems while communicating with team members” (M = 4. 42) • “Being able to communicate smoothly with team members” (M = 4. 23) – Analysis of post-usage data resulted in the same top 3 list 06/10/2011 ICEC 2011 24

In-Game Questionnaire 1. I feel totally absorbed in my role in the game 2. With the information that I am receiving about the position of the other players, I am able to localize them without a problem 3. I am having fun while playing the game 4. I have to put a lot of effort in the game 5. The kind of information that I am getting about the location of objects, allows me to localize these objects without a problem 6. I expected more from the game 7. While communicating with team members I experienced technical problems 8. I feel frustrated 9. The location of the other players is updated sufficiently 10. I am able to communicate smoothly with my team members 11. The location of objects is updated sufficiently 06/10/2011 ICEC 2011 25