Platform for real-time subjective assessment of interactive multimedia

Platform for real-time subjective assessment of interactive multimedia applications Bert Vankeirsbilck Bert. Vankeirsbilck@intec. ugent. be www. ibcn. intec. ugent. be Internet Based Communication Networks and Services (IBCN) Department of Information Technology (INTEC) Ghent University - i. Minds Department of Information Technology – Internet Based Communication Networks and Services (IBCN)

Rahul’s proposal Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 2

Context: interactive app Qo. E n Audiovisual Qo. E assessment focus: l l l Non-interactive applications Replay multiple distortions of prerecorded fragment Qo. E = F(codec parameters, visual impairments) [QP, freezes, frame loss, P/B frame prediction errors] n Interactive application Qo. E focus: l l l Games or low interactivity (telephony, video conf. ) Ad-hoc test platform Qo. E = F(network parameters, indirect metrics) [latency, packet-loss, jitter] [death/kill ratio, usage times, departure rates, task execution time] Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 3

Platform for … n Characteristics of our platform: l l Assessment of broad range of interactive applications (thin client remote desktop, games, remotely rendered engineering applications) Dynamic reconfiguration of media encoding and decoding Subjects interact with the media in real-time, and parameters can be adapted on-the-fly Assessment of algorithms that change media parameters to bandwidth and application characteristics Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 4

Presentation index n Platform architecture and performance l l l Configuration of network conditions On-the-fly adaptation of media encoder parameters Modular design Real-time user interaction Experiment replay n Example scenario subjective assessment results Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 5

Architecture l l l l l Real-time media codec reconfiguration Codec support Network impairment provisioning Real-time monitoring Remote control Hardware acceleration Session recording Subjective assessment method support Extensibility

Current limitations n Mouse movement issues l l Unable to assess shooter games (popular topic) Our game was car racing keyboard control n Focus entirely on video l l l No audio codec integration No immediate intention or adequate experience to support audio No possibility to test effect of A/V synchronization n Limits of framerate vs resolution l l gamers expect 60 fps Full. HD – 30 fps Full. HD = hard Hardware encoding/decoding could help? Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 7

Performance n System latency n Achieved frame rate n Test setup H 264 x 264 0. 120. x ultrafast profile, zero latency tuning ffmpeg version 0. 9. 1. git-1869237 Open. GL Virtual. GL 2. 3 Server Intel Core i 7 CPU 920@2. 67 Ghz 6 GB RAM Graphics Card NVidia Ge. Force GTS 250, 128 CUDA cores Server and viewer colocated on same machine Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 8

End-to-end system latency Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 9

Interception points Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 10

Latency Breakdown 640 x 480 1920 x 1080 Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 11

Achieved Frame Rate Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 12

Example scenarios n Use cases l l l Automated gameplay assessment Administered subjective interactivity with network impairment Test subject finds subjective optimal parameter constellation given various impairment settings Collaborative applications Defining novel objective quality metrics Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 13

Example scenarios n Use cases l l l Actually performed Automated gameplay assessment Administered subjective interactivity with network impairment Test subject finds subjective optimal parameter constellation given various impairment settings Collaborative applications Defining novel objective quality metrics Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 14

Example scenarios n Use cases l Automated gameplay assessment w w w l l Actually performed Race game (Vdrift Open Source racing simulator) ~20 test subjects, screen resolution 1024 x 768, 3 min free racing, 2 iterations training, 40 s per iteration Focus on frame rate versus still-image quality influence on games Administered subjective interactivity with network impairment Test subject finds subjective optimal parameter constellation given various impairment settings Collaborative applications Defining novel objective quality metrics Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 15

Assessment method n Compression modeling l l Identical situation compression Automated key-input for making ‘donuts’ n Choosing sample set of parameters l l Total assessment time = F(sample set size) Relevant part pre-assessment selection n Rating method and scale Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 16

Assessment method n Compression modeling n Choosing sample set of parameters n Rating method and scale l l Play game for 40 seconds Overlay to rate quality: w w w l l How well were you able to play the game, irrespective of visual quality or perceived artefacts? Only 1 question, always the same for each iteration Total testing time: about 30 minutes training incl. Rate along [1 – 5] scale Answer registered immediately back in the game Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 17

Compression model Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 18

Assessment method Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 19

Assessment method Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 20

Assessment method Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 21

Assessment method Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 22

MOS score results Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 23

MOS model Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 24

Real-time adaptation algorithm Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 25

Naive Qo. E model Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 26

Cross-referenced Qo. E model Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 27

Better decision support Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 28

Real-time adaptation algorithm Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 29

Real-time adaptation algorithm Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 30

Algorithm comparison Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 31

Algorithm comparison Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 32

Discussion n Games as subset of interactive applications n Context l Task: drive best time, least accidents, follow optimal lines. . w l My test: no task specified. Just race the track, no use in performing a task as user is interrupted for scoring Qo. E Taking a corner != straight section w 40 s iteration: will have corners n Frequent switching expected to be experienced as annoying Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 33

Discussion n Using models: + just order the compression alternatives according to the Qo. E model to find the best option for a situation - how generically applicable are the models - Different applications Different users with e. g. gaming experience Context: leisure game, professional 3 D modeling application Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 34

Related publications l l Vankeirsbilck, B. et al. (2013). Platform for real-time subjective assessment of interactive multimedia applications. Multimedia Tools and Applications, 1 -27. Vankeirsbilck, B. et al (2013). Quality of experience driven control of interactive media stream parameters. In Proc. Integrated Network Management (IM 2013), 2013 IFIP/IEEE International Symposium on (pp. 12821287). IEEE. Department of Information Technology – Internet Based Communication Networks and Services (IBCN) 35

Questions ? Bert Vankeirsbilck Bert. Vankeirsbilck@intec. ugent. be www. ibcn. intec. ugent. be Internet Based Communication Networks and Services (IBCN) Department of Information Technology (INTEC) Ghent University - i. Minds Department of Information Technology – Internet Based Communication Networks and Services (IBCN)