Cross-modal perception of motionbased visual-haptic stimuli Ian Oakley

Cross-modal perception of motionbased visual-haptic stimuli Ian Oakley & Sile O’Modhrain Palpable Machines Research Group http: //www. mle. ie/palpable ian@mle. ie sile@media. mit. edu Media Lab Europe Sugar House Lane Bellevue Dublin Ireland

Overview • • • Background – Touch in Broadcast Media The Touch TV Project Role of cross modal perception in TTV Experimental Study - Pilot Conclusions and Future Work

Touch in Broadcast Media • Adding haptic information to a broadcast may enhance viewing • Motivation § Consumer demand for richer media • Led to advances in audio and video technologies • Touch may provide greater richness • Latest advances not enhanced viewing experience § e. g. digital and interactive TV § Evidence supporting this assertion • Touch in virtual environments linked to increased ratings of presence and co-presence • Adoption of touch in gaming and simulator systems

The Touch TV Project • Ongoing work investigating integration of touch into broadcast content § Currently, focused on a practical and theoretical exploration of the design space available for this work § Involves a number of research threads ranging from authoring of media to psychological study

Cross-Modal Perception in TTV • Touch TV inherently cross-modal § Involves construction of immersive and involving media with graphics, audio and touch components § Techniques for combining graphics and audio established in this context • E. g. visual capture § How to add touch to this format is an open question • How do we add haptic information to an audio-visual stream ensuring that it will be interpreted as we intend?

Experimental Study • Interested in combination of visual and haptic stimuli representing movement in complex graphical scenes § Relatively complex stimuli • Initial questions: § § Can haptic cues be associated with visual movement? What is the effect of visual distractors? What is the effect of visual target position? What kind of haptic cues should we use? • Attempted to address these questions in a pilot study

Experimental Study • In each trial, the visual display consisted of a number of bouncing balls controlled by a simple mathematical model • Each ball had a different spring constant – different “bounciness” and was subject to gravity • Force displayed on a PHANTo. M haptic interface either represented the velocity of one of the displayed balls, or of an unseen ball • Participant’s task was to determine whether or not what they could feel related to one of the displayed balls

Experimental Study • Varied: § § Number of distractors (5 possibilities) Target spring constant (6 possibilities) Target position (5 possibilities) Haptic-Visual match (2 possibilities) • Total of 300 trials presented in random order • Practice consisted of a random selection of 50 trials • 10 Participants § All employees at Media Lab Europe § 6 male, 4 female, all right-handed, none reported tactile or visual impairment, range of familiarity with haptic devices (2. 5 on a scale from 0 -5) • Participants wore noise cancelling headphones • Were able to rest between each trial

Experimental Study • Gathered trial time, error rate • Trial time for visual presence/absence of haptic stimuli: • T-test revealed no significant differences

Experimental Study • Further analysis restricted to trials in which haptic ball was visually present • Used ANOVA, post-hoc t-tests with Bonferroni CI adjustments • Did not include trials in which subject made errors in temporal analysis § However, relatively large error rate § Collapsed position factor for temporal analysis • Used a 5 (distractors) x 6 (spring values) • No interactions found (F=0. 399, p=0. 991) • Error data 5 (distractors) x 6 (spring values) x 5 (positions) § One interaction: Distractors by Spring Value (F=1. 974, p<0. 01)

Results • Trial time for differing numbers of visual distractors • ANOVA: F=8. 235, p<0. 001 No of Distractors 0 1 2 3 4 0 X Not Sig P<0. 01 1 X x Not Sig P<0. 05 P<0. 01 2 X X X Not Sig P<0. 05 3 X X Not Sig 4 X X x X X Results of post-hoc t-tests

Results • Trial time for differing spring values • ANOVA: F=83. 881, p<0. 001 Spring Values 0. 15 0. 25 0. 35 0. 4 0. 15 X Not Sig P<=0. 01 0. 2 X X P<=0. 01 0. 25 X X X P<=0. 01 0. 3 X X P<=0. 01 0. 35 X X X Not Sig 0. 4 X X X Results of post-hoc t-tests

Results • Error rate for differing numbers of distractors • ANOVA: F=20. 614, p<0. 001 No of Distractors 0 1 2 3 4 0 X P<0. 05 P<0. 01 1 X x Not Sig P<0. 05 P<0. 01 2 X X X Not Sig P<0. 01 3 X X Not Sig 4 X X x X X Results of post-hoc t-tests

Results • Error rate for differing spring values • ANOVA: F=4. 861, p<0. 001 • Post-hoc t-tests revealed 0. 15 spring value led to significantly higher error rate than 0. 4 spring value

Results • Error rate for target positions • ANOVA: F=0. 399, p<0. 05 • Post-hoc t-tests did not attain significance

Interpretation • Subjects merged visual-haptic motion stimuli § Error rate always less than chance • Increase in trial time and error rate with increased numbers of distractors indicates serial search § Haptic target did not “pop out” as in parallel search • Equality of present and absent results suggests subjects performed an exhaustive search • Possible speed/accuracy trade off § Errors decreased, time increased with larger spring values • Eccentricity effect in position error data suggests subjects focused on the centre of the screen

Conclusions and Future Work • Study successfully informed basic questions relevant for our TTV domain § Relevant to our authoring process • Many questions remain § Contrast same scenario with visual/audio and visual/audio/haptic stimuli § Role of visually distinct objects § Is mapping force to motion the best solution? • Basic scaling research on force presentation in this domain § …

Cross-modal perception of motion based visual-haptic stimuli Ian Oakley & Sile O’Modhrain Palpable Machines Research Group http: //www. mle. ie/palpable sile@media. mit. edu ian@mle. ie Media Lab Europe Sugar House Lane Bellevue Dublin Ireland Euro. Haptics 2003 6 th-9 th July www. mle. ie/palpable/eurohaptics 2003