Choreographer-mathematician collaboration developing machine segmentation techniques for motion

Choreographer-mathematician collaboration developing machine segmentation techniques for motion capture analysis of dance

Associate Professor Kim Vincs Dr Wai Kuan Yip Dr Vicky Mak Ms Kim Barbour

Motion capture & dance analysis v. Motion capture offers precision and true 3 D tracking of dance movement v. Recent development over the last 10 years v. Relatively few motion capture labs with dance focus v. A new gold standard for quantitative dance analysis?

Challenges in motion capture dance analysis v. Analysis needs to be culturally and artistically relevant (de. Lahunta) v. Pattern recognition in dance crosses multiple frames of reference both between and within genres (? ref) v. Approach to capture protocol, marker set design and feature extraction need to be appropriate to the needs of end-users, i. e. dance artists (Norman)

‘Capturing Dance’ project at Deakin Motion. Lab v. Aims to isolate features of motion capture data that are artistically useful to choreographers v. Features may be different for different dance genres and different dance artists v. Will work with artists across 3 dance genres; contemporary dance, ballet and Australian indigenous dance over 3 years

Project team Associate Professor Kim Vincs Dr Vick Mak Associate Professor Richard Smith Dr Wai Kuan Yip Ms Kim Barbour Mr Daniel Skovli Mr Peter Driver Ms Lisa Bolte Ms Carlee Mellow Ms Phoebe Robinson Ms Mee Young Yuk

This paper v. Analysis of a single, complex contemporary dance phrase v 20 repeats of the phrase by a single dancer v 64 marker-set adapted to capture spine-arm-foot relationships important to dance v. Approaches to segmentation into dance-meaningful chunks for analysis v. Necessity of choreographer-mathematician collaboration in developing appropriate analysis techniques

Pattern recognition process

Segmentation v. Manual segmentation (human/expert) v. Unsupervised machine segmentation based on kinematic assumptions v. Supervised machine learning v. Unsupervised machine learning

Challenges in segmenting dance v. Dance phrases tend to overlap v. Different body segments need different levels of smoothing v. Both large and small movements may be artistically significant v. Different combinations of body segments may initiate new movements v. Dance artists themselves don’t agree on how phrases are segmented (de. Lahunta 2005)

Some previous approaches to segmenting dance movement v. Inter-limb correlation (Nakata 2007) v. Laban Movement Analysis, (Bouchard & Badler 2007) v. Minimum velocity (Hachimura & Nakamura 2001) v. Triple minima; force, kinetic energy & momentum (Kahol, Tripathi & Panchanathan 2004)

Preliminary data – manual segmentation of a ‘tendu’ v. Expert segmentation problematic as depends on weighting conflicting factors v. Surprising variability within samples (5 samples, elite ballet dancer) v. Highlights difference between dancers’ conceptual map of the movement and the detailed ‘motion capture view’

Cortex file – tendu segment

Comparison of segmentation methods

Our approach v. Primary question is what segmentation is artistically meaningful? v. We used a collaborative, practice-based approach to develop segmentation ‘schema’ for the phrase v. Choreographer, 2 dancers and 2 mathematicians developed definition of segments based on kinematics v. Aimed to create an automated system based on the artistic schema

Problems identified v. Dancers used multiple conceptual frameworks, e. g. velocities, heights, correlations of body parts v. Dancers’ framework is ‘procedural’ – segment definitions only make sense in relation to the preceding movement v. Segmentation is phrase-specific and cannot be generalized

Our schema

Segmentation steps v. Conversion from positional information to hierarchical (parent-child) translation and rotation format to make analysis scale & position invariant v. Smoothing to remove noise – Butterworth 6 Hz, moving average 20 -50 window, Gaussian smoothing

Segmentation steps v. Polygonal approximation to estimate gradients of consecutive points to find local minima v 3 possible scenarios; Negative gradient Zero-gradient Positive gradient positive gradient zero gradient

Examples of polygonal approximation Important to select the right threshold to estimate local minima Above: Maximum tolerable threshold 5*SD (just nice) Above Right: 15*SD (too loose) Right: 30*SD (way too loose)

Local minimum segmentation Root location Absolute rotational velocity Right elbow location Absolute rotational velocity

Local minimum segmentation Example of compliant sample (#11) Example of non-compliant sample (#2)

Local minimum segmentation Example of compliant sample (#17) Example of non-compliant sample (#12)

Accuracy

Comparison of methods

Insert cortex file of Carlee Bitter

Summary v. Method works very well for movements that rely on body-shape change, but breaks down for large loco-motor and turning movements v. Large variation in ‘dancer compliance’ v. Variation or ‘dancer compliance’ does not correlate with dancer/choreographer ratings – variation in inherent in the style

Conclusions v. Dancer-mathematician collaboration useful for identifying recognition problems that need to be solved v. Dance recognition process needs to be able to deal with a high level of intra-performer variability v. Different types of movement may require different approaches to segmentation v. Further techniques for optimizing machine segmentation (eliminating false segmentation points) needed to supplement dance-generated phrase descriptions

References v. Bouchard, D & Badler, N 2007, 'Semantic Segmentation of Motion Capture Using Laban Movement Analysis', in C. Pelachaud et al. (ed. ), Intelligent Virtual Agents, Springer-Verlag, Berlin Heidelberg, vol. 4722, pp. 37 -44. v. De. Lahunta, S & Barnard, P 2005, 'What's in a Phrase? ', in J Birringer & J Fenger (eds), Tanz im Kopf: Jarbuch 15 der Gesellschaft für Tanzforschung, LIT Verlag, Münster. v. Hachimura, K & Nakamura, M 2001, 'Method of generating coded description of human body motion from motion-captured data', Robot and Human Interactive Communication, 2001. Proceedings. 10 th IEEE International Workshop on, pp. 122 -7. v. Kahol, K, Tripathi, P & Panchanathan, S 2004, 'Automated gesture segmentation from dance sequences', Proceedings, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 17 -19 May pp. 883 -8. v. Nakata 2007, 'Temporal Segmentation and Recognition of Body Motion Data Based on Inter-Limb Correlation Analysis', Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2007), Oct. 29 2007 -Nov. 2 2007 pp. 1383 - 8 v. Norman, SJ 2006, 'Generic Versus Idiosycratic Expression in Live Performance Using Digital Tools', Performance Research, vol. 11, no. 4, pp. 23 -9.