Computer Animation Where we are overview Where we

Computer Animation Where we are (overview) Where we are going (perhaps)

Animation overview Computer Animation Popular perception - CGI is animation (full length animations, CGI effects films or computer games).

Animation overview Off-line/pre-recorded Animation is expensive Production ´effort´ same as handmade animation The Fox and the Hound Time 4 years Frames 110, 000 Time 2. 9 hours/frame Paint 450 gallons Toy Story (1) 4 years (1. 5 story + 2. 5 production) 110, 064 45 mins-24 hrs/frame 110 SUNs

Animation overview ‘Real-time’ Computer Animation in Games Animation control (script) in games is: • pre-recorded (MOCAP) or predesigned (currently the de facto standard in games) • calculated in real time (IK and dynamics) • a mix of pre-recorded and real time

Animation overview MOCAP in Games is select and blend Game events Mo. Cap 1 Animate skeleton skin Render Mo. Cap 2 Mo. Cap n Mo. Cap X For example a football game will have 200 -300 sequences. blend Mo. Cap Y

Animation overview Script creation methods Recording real motion (MOCAP) [1 st=] By ´hand´ using proprietary or in-house software, keyframe animation [1 st=] Posing real motion using a digital input device (DID) (film special effects) Executing dynamic equations (scientific visualisation, computer games) Behaviour models (film special effects)

Animation overview Script creation -Motion quality-best is MOCAP Animación (portero)

Animation overview Script creation -Motion quality-best is MOCAP z x Hombro = 3 DOFs y

Animation overview Script creation -Motion quality-best is MOCAP Applying MOCAP to a skeleton y x z

Animation overview Motion Capture – quality motion is always perceivable as such (even with stick figures)

Animation overview MOCAP-bones-skinning is a well-established technology

Animation overview Script creation By ´hand´ using proprietary or in-house software. The most popular method is keyframe animation.

Animation overview Real time dynamics Executing dynamic equations (computer games, scientific visualisation) Flong = Ftraction+ Fdrag a = F/m v = v + dt*a p = p + dt*v

Animation overview Script creation methods High level behavioural models – original was “flocking”

Animation overview Script creation methods Posing real motion - stop motion animation was used in Jurassic Park (Dinosaur Input Device) to script the computer models

Animation overview Animation in Science Zajac 1966 Bell Telephone Lab First Computer animation in science

Animation overview Animation in Science Max Born 1935

Animation overview Animation in Science Max Born 1935 The Restless Universe

Animation overview Animation in Science Muscle Fibres of the heart

Animation overview Forensic Animation - ethics?

Animation overview Forensic Animation – ethics? Technology blesses the production with veracity? Who controls the content of simulation? How can the accuracy be guaranteed? No cross examination possible

Animation overview Synthetic vision Provides a synthetic view of reality, constructed from a database, which cannot be seen because of, for example, weather conditions. The best example is civil aviation. Principles used are exactlty the same as games where a view frustum is ‘driven’ through an environment under user control.

Animation overview Synthetic vision in civil aviation Cockpit view

Animation overview Synthetic vision in civil aviation Landing display

Animation overview Synthetic vision in civil aviation Uses as database Shuttle Radar Topography Mission (SRTM) Wide Area Augmentation System (WASS) Local Area Augmentation System (LASS) Derives 3 D position (Accuracy < 1 m) from GPS + INS On-board sensors (such as RADAR altimeters)

Animation overview Synthetic vision in civil aviation Animation of an approach

Animation overview Where we are • Off-line -manual • Combining off-line + event driven • Event driven – dynamic simulations – walk throughs

Animation overview The future ? Whats wrong with MOCAP • Although pre-reorded aninimation is of high quality, it is inherently limited – the more complex the game the more clips are required. • Cannot MOCAP animals. • MOCAP transitions – blending is unsatifactory What we would like • Increase the quality of real-time animation and obtain any motion in real time accoording to the ‘action demand’ – event driven • Speech/emotion expression needs to be event driven

Event driven animation for humanoids What we have now – event driven recorded animation Game events Mo. Cap 1 Animate skeleton skin Render Mo. Cap 2 Mo. Cap n Mo. Cap X This model can only react to completely predetermined actions blend Mo. Cap Y

Event driven animation for humanoids What we have now- MOCAP – more general One generic motion fits all characters

Event driven animation for humanoids Why do we need it? Important element in an anthropomorphic interface camera computer vision speech recogn. NLP visual speech text generatn. expressn emotion generatn. query system game

Event driven animation for humanoids What do we aim for • Seems sensible to retain MOCAP technology – high quality, well established so increase its flexibility - adaptation • BUT oranges are not the only fruit. Can we generate animation in real time.

Event driven animation for humanoids Examples • Using IK adapted MOCAP in human motion • ‘Total’ IK solution for human motion • Using MOCAP in visual speech • ‘total’ solution for visual speech

Event driven animation for humanoids Character adaptation not straight forward Change scale joint angles change in non-linear manner From Shin et al 2001

Event driven animation for humanoids Cheating for real-time Use v. simple skeleton and complex skin. C. G skeletons – 50 DOFs human skeletons - >250 DOFs Motion from skeleton, visual complexity from skin

Event driven animation for humanoids MOCAP is forward kinematics Motion of end effector X = f( MOCAP = f( ) )

Event driven animation for humanoids Inverse Kinematics – an old idea x = f ( ) joint space Forward Kinematics = f-1 (x) Inverse Kinematics Circa 1985 use for complete soln. use to adapt MOCAP Cartesian space x

Event driven animation for humanoids Inverse Kinematics – solutions • Geometric/Analytical: This class of solvers generate a solution in a single step for a given goal and therefore fast. They can be used as part of a solution in a hybrid method. • Differential Algorithms: The task is transformed into a linear problem based on small changes using the Jacobian and iteratively refining the system to meet the goal position. • Cyclic Co-ordinate Descent: An algorithm which again moves towards a solution in small steps. This time, however, the steps are formed heuristically. • Hybrid Methods: Uses a combination of approaches. Their motivation is usually real-time performance.

Event driven animation for humanoids Differential IK – the Jacobian The Jacobian is the multidimensional extension to differentiation of a single variable. Given a function: X = f( ) where X is of dimension n and of dimension m, the Jacobian J is the n x m matrix of partial derivatives relating differential changes of , to differential changes in X, written as: d. X = J( )d d = J-1( )d. X where the (i, j)th element of J is given by: Jij = fi/ j

Event driven animation for humanoids Inverse Kinematics – an old idea Differential IK - iteration 1. 2. 3. 4. 5. 6. Calculate the incremental step X = Xgoal – X Calculate the Jacobian matrix using the current joint angles Calculate the inverse of the Jacobian – using righthand generalised inverse if required; J-1 = JT(JJT)-1 Check for iterative convergence – i. e. make sure the Jacobian inverse is suitably accurate (a) If ||(I – JJ-1)|| > e, reduce X= X/2 and repeat 4 (where e is a convergence threshold) (b) If ||(I – JJ-1)|| > e after a number of steps then the goal is likely out of reach so terminate Calculate the updated values for the joint angles where = J-1 X Using forward kinematics to determine whether the solution is close enough to the goal. If the solution is adequate then terminate iteration else go back to step 1 (as step 4 could have reduced X).

Event driven animation for humanoids Inverse Kinematics – an old idea Differential IK – example Jacobian Determining the Jacobian Consider:

Event driven animation for humanoids Inverse Kinematics – an old idea Differential IK – example Jacobian where

Event driven animation for humanoids Differential IK – the Jacobian • For large articulations the complexity of analytically expressing the differentiation is very tedious. • The Jacobian can be viewed as expressing the velocity of the end of the chain in terms of local angular velocities with respect to a base frame. • This information is easily extracted from transformation matrices that already exist in the graphics pipeline – i. e. the matrix concatenation of child-parent relationships as the articulation is built up. • When the Jacobian is not square (whenever the number of DOFs in the chain increase past the dimension of the end-effector), a pseudoinverse is required, which could lead to numerical error. • Singularities – a decrease in the rank of the Jacobian can result in the loss of a degree of freedom that usually happens when the chain is fully extended

Event driven animation for humanoids Differential IK- main problem • Underdetermined System: The purpose of Inverse kinematics is to produce a set of joint angles that allows an end-effector to be positioned in a given location. This is an underdetermined system therefore many solutions exist. • •

Event driven animation for humanoids Differential IK- joint constraints • Removal of redundant DOFs from the Jacobian • Angular Constraints – Modification of step 5 of the iterative algorithm to include boundary constraints on specified DOFs = lower bound if J-1 P < lower bound = upper bound if J-1 P > upper bound = J-1 P otherwise

Event driven animation for humanoids Differential IK- demo Unconstrained IK chain Constrained IK chain 0 0 180 0 1 90 -30 2 30 -18 3 -18

Event driven animation for humanoids Differential IK- MOCAP adaptation Change scale joint angles change in non-linear manner

Event driven animation for humanoids Differential IK- MOCAP adaptation Retargetting by simply scaling Retargetting using IK constraints to maintain foot plants

Event driven animation for humanoids Differential IK Scaled Retargetting IK Retargetting to maintain foot plants

Event driven animation for humanoids Differential IK- total solution-walking Foot flight curve (a) (b) (a)

Event driven animation for humanoids Differential IK- main problem

Event driven animation for humanoids Facial Animation – two level model • Apply motion to ‘bones’ which control the skin • Motivation is identical – script applied to bones and bones control face vertices • No. of bones 2 -3 orders of magnitude less than face vertices

Event driven animation for humanoids Facial Animation – two level model - muscles

Event driven animation for humanoids MOCAP – concatenating text Blahblahblah ………………… phonemes visemes Muscle values Keys and interpolate

Event driven animation for humanoids Facial Animation – muscles –problem? Interpolating between static targets does NOT produce convincing mouth motion

Event driven animation for humanoids Facial Animation – two level - bones

Event driven animation for humanoids Facial Animation – two level - bones

Event driven animation for humanoids Facial Animation – bones –problem? Incapable of particularly subtle expressions and so unsuitable for expressive speech

Event driven animation for humanoids Facial Animation – visual speech MOCAP can be used to: 1) Cure subtlety problem 2) Implement ‘general domain’ speech by concatenation 3) BUT 4) How do we retarget? Face changes both scale and shape 5) How do we concatenate motion? – will conventional blending work?

Event driven animation for humanoids MOCAP – mesh control from sparse markers MOCAP markers SOFFD mesh from markers

Event driven animation for humanoids MOCAP – mesh control from sparse markers

2) Deform reference mesh to fit target mesh 1 1) Position markers on reference mesh to define a control surface - 66+7 for head motion MOCAP – retargetting 2 3 3) Retargetted control surface

Event driven animation for humanoids MOCAP – retargetting Marker motion speech

Event driven animation for humanoids MOCAP – retargetting

Event driven animation for humanoids MOCAP – retargetting

Event driven animation for humanoids MOCAP – for visual speech • Can use variable length fragments (sentences, words or syllables) • Overcomes the co-articulation problem • Conventional blending seems to work

Event driven animation for humanoids MOCAP – concatenating

Event driven animation for humanoids MOCAP – concatenating So is MOCAP speech the answer? NO Because: 1) The inherent quality advantage derives from using variable length units (sentences, phrases, words) and this would demand masses of data for general domain speech. 2) Expressive speech? E. g combine a smile with an utterance.

Event driven animation for humanoids Facial Animation – the return of static phonemes Phonemes/visemes as static a units are a good solution for general domain speech text Blahblahblah ………………… phonemes Can we do better than interpolation? visemes Muscle values Keys and interpolate

Event driven animation for humanoids Facial Animation – the return of static phonemes

Event driven animation for humanoids Facial Animation – constraint based global solution • Treat V as a point in 13 D space • Assign a weight/dominance to each V • For each unit (sentence…) find a global solution – a trajectory through this space • Solution does NOT interpolate the means exactly

Event driven animation for humanoids Facial Animation – the return of static phonemes

Event driven animation for humanoids Facial Animation – the return of static phonemes Decreasing the dominance of the 4 th segment reduces its effect over the entire trajectory

Acknowledgments/contacts Mocap/inverse kinematics m. meredith@dcs. shef. ac. uk Visual speech j. edge@dcs. shef. ac. uk