Physically Based Sound COMP 259 Nikunj Raghuvanshi

Overview l Background l FEM Simulation l Modal Synthesis (Foley. Automatic) l Comparison/Conclusions

Motivation l Sounds could in-principle be produced automatically, just like graphics: Sound Rendering l Sound Rendering has not received much research effort l Main Goal: Automatic generation of non -music, non-dialogue sound

Sound Production Today l Movies: Foley Artists http: //www. marblehead. net/foley/index. html l Games: Anyone noticed the huge sound directory in Unreal Tournament?

PBS: Sound Production in Nature l Collisions/Other interactions lead to surface vibrations l Vibrations create pressure waves in air l Pressure waves sensed by ear Vibration Surface Vibration Propagation Pressure Wave Perception Ear

Main Aims of PBS l Physics simulator gives contact/collision information l Assign material properties for sound, Wood, concrete, metal etc. l Sound simulator generates sound using this data (in real time? )

Challenges l Sound must be produced at a minimum of ~44, 000 Hz l Extremely High Temporal Resolution (timesteps in the range of 10 -6 -10 -8 s) l Stiffness of underlying systems (eg. Metallic sounds. K/m~=108) l Stability may require even smaller timesteps

Two Approaches l FEM deformable simulation O'Brien, J. F. et. al. , “Synthesizing Sounds from Physically Based Motion. ” SIGGRAPH 2001. l Foley. Automatic (Modal Synthesis) Kees van den Doel et. Al. , “Foley. Automatic: Physicallybased Sound Effects for Interactive Simulation and Animation. ” SIGGRAPH 2001.

Main ideas Deformable Simulation (arguably) much more “physically based” l Foley Automatic: Additive Synthesis l Component Sinusoids Sound Signal

Overview l Background l FEM Simulation l Modal Synthesis (Foley. Automatic) l Comparison/Conclusions

Simulation Requirements l Temporal Resolution l Simulate Vibration as well as Propagation l Vibration Modeling: Deformable Model for Objects l Propagation Modeling: Explicit Surface Representation l Physical/Perceptual Realism

System Structure

Vibration Modelling FEM with Tetrahedral Elements l Linear Basis Functions, green’s strain l Explicit Time Integration l Typically #nodes = 500, #elements = 1500, dt = 10 -6 -10 -7 s l

Sound Propagation Modelling Fluid Dynamic FEM simulation of surrounding air? Very expensive. Instead… l Employ Huygen’s Principle: Pressure Wave may be seen as sum of pressure wavelets l Receiver Pressure Wave Pressure “Wavelets”

Surface Vibrations and Sound Pressure contribution of a patch, Unit Normal Velocity Density of Air Sound Propagation Speed in Air Acoustic Impedance of Air

Surface Vibrations and Sound l Approximate differential elements with surface triangles l Apply band pass filters: ¡ Low pass: windowed sinc filter ¡ High pass: DC blocking filter l Result: Pressure known for all surface triangles

Putting it all together Pressure/Signal at Receiver Filtered Average Pressure Area of Triangle Visibility Term Receiver Vibration Approximation of Beam Pattern Distance Falloff

Propagation Delay Accumulation Buffer Receiver Distance from Source 1 d 1 Source d 2 t=0 Sound Propagation Speed t 1= d 1/c 2 t 2= d 2/c Receiver

Results: Capabilities l General models l Generated sounds are accurate l Stereo Sound l Doppler’s Effect

Demo

Results: Accuracy

Results: Speed Scene Time. Step(s) Nodes/Elems Time/Audio Time Bowl 10 -6 387/1081 91. 3/4. 01 mins Clamped Bar 10 -7 125/265 240. 4/1. 26 mins Vibraphone 539/1484 1309. 7/5. 31 mins 10 -7 (~1 day) Timings on a 350 MHz SGI Origin MIPS R 12 K processor

Overview l Background l FEM Simulation l Modal Synthesis (Foley. Automatic) l Comparison/Conclusions

Features l l l l Modal resonance model of solids Location dependent sounds Impact, slide, roll excitation models Real-time, low latency Easy integration with simulation/animation Practical Do not model propagation of sound from source to receiver

Synthesis Method Emission Sound. Vibration Samples Force User Propagation Listener Speakers

Vibration Surface u(x, t) of body responds to external contact force F(x, t) u(x, t) Strain Functional F(x, t) Speed of Sound Under suitable boundary conditions, the solution to the PDE is a sum of sinusoids

Emission Sound pressure s(t) linear functional L of surface vibration u(x, t) s(t) L u(x, t) Note that propagation is not modeled in above

The Modal Synthesis Model s(t) L u(x, t) F(p, t) “The response u(x, t) of an arbitrary solid object to an external force can be described as a weighted sum of damped sinusoids” Impulse response/modal model Since L is linear, it implies at s(t) must be a sum of damped sinusoids too

Example: A 1 D string a 1 a 0 1 st Mode Frequency = f 0 ak 2 nd Mode Frequency = f 1= 2*f 0 + …Higher modes Frequency = fk= k*f 0 +. . . + Main Idea: Sum contributions of all the modes The point of impact decides the proportions in which the modes are to be mixed: ak. Therefore, ak is a function of p, the point of impact The frequencies and damping parameters are a property of the object, and independent of how the object is hit

The Modal Synthesis Model s(t) L u(x, t) F(p, t) Impulse response, modal model Kth mode: Gain Factor Point Damping of impact Term Vibration Frequency Parameters measured experimentally

Force Modeling At runtime: Find gain parameters given the location, strength and kind of force. Synthesize sound from previous equation. l Impact l Sliding l Rolling Wavetable Stochastic

Impact Forces • Duration: hardness (T) • Magnitude: energy transfer (w) • Multiple micro-collisions Example:

Sliding/Scraping Micro-collisions lead to noisy audio-force

Sliding/Scraping l Wavetable approach ¡ Store force parameters ¡ Modulate amplitude with energy transfer ¡ Modulate rate with contact speed l Synthesis Approach ¡ Fractal noise represents roughness ¡ Filter through reson filter ¡ Resonance ~ contact speed ¡ Width ~ randomness of surface

Rolling No relative surface motion Differences with sliding: • Smoother: Use low pass • More damping • Harder to create • Less understood • Essential coupling?

Rolling: Smooth Surfaces ¡ ¡ Polyhedral objects do not lead to smooth rolling forces Instead use smooth surfaces directly

Rolling: Contact Evolution l c(u, v) Evolve the contact in Reduced coordinates q = (u, v, s, t, ). . q d(s, t) . q q

Rolling: Contact Evolution Piecewise parametric surfaces, loop subdivision surfaces l Explicit integration, no stabilization l Multiple contacts and conforming contacts are not handled l Used only when multiple contacts in close spatio-temporal proximity l

Demo

Dynamic Forces Pebble-in-Wok Demo Contact force Slipping speed Rolling speed Impulses …and locations

Results 0. 1% CPU time per mode l Graceful degradation of quality l The bell demo is interactive l Uses a PHANTo. M for interaction l Authors do not report any real timings l State that “sound quality” is perceptionbased and has no metric as of now l

Overview l Background l FEM Simulation l Modal Synthesis (Foley. Automatic) l Comparison/Conclusions

Discussion l FEM: Physically Rigorous and General l Too slow for interactive applications l Doesn’t scale well l Inappropriate to apply a 30 fps technique to 44000 fps? l Maybe too general for the problem domain?

Discussion l Modal model exploits the vibrational nature l Higher Efficiency l But, not rigorously physically based l Finding the parameters requires experimentation and “earballing” l No rigorous correlation between physical and perceptual parameters

Discussion l For Realtime: Need for a technique to cover the middle ground l Extracting modal parameters in general requires solving PDEs l Not possible to do in an automated manner l Approximate modal parameters and then use modal synthesis?

Conclusion l PBS involves orders of magnitude smaller temporal and spatial scales l Research is sparse, problems are dense l Main contributions of the two papers besides vibration modeling: ¡ FEM: Efficient modeling of sound propagation ¡ Foley. Automatic: Efficient, Approximate models to handle surface properties and contact forces

References O'Brien, J. F. , Cook, P. R. , Essl G. , "Synthesizing Sounds from Physically Based Motion. " The proceedings of ACM SIGGRAPH 2001, Los Angeles, California, August 11 -17, pp. 529536. l Kees van den Doel, Paul G. Kry and Dinesh K. Pai, “Foley. Automatic: Physically-based Sound Effects for Interactive Simulation and Animation” Computer Graphics (ACM SIGGRAPH 01 Conference Proceedings), pp. 537 -544, 2001. l

Acknowledgements Some images were taken from the referred papers and the corresponding SIGGRAPH slides