DSP SHADERS AND LIGHTING FOR AUDIO Guy Whitmore

DSP: SHADERS AND LIGHTING FOR AUDIO Guy Whitmore Director of Audio Microsoft Game Studios

An Apt Analogy If raw SFX are akin to 3 D visual models, then runtime audio DSP (Digital Signal Processing) is akin to lighting, pixel shaders, and particle effects.

Agenda » Current state of DSP in games » Potential for DSP use in games » DSP middleware in traditional linear audio tools » Creating a DSP middleware market for games

Why Run-time DSP? Why not simply bake it in? » Blending and cohesion » Variation » Dimension and depth » Flexibility » Mix iteration and fine control » Shape the mix: Sweeten or fatten

Keepin’ It Real » In Linear media, DSP is typically kept 'real' until the final mix stage » In games, final mix is the output of the console as it's being played! > The work of the sound designer continues within the game itself.

Common Uses of DSP in Games » Environmental Reverb > Hardware based vs. software » Basic EQ and Low Pass Filters > LPF over distance or underwater » Pitch shifting and dopler Variation > Car Engines >

Further Uses for DSP » Radio chat DSP » Early reflections for environmental reverb » USB microphone input » Pitch detection/correction » Time stretch/compression » Aural exciter on swords » Live chat DSP to morph voice

Game Parameters Directly Tied to DSP » » Time of day, change of seasons Weapon magic or quality over time Distortion on highest car RPMs LPF on unloaded auto RPMs (deceleration)

Run-time Mixing and DSP » Mix sets with different DSP settings > Change the mood from one scene to another » EQ and compression intended for mixing via groups > Channel strips for character VO » Mastering chain DSP > Mastering EQ, compression, limiting

Where Does DSP for Games Currently Come From? » As part of Console SDKs > Reverbs, compression, EQ » As part of an Audio Engine license » Coded in-house > Centralize your custom DSP Result: Limited range and variety of DSP

DSP in Traditional Linear Audio Production Digital Audio Workstations (DAWs) » Come with many DSP plug-ins » Plug-in archetechture enables 3 rd party DSP makers to participate in a DAW's ecosystem. Result: » Large and varied 3 rd party selection of DSP » High end brand name to free-ware

Why so Many DSP Makers? Isn’t a reverb? » » Choice Aesthetics and taste Variety Differentiation > Currently most 360 titles use the XDK reverb » Appropriateness to the title » CPU and tech requirments vary

DSP Licensing For Games …so far » Console makers/card makers > > Xbox 360 – Princeton Digital Reverb Xbox (and Direct. Music) – Waves Reverb » Audio Engine companies? > Wwise - Mc. DSP » Developer licenses > Bungie - Waves » Publisher licenses > MGS - Wave Arts Ltd.

MGS and Wave Arts A pattern for future deals In short, it’s a… » multi-license » multi-algorithm > Master Verb, Trackplug, Final Plug, Multidynamics » multi-platform > Xbox 360 and PC » Multi-format deal > > XAPO, generic, Wwise and FMOD plug-in compatibility coming » Structured to give Wave Arts the upfront capital to port the plug-ins.

Wave Arts Now that the upfront work is done, other publishers and developers can directly license DSP from Wave Arts. Possibilities: » More algorithms » Port to PS 3? …to Wii?

The Dream Wouldn’t it be cool if… » The extensive variety of DSP brands used our linear DAWs were available to us in our game environments? » We could easily 'plug-and-play' new DSP in our audio engines (no coders) for seamless auditioning? » Licencing 3 rd party DSP was the norm, relatively easy, and affordable?

A 3 rd Party DSP Market for Game Audio Benefits: » Access to a wide range of quality algorithms » Independent of game platforms and audio engines » Easy to audition and edit in game and offline » Sound designers are familiar with many of the brands and algorithms

Technical Challenges The Easy Part (relatively speaking) » Porting DSP to various game platforms » Optimizing DSP to console archetechture and for efficiency. » Availability of dev kits for DSP makers » Various plug-in formats > An industry standard plug-in format is a worthy goal and should be pursued, but it’s not mandatory for a successful middleware ecosystem.

Financial Challenges What will the market bare? » How should DSP licenses be charged? » What's a fair price per SKU? > > > Affordable by developers/publishers While making the effort worthwhile for a DSP maker Differentiation between DVD and downloadable games? » Audio/game budgets for DSP licenses

Chicken Egg Conundrum How to kick-start a middleware ecosystem Kindling for the fire » Build a critical mass of licensors > 3 -5 deals similar to Wave Arts

An Industry Challenge To major game publishers and audio leaders: Strike new DSP porting/licensing deals with existing DSP makers » Modeled similarly to the MGS – Wave Arts deal i. e. > > Multi-license and multi-algorithm DSP company able to license algorithms to other developers after contract term » Work with DSP companies that… > > Make interesting quality algorithms Show willingness to forge into a new market Create CPU efficient algorithms Will support their products into the future

? ? Questions? ? ? ? ? ? ? ? ?