Intelligent Hearing Aid Project Beckman Institute University of

Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Collaborators: A. Feng, B. Wheeler, D. Jones, W. O’Brien, C. Lansing, R. Bilger He killed the dragon with his sword Stir your coffee with a spoon His plan meant taking a big risk Combined Speech Waveform Intelligent Hearing Aid Device Goal: Develop high performance auditory processors which can effectively extract a desired speech signal in the presence of multiple competing sounds. He killed the dragon with his sword Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 1

What We’ve Done 133 MHz TI C 62 fixed point DSP EVM. • Two microphones on eyeglasses plus Headphones • Real-time Operation • Highly directionally selective over short distances • Directional selectivity is separately adaptive in each frequency band • Short (e. g. 50 msec) adaptation time Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 2

Organization of Talk How We Got There • Biological Starting Points • Localization and Extraction Algorithm • Frequency Domain Minimum Variance Beamformer • Human Subjects Results • Future Directions Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 3

Biological Starting Point: Frequency Analysis in the Cochlea Courtesy: Phonak Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 4

Biological Starting Point Model of the frog auditory receiver as a fundamentally binaural device Frog’s ears are mechanically coupled to form a combination pressure/pressure-difference receiver (Beranek, 1954). Frogs are excellent at Localizing sound with Both ears.

Biological Starting Point: Dual Delay Line Coincidence Pattern Indicates Azimuth For each frequency: The location of the coincident firing is a measure of both the relative time delay and the azimuthal angle Jeffress, 1948 Konishi, Masakazu, Scientific. American. v. 268, Apr. 1993, p. 66 -73. Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 6

Ideas We Didn’t Use • Separate processing of sudden onset signals (e. g. consonants) and continuous signals (e. g. vowels) • Comb filters matching vocal track characteristics of speakers • Results of masking experiments from psychophysics Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 7

Localization Strategy Eq. (5) microphones f 1 dual delay-line Eq. (9) build 2 -D coincidence pattern “direct” - Eq. (12) or “stencil” - Eq. (15) integration over time … … A/D frequency analysis … R coincidence detection L A/D frequency analysis Cochlea (Basilar papilla in Barn Owl) … f. M dual delay-line build 2 -D coincidence pattern integration over time coincidence detection medial superior olivary nucleus (nucleus laminaris) Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign integration over frequency localization (external nucleus of the inferior colliculus) Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 8

Biologically Inspired Algorithm: Localization and Cancellation • Separation by Frequency • Dual Delay Lines • Identify Source Locations • Cancel Noise by Steering Nulls Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 9

Strategy • Localize Sound Sources as described • At each frequency at each time point: – Identify loudest sound source – Cancel that sound source • Computationally challenging – Matlab simulation requires 600 X more time than realtime • Results are Encouraging – 6 to 10 d. B improvement when four speakers directly overlap in time Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 10

Experimental Examples Stir your coffee with a spoon. Stir your The old train was powered by steam Interferer @ 65º Target @ 0º Interferer @ 30º Twelve talker babble He killed the dragon with his sword. Target @ 0º His plan meant taking a big risk. His pl Interferer @ 22º Combined Waveform Reconstructed Waveform Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Reconstructed Waveform Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 11

Experimental Summary Four Talkers Recordings made in a Conference Room Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 12

Algorithm 2: Minimum Variance Cancellation microphones L R A/D f 1 frequency … analysis fm f 1 optimal weight computation fm … … fm frequency buffer f 1 • Separation by Frequency • Minimize Off-Axis Signal Strength • Computationally Efficient if Done in the Frequency Domain Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign f 1 frequency buffer noise cancellation … fm frequency synthesis noise cancellation Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 13

Minimum Variance Beamforming • Constraint: Gain in “Look Direction” is Unity • Minimize Total Energy of Output Signal • Requires: Computation and Inversion of Correlation Matrix • Time Domain Solutions: Matrix is square of dimension C*N (#channels * length of filter) • Time Domain Solutions: matrix inversion is prohibitive leading to adaptive filter approaches Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 14

Frequency Domain Minimum Variance Beamforming • The sound signals are divided into windows • FFT taken with ¾ overlap • For each frequency compute – 2 x 2 correlation matrices (forgetting average over blocks of samples) – Optimal filter coefficients (1 divide) – Optimal magnitude/phase (2 complex mult/add) • IFFT • Output Sound Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 15

Different Beampatterns at Different Frequencies every 8 msec 650 Hz 800 Hz 950 Hz Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 16

FMV Algorithm Beamformer Sharpened Response Using Algorithm Microphone Responses (unmatched omnidirectional) 120 90 30 20 150 Left Right 60 30 10 0 210 330 270 60 Unprocessed 20 150 Processed 30 10 180 240 120 90 30 300 Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign 180 0 210 330 240 270 300 Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 17

FMV Algorithm Beamformer Cardioid Microphone Response 120 90 30 Left Right 60 20 150 Sharpened Response Using Algorithm 30 10 0 210 330 270 Unprocessed 60 Processed 20 150 30 10 180 240 120 90 30 300 Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign 0 180 210 330 240 270 300 Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 18

Cluttered Signal (4 talkers) a Processed Sound b Noiseless Target c Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 19

Comparisons with other algorithms • Frost Adaptive Beamformer: Time domain, same minimization goal as FMV. Uses LMS descent (approximated gradient) to adapt weights toward the optimum solution • Griffiths-Jim Sidelobe Canceler: Time domain, structure attempts to separate interference, then uses LMS. (Trajectory is different than Frost) • Peissig-Kollmeier: Frequency domain, uses the coherence function and functions of phase and level difference as variables in a function that calculates a weight for each frequency band Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 20

Comparison to Leading Algorithms Omnidirectional microphones on KEMAR acoustic mannikin 15 SNRG (d. B) 10 5 SNR Gain Averaged over Multiple input SNRs, sentences Minimum-Variance (FMV) Peissig-Kollmeier Frost Griffiths Sidelobe Canceller Conventional 0 1 2 3 1 interferer 4 5 6 2 interferers 7 8 9 3 interferers 10 11 4 interferers Different Recording Environments Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 21

FMV Azimuth (deg) Rapid Adaptation to Onset/Offset of Two Interfering Speakers (+45 deg) (at 800 Hz) 0 d. B 50 Note difference in onset sharpness of filter null 0 -20 d. B -40 d. B -50 -60 d. B Frost Azimuth (deg) 0 d. B 50 Target is at 0 degrees 0 -20 d. B -40 d. B -50 Time (s) 0. 5 1. 0 Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign 1. 5 2. 0 Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 22

Precise Adaptation Across Frequency to Onset/Offset of Two Interfering Speakers (+45 deg) at t=0. 6875 sec FMV Frequency (Hz) 6000 5000 4000 Arrows show aliased directions (2 p phase shifts) -20 d. B 3000 2000 -40 d. B 1000 0 Compare frequency range over which null at – 45° occurs 6000 Frequency (Hz) Frost 0 d. B 5000 0 d. B 4000 3000 -20 d. B 2000 -40 d. B 1000 0 -80 -60 -40 -20 0 Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign 20 40 60 80 Angle (deg) Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 23

Experiment #1 - Intelligibility ratings at different listening conditions (different SNRs) § Listeners: 9 adults with normal hearing and 9 with sensorineural hearing loss (SNHL). § Target stimuli: 24 high-context sentences from 8 talkers (4 males & 4 females) at 0˚ (from the R-SPIN - Bilger et al. , 1984). § Competing sound: A recording of multi-talker babble of 12 talkers at -22˚ at 4 different SNRs.

Pure tone audiograms for a sample of listeners with sensory neural hearing loss High frequency Hearing loss

Test Scene Emphasizes Highly Directional Nature of Algorithm (one of the distinguishing features of the algorithm) -22 -45 o 0 o +22 o o +45 o ~15 cm Scene: Thanksgiving Dinner Table

How Hearing Impaired Individuals Rate the Intelligibility of Sample Sentences with Closely Spaced Sound Sources Test Done in Simulation Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 27

* multi-talker babble at -22° from one source only Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 28

Real-Time Implementation Permits Real-Time Testing with Hearing Impaired Subjects • Do real-time testing results confirm simulation results? • Quantify improved intelligibility beyond that for directional microphones alone • Quantify the hearing threshold (SNR) benefit Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 29

CD player Test environment * * rg et Computer * * Cardioid mics Ta * Reversed speech of four talkers speaking R -SPIN sentences at -40°, -20°, and 40° from target at 0° Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Real-time system Probe mic system Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 30

Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 31

Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 32

Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 33

CONCLUSIONS • Our algorithm provides benefit in a multi-source environment with competing speech (e. g. Thanksgiving dinner table) • Speech understanding in noise rated consistently higher than for directional microphones alone • Hearing threshold in noise significantly better than with directional microphones – 6 to 8 d. B for normal hearing listeners – 8 to 11 d. B for listeners with SNHL Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 34

What Else Can We Do? Toward a truly intelligent generation of hearing aids What Everyone Wants in a Hearing Aid: Localize, extract, and deliver a clean, high fidelity, desired signal from a cluttered, highly reverberant, auditory scene. Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 35

Other Ongoing Work • Inclusion of Head Related Transfer Functions in Simulation of Data – Result: FMV algorithm performs even better • Inter-hearing aid communication – Needed for binaural processing – Power limits are a major problem Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 36

Other Ongoing Work • Auditory Scene Analysis – Better localization techniques – Automated detection of noisy and reverberant environments – Characterizing simple acoustical environments Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 37

Future Generations of Intelligent Hearing Aids • • Automatic selection of desired signals Interactive binaural amplification Wireless communication links Covert steering using the perceiver’s eye movements • Speech cue enhancement • Frequency transposition Intelligent Hearing Aid Project Beckman Institute University of Illinois at Urbana-Champaign Wheeler, Presentation to Penn State, Nov. 8, 2001 Slide 38

The Next Biological Lesson? (frogs again) Masking in the Auditory Nerve Sharper Masking in the Inferior Colliculus (due to GABA inhibition? ) No Masking in the Inferior Colliculus With application of GABA receptor Antagonist bicuculline Each graph is the change in threshold of response of a neuron to the frog’s mating trill as a noise source is moved in azimuth. (Ipsilateral in nerve, contralateral in IC)