Education and Research in the Center for Signal

Education and Research in the Center for Signal and Image Processing http: //www. eedsp. gatech. edu/

2 CSIP Summary • • Our Ph. D. graduates have impact worldwide in DSP education and research Distinguished faculty – – • • • 17 faculty (7 IEEE Fellows, 2 National Academy members) Co-authors of over 25 books on DSP & its applications Over 80 current Ph. D. students Located in GCATT building with excellent, modern facility Support from Georgia Research Alliance has provided outstanding well equipped labs. Center for Signal and Image Processing

Beowulf Cluster 26 dual processors 1 Gbyte memories Center for Signal and Image Processing

CSIP Faculty • Yucel Altunbasak • Chin Lee • David V. Anderson • Vijay K. Madisetti • Thomas P. Barnwell • Francois Malassenet • Mark A. Clements • James H. Mc. Clellan • Faramarz Fekri • Monson H. Hayes • Joel R. Jackson • Fred Juang • Aaron Lanterman • Russell M. Mersereau • Ronald W. Schafer • Douglas B. Williams • G. Tong Zhou Center for Signal and Image Processing

Past and Present Funding • Industry: Texas Instruments, Intel, BAE Systems, Hewlett-Packard, Mathworks, National Semiconductor, Analog Devices, Lucent, Harris, Hughes, Prentice-Hall • Federal: NSF, U. S. Army, DARPA, ONR, NASA, MPO • State: Georgia Research Alliance • Private Foundation: John and Mary Franklin Foundation • Total Funding: Current funding from government and industry totals about $6. 5 M Center for Signal and Image Processing

Current Research Areas - I • Speech Processing – – Robust automatic speech recognition New architectures for speech recognition High-quality low-bit-rate speech coding for voice over IP Blind separation of speech signals • Audio Signal Processing – Music analysis and synthesis – Compressed-domain processing of audio • Acoustic Signal Processing – Noise and reverberation removal – Microphone array processing – Spatialization Center for Signal and Image Processing

Current Research Areas - II • Video Signal Processing – – – – Target tracking in video Video streaming with error concealment and MDC Graphics streaming for the Internet Automated analysis of video Video indexing for smart VCR Super-resolution of video Face Recognition Video compression • Image Processing – Image-based graphical rendering – Image interpolation for digital color cameras Center for Signal and Image Processing

Current Research Areas - III • Multimedia & Multi-modal Signal Processing – – – “Intelligent Environments” Automatic storage/retrieval of speech and audio Audio-visual speech recognition Speech-driven facial animation Application of multimedia processing in education • Communications Signal Processing – – Chaos in wireless communication systems Space-time coding and OFDM Compensation for selective fading effects Finite field wavelet transforms and applications to error control coding and cryptography – Compensation of nonlinear power amps Center for Signal and Image Processing

Current Research Areas - IV • Signal Modeling – Multi-scale sinusoidal modeling • Biological Signal Processing – Automated measurement and modeling of behavior in biological systems • Military Signal Processing – – Buried mine detection using GPR, seismic & EMI Target Tracking in sensor networks Hyperspectral imaging and target classification SAR imaging • Medical Signal Processing – Segmentation of cardiac MRI images • DSP for hand-held communication devices Center for Signal and Image Processing

Industrial Partnership Examples • Texas Instruments Leadership Univ. Program – Members with MIT and Rice U. – Seven projects - 7 faculty and 7 Ph. D. students – Wireless video, CFA interpolation, speech coding, speech recognition, chaotic systems, face recognition, MIMO communication systems • Hewlett Packard Laboratories – Four faculty and six students – Focus on PDAs: low-power analog front-ends, structured audio, applications in education. Also, 3 D video for video conferencing, – HP Labs researcher in residence Center for Signal and Image Processing

Linearization of RF Power Amplifiers G. Tong Zhou, J. Stevenson Kenney • Power amplifiers (PAs) are inherently nonlinear. – Desire: high efficiency PAs, leading to low cost. – Downside of high efficiency: high nonlinearity. – Nonlinearity causes: (1) high bit error rate; (2) adjacent channel interference: must satisfy FCC. • DSP-based predistortion linearization. – Challenging issue: memory nonlinear effects in high power amplifiers (e. g. , base station PAs). • Indirect Learning Architecture adapts to changing characteristics • RF TESTBED Center for Signal and Image Processing

Indirect Learning Architecture A/D Advantage: No need to model or identify the PA. Center for Signal and Image Processing

8 -Tone Test Result • 8 -tone, 1. 2 MHz signal, Siemens CGY 0819 dual-band PA • Purple: w/o PD; green: w/ memoryless PD (K=7); cyan: w/ memory polynomial PD (K=7, Q=10) • 35 d. B of spectral regrowth suppression w/ memory polynomial PD Center for Signal and Image Processing

Video Resolution Enhancement Y. Altunbasak and R. Mersereau • Future broadcasting will be all digital. • High definition displays will dominate the market. • However, most programming is expected to be in SDTV format. HDTV Set NTSC SDTV PC Video Multi-frame Spatial Resolution Enhancement PC Monitor HDTV There is a clear need and technical opportunity to design systems to enhance the quality of the SDTV signal so that it matches the quality and capabilities of high definition displays. Center for Signal and Image Processing

Applications - Digital Cameras Subsequent multiple pictures (JPEG format) Reconstructed high-resolution picture Also applicable to high-quality printing from video sources such as DVD players, set-top boxes, TV sets, software MPEG players and camcorders. Requires a resolution enhancing print driver. Center for Signal and Image Processing

Face Recognition Monson H. Hayes Major problem is lighting and pose variations. Center for Signal and Image Processing

Results and Next Step • We have developed a new face recognition system based on a segmented linear subspace model – Robust to varying illuminations and tolerant to different poses, – Has recognition accuracy equaling or exceeding (>99%) other state-of-the-art systems, and – Has a fraction of the complexity. • Next Step: Face Recognition from Video – Face detection (patent awarded). – Pose detection (find best frontal view). – Face recognition (robust to varying illuminations, poses, facial expressions). • The Intriguing Question – How can we incorporate the multitude of images that are extracted from video to enhance the recognition system? Center for Signal and Image Processing

Finite Field Wavelet Transforms F. Fekri and D. Williams Goal: Establishment of a new research field that brings together researchers from signal processing, error control coding, data security and multicarrier signaling systems. Finite Field Wavelets Error Control Coding OFDM Modulation Security coding Center for Signal and Image Processing

New Research Directions in Data Security LL row-wise LH HL HH column-wise LL LH HL HH New Research Directions in Error Control Coding Center for Signal and Image Processing

Passive Radar Systems Aaron Lanterman Exploit “illuminators of opportunity” such as commercial TV and FM radio broadcasts for covert operation Target Tracking Positions Velocities Passive Radar System Signature Prediction via Computational EM Radar Cross Section Radar Imaging Target Classification Target Library Center for Signal and Image Processing

Imaging With 100. 0 on Your FM Dial Target Shape Formatted Raw Data Image Formed Via Processing F-22 Falcon-100 VFY-218 Center for Signal and Image Processing

Detection of Obscured Targets Jim Mc. Clellan & Waymond Scott • Landmines – No single sensor has proven capable of reliable detection across many types of “targets” – Can multiple sensors be used cooperatively to produce a system with robust performance? • A three sensor experiment – Electromagnetic Induction (EMI) Sensor – Ground Penetrating Radar (GPR) Sensor – Seismic Sensor • Multimodal processing – Imaging & Inversion – Cooperative Fusion of multiple sensors Center for Signal and Image Processing

EMI Sensor and GPR Physical Properties of Target EMI Sensor: 0. 6 - 60 k. Hz Permittivity Contrast Low Conductivity (Dielectric) High Conductivity (Metal) Mechanical Contrast EMI No Weak Yes No GPR Yes Yes* No Seismic No No No Yes Sensor GPR: 500 MHz – 8 GHz Tx 4. 5” Rx Center for Signal and Image Processing

Seismic Sensor: Surface Waves Man-made items often resonate Center for Signal and Image Processing

Comparison of EMI, GPR and Seismic Responses: VS-1. 6, 6. 5 cm deep EMI Seismic GPR x depth y t Center for Signal and Image Processing

Comparison of EMI, GPR & Seismic Responses Uncrushed Aluminum Can, 2 cm deep EMI GPR Seismic x depth y t Center for Signal and Image Processing

Cooperative Analog/Digital Signal Processing D. Anderson and P. Hasler • Target: Complex signal processing functionality with extremely low power • Approach: Perform substantial amounts of the processing in programmable analog VLSI Real world (analog) DSP Processor A/D Convertor Computer (digital) Specialized A/D Real world (analog) ASP IC A/D DSP Processor Computer (digital) Center for Signal and Image Processing

Cooperative Analog/Digital Signal Processing • Advantages of CADSP: – Better problem “fit” – Orders of magnitude improvement in power consumption / efficiency – Simpler A/D converter requirements, – Smaller size. • Current Applications Include: – – – Audio noise suppression Audio source localization / beam-steering Focal plane image / video processing Speech Recognition Field Programmable Analog Processor Arrays Center for Signal and Image Processing

Digital Media Asset Management Mark Clements • Sam Nunn Archives: Cooperative Effort between CSIP, IMTC, GT and Emory Libraries. • Fast searching of audio based on phonetic content. Typical speed of search: 72, 000 x real time (20 hours of content searched in 1 elapsed second). – Basis for startup company Fast-Talk which has received over $10 M venture funding. • New results demonstrate rapid searching of music by lyrics and melodies using same approach. Center for Signal and Image Processing

An Integrated Auditory-Cognitive Model speech Auditory Model Neural Transduction Model 3 -D Cortical Representation s f Cortical Scene Analysis Language Model Semantics & Schema Understanding results Enhancement results Sound Units (Phonemic Detection) Multi-target tracking Reinforcement Syntactic & Semantic Analysis (error correcting) Cortical Scene Analysis (Phonological Tracking) Segment Units Re-generation Recognition results Center for Signal and Image Processing

Immersive Telecollaboration • Presentation – Capturing, transmission and reconstruction of audio and visual information (conventional view) – Projection and rendering of the interaction in a 3 dimensional space (virtual view) • Participation – Coexistence of all participants in a shared virtual space (“shared reality”) – Control and manipulation of shared virtual objects (“virtual collaboration” for hands-on experience) Center for Signal and Image Processing

Perceptual Spatialization Sound spatialization makes talker-tracking easier in multi-party conferencing environments, resulting in improved effectiveness in communication Binaural Hearing & Cocktail Party Effect Spatial separation plays a role. § Compare mono with stereo Stream segregation also plays a role. § Compare one talker (m 1+m 2) (m 1 m 2 f 2 ) with two (m 1+f 2) Stereophonic Conferencing Demonstration Center for Signal and Image Processing

Multi-channel Source Separation s 1 H 11 W 11 H 21 s’ 2 W 21 H 12 s’ 1 W 22 x 1 W 12 s 2 x 2 H 22 mixing un-mixing (room impulse responses) One possible approach (Ikram of Gatech and Morgan of Bell Labs): x=Hs R’ = x x. H s’ = W x Find un-mixing filter matrix W such that s’ = W R’ WH is diagonalized by minimizing the squared Frobenius norm of the off-diagonal matrix of s’ Center for Signal and Image Processing

Sound Source Localization 1. Time Delay Estimation 2. Source Location Estimation Various methods: Developed at Bell Labs & Georgia Tech talker • triangulation - solve a set of hyperbolic equations • spherical intersection - solve a set of linearized spherical equations • spherical interpolation - similar to SI, but with reduced constraint • one-step-least-squares – transforms the problem into an estimation/minimization problem; works the best Applications: • Conferencing with participant tracking • Improved sound and sight pickup Further challenge Center for Signal and Image Processing

Low Complexity Rate-Distortion Optimal Coding Mode Selection Hyungjoon Kim and Yucel Altunbasak Proposed Rate-Distortion Model D= Distortion Standard deviation Model based Model Selection Candidate Mode 0 modes 2 e - R Model parameter Mode 1 Mode N Rate for DCT coefficients • Provides 10 -15% bit-rate savings • Patented, licensed, and commercialized • Based on General Gaussian R-D model • Calculation of D has low computational complexity • Adaptive model parameter R-D cost calculation Minimum cost Mode k Best mode Experimental Results R-D model-based approach (Proposed) Distortion-based approach (TM 5+Rho) Center for Signal and Image Processing

R-D Optimized Multi-Server Streaming Ali C. Begen and Yucel Altunbasak Server Client Server Goal: Developing media-aware and network-adaptive packet delivery and error recovery mechanisms for multipoint-to-point networks Approach: Client-driven rate-distortion optimized streaming Suitable For: Multi-homed clients, wireless systems, CDNs Center for Signal and Image Processing

Mobile Video Streaming Bandwidth Umut Demircin and Yucel Altunbasak Video Rate Available Bandwidth Error Propagation and Frame Freeze • Challenges: – Fluctuating wireless channel error-rate and bandwidth – Video error propagation • Solution Approaches: – Video and channel aware • FEC code rate and link-layer ARQ adaptation. • Rate reduction and error-resiliency video transcoding. • R-D optimized packet scheduling • Diversity methods Center for Signal and Image Processing

Video Resolution Enhancement Yucel Altunbasak Sequence of limited dynamic range images Composite image with higher dynamic range and resolution • Compressed-domain resolution enhancement • Bit-depth and contrast enhancement • Resolution enhancement for FACE video • Three patents, one licensed Zoom: Center for Signal and Image Processing

40 Hyper-Spectral Super-Resolution Panchromatic Multi-spectral Hyper-spectral • Hyper-spectral images offer huge amounts of data. • Spectrum is sampled at more than 200 wavelengths. • Spatial resolution is the key parameter in many related applications. • To improve spatial resolution we combine Ø A precise physical model of the imaging model, and Ø The intrinsic low dimensionality of hyper-spectral data. • The result is an efficient and noise-robust super-resolution (SR) method. Bilinear interpolation Separate band SR Our method Center for Signal and Image Processing

Demosaicing Yucel Altunbasak Sensors CFA Optical system Scene § Digital cameras use a single sensor array with a color filter array (CFA) to sample different spectral components. § At each pixel location, only one color sample is taken, and the other colors must be interpolated. § This color plane interpolation is known as demosaicing. Center for Signal and Image Processing

Advanced Collaborative Systems Fred Juang and Ghassan Al. Regib Shared Virtual Space 3 D Collaboration System • Display • Display Participant in one location 3 D Collaboration System 3 D Networking Realtime Registration Kinematics Control Current System built at GT • 3/19/2018 • • Shared Reality: Allows the virtual world to coexist and to interact with the real world undividedly and seamlessly. Sensors are used to capture users’ motions in the real world and are used to control objects in the virtual world synchronously. Smart Objects: A new data structure creates smart objects and introduces efficient usability. Details Count: The multimodal micro-tool supports dexterous and real -time control of remote virtual objects. Speech-enabled commands facilitate micro-level manipulation and control. 42

Distributed Processing & Communications Protocols for Distributed Sensor Systems Ghassan Al. Regib Application Data Models Digitized Observations / Decisions Analog Waveform Distributed Detection Parameter Estimation (which sensor to send and what to send) Communication Protocols (how sensors communicate among each other) Data Processing Data Communication • Distributed Parameter Estimation -System Overview: – Distributed Sensors: observe, quantize and transmit their observations – Fusion Center: perform the final estimation based on the received messages – Goal: Minimize the estimation MSE under the constrained total bit rate Center for Signal and Image Processing

Bit Allocation for Textured 3 D Models Ghassan Al. Regib • Target: Best display quality of the 3 D model during progressive streaming • Approach: Optimal bit allocation between geometry and texture in bitstream Original model Case II Center for Signal and Image Processing

Streaming Meshes over Lossy Networks Ghassan Al. Regib • Interactive 3 D applications Packet losses – Large amounts of data – Real-time interactivity – High-resolution visualization Sender • Technical problems 3 D data – Packet losses – Stringent delay constraints Server-side: – Best-effort network: • Multi-resolution compression bandwidth bottleneck, congestion… Delay constraints Best-effort service receiver 3 D data Network side: • FEC-based packet loss protection • Feedback-based retransmission • Congestion control RS: Reed-Solomon code From: 28. 35 d. B To: 37. 76 d. B Center for Signal and Image Processing

Summary • The premier academic program in the country in the signal processing field is in the Georgia Tech School of Electrical and Computer Engineering. • We have many outstanding graduate students. – Internships – Long-term contributors • We have lots of outstanding technology waiting to be developed. • We have a demonstrated capability to work with industry. • Contact jim. mcclellan@ece. gatech. edu if you want to come for a visit. Center for Signal and Image Processing