Integrated Media Systems Center Ulrich Neumann Director Charles

Integrated Media Systems Center Ulrich Neumann, Director Charles Lee Powell Professor of Engineering Associate Professor, Computer Science Isaac Maya, Ph. D. , P. E. Director, Industry and Technology Transfer Programs IMSC’s Immersive Technologies National Science Foundation Engineering Research Center

Integrated Media Systems Center a partnership among: National Science Foundation University of Southern California School of Engineering Annenberg Center for Communication State of California The City of Los Angeles Industry Partners: Computer Hardware and Software Entertainment Broadcasting Telecommunications Publishing Aerospace NSF Downtown LA Other Government Agencies: DARPA, NASA, JPL, ONR, US Army $10 M operating budget IMSC Headquarters National Science Foundation Engineering Research Center

Faculty and Academia A top-notch array of investigators – 14 of whom (~ 50%) were sought specifically because of IMSC 29 investigators working with 87 Phd, 26 MS, and 33 UG students Investigators come primarily from EE and CS – others from Psychology, Industrial and System Eng. , School of Cinema/Television, Annenberg School for Comm. , School of Gerontology, Biomedical Engineering, and the Information Sciences Institute Two IMSC investigators have PYI awards Seven IMSC faculty have CAREER awards (3 CAREER awards this year) Tomlinson Holman won a 2001 Technology Achievement Award from the Academy of Motion Pictures Robert Sholtz won a MILCOM Achievement Award for UWB research (only 4 th in 20 yrs) IMSC faculty published 53 peer-reviewed journal articles and 205 peer-reviewed conference papers articles appearing in print over a 12 month period in this past year (2001 -2002) National Science Foundation Engineering Research Center

Education Programs created by IMSC 171 students graduated with IMSC providing funding, classes, and research aspects of their education experience 81 with PHD, 77 with MS, and 13 with BS IMSC created 3 MS programs enrolling 358 students unique among U. S. universities IMSC created 2 UG minor programs enrolling 207 students IMSC gave research fellowships to 32 UG students Created more than 20 courses for programs – e. g. : Human Factors in Integrated Media Systems and Architectures School of Engineering/Fine Arts (joint project course) National Science Foundation Engineering Research Center

Industry Program IMSC offers cost-effective research partnership at the 5 -10 yr horizon Pursue science and discovery of methods and algorithms – IP licenses Develop technology-impact and proof-of-concept before “start up” Prototype feasibility applications and testbeds Provide partners a view to potential opportunities and threats Strong support through six years 28 members this past year (2001 -2002) Recognized 5 members for consistent partnership over 6 yr life of center HP, NCR, FXPAL, TRW, Lockheed Martin Strong growth in international corporations and government programs Taiwan (III, ITRI) Korea (LG Electronics) Japan (FXPAL, Hitachi, NTT Do. Co. Mo, Toshiba) Netherlands (Philips Research) Italy (ST Microelectronics) Patents and tech transfer 46 patents filed over the life of center 24 patents licensed and 39 software licenses National Science Foundation Engineering Research Center

IMSC Research Members Japan NTT Do. Co. Mo III ITRI Taiwan Microsoft Intel WA Nav. Tech IL Eastman Kodak NY DRS Technologies Prentice Hall PTR NJ OR See California Map Collaborations with Government Agencies and Foundations: NCR Time Domain US Army DARPA NASA Toyota Foundation National Science Foundation Engineering Research Center OH AL

IMSC Research Members Los Angeles/San Diego • Above. Net Communications • Boeing • HRL • JPL • Los Angeles Times • NMUA • Panoram Technologies • TMH Corporation • TRW Silicon Valley • Concept. Labs • F-X Palo Alto Laboratory • Geometrix • Hewlett-Packard • Inter. Video • Lockheed Martin • Sun Microsystems Silicon Valley Hollywood USC IMSC Los Angeles National Science Foundation Engineering Research Center

IMSC Research Program -- Immersipresence Motivated by Moore’s law: exponential performance growth. . . enables advances towards the future of the internet, grid computing, nano-technology, . . . National Science Foundation Engineering Research Center

People are often the ultimate purpose behind silicon systems In pursuit of an experience, or the performance of a task, or understanding of information Integrated Media System (IMS) mediate the gap between silicon and humans An extension of the human senses, actuators, or cognitive process Our goal is to advance the science and engineering of “well designed” Integrated Media Systems that exploit human information processing capabilities - providing effectiveness that we associate with “immersipresense”. National Science Foundation Engineering Research Center

To better understand immersipresence and help set specific goals, we develop visions of how IMS technologies will affect people and their activities We examine four domains for IMS impacts in the future These four vision scenarios drive the center’s program of six research and engineering areas National Science Foundation Engineering Research Center

UWB console captures 3 D model and haptics scan PDA provides spatial array of participants and sonic translation of haptic trace Immersive workstation coordinates video, avatars, graphics, audio, and haptics to facilitate problem resolution

Theory of Perceptual and Cognitive Pleasure Immersive Audio User State Sensing and Perceptual User Interfaces Facial Gesture Analysis and Animation Virtual Environments for Performance Testing and Training Robust Vision Analysis Immersidata Content Extraction & Analysis Multimedia Networks, Transmission, and Communication Ultra Wideband Wireless Remote Media Immersion Software Architecture for Immersipresence HAPTICS – devices and translations National Science Foundation Engineering Research Center

Immersive on-demand streaming of audio/video experiences - fusion of cinema and internet Virtual/video peer-to-peer telepresence experiences for multiple participants Full body immersion role-playing game experiences with realistic or synthetic characters

Emotional Expressions User-State Sensing and Perceptual User Interfaces Immersive Audio Speech Recognition and Synthesis Facial Gesture Analysis and Animation Immersive Media Real. Time Storage & Retrieval Multimedia Networks, Transmission, and Communication Ultra Wideband Wireless Remote Media Immersion Software Architecture for Immersipresence Remote Media Immersion - fusion of internet and cinema Robust Vision Analysis Digital Geometry Processing National Science Foundation Engineering Research Center

IMSC research program Theory of Perceptual and Cognitive Pleasure Emotional Expressions User State Sensing and Perceptual User Interfaces Virtual Environments for Performance Testing and Training Immersive audio Information Integration Speech Recognition and Synthesis Immersidata Content Extraction & Analysis Facial Gesture Analysis and Animation Robust Vision Analysis Digital Geometry Processing Customized Querying & Rendering Multimedia Networks, Transmission, and Communication Ultra Wideband Wireless Compression Remote Media Immersion RMI - fusion of internet and cinema Software Architecture for Immersipresence Bio. SIGHT - authoring and assessment I-NEWS - story models and user preferences HAPTICS - devices and translations Immersive Media Real. Time Storage & Retrieval National Science Foundation Engineering Research Center

Research Highlights IMSC has produced ground breaking results and fundamental research in: immersive audio streaming servers computational framework for grouping based on tensor voting graphics & animation theoretical foundations for understanding benefits of media immersion computer vision leadership in technology, FCC regulation, and commercialization perception & cognitive modeling distributed and scalable architecture UWB wireless multichannel and HRTF approaches - holistic DSP approach 3 D DSP mesh processing, facial expression analysis and animation, hair modeling virtual reality -- chair of VR 2003 in LA applications to psychology (ADD diagnosis), haptics applications, and user studies National Science Foundation Engineering Research Center

IMSC Immersipresence Vision: Applications in Entertainment USC Homecoming Game at the Los Angeles Coliseum Animation File Goes Here: PC – panoramic_animation. avi MAC - panoramic_animation. mov 360º Camera Panoramic Visual Immersion: Malibu Hills National Science Foundation Engineering Research Center

IMSC Immersipresence Vision: Medical Applications Animation File Goes Here: PC – 200_0003. AVI Evaluation of ADHD in Children National Science Foundation Engineering Research Center

Animation and Modeling of People for Immersipresence Avatar: Digital Cloning Video Files Mac: face_1. mov PC: avatars_doug. avi 3 D Visual Environment Avatar simulates live video National Science Foundation Engineering Research Center

Animation and Modeling: Expression Cloning Real Expression can be “cloned” to any model National Science Foundation Engineering Research Center

Animation and Modeling: Expression Analysis Video Files Mac: face_1. mov PC: avatars_doug. avi Real Expression can be “cloned” and analyzed National Science Foundation Engineering Research Center

Digital Geometry Processing: 3 D Modeling and Elasticity Animation File Goes Here: PC – mat_1. avi MAC - mat_1. mov Animation File Goes Here: PC – mat_3. avi MAC - mat_3. mov Animation File Goes Here: PC – mat_2. avi MAC - mat_2. mov Next wave of visual data: Better understanding and manipulation of geometric data National Science Foundation Engineering Research Center

Mixed Reality and Visualization Augmented Reality for Space Flight (NASA funded) Develop AR authoring tools for video-based training Anthony Majoros, Human Factors group, The Boeing Corp. 4 D Battlefield Visualization (MURI funded) Develop fusion of video, images, and 3 D models Avideh Zakhor (Berkeley), Suresh Lodha (UC Santa Cruz), Bill Ribarsky (Georgia Tech), Pramod Varshney (Syracuse) Wide Area AR Tracking (ONR funded) Novel sensors and fusion for tracking position/orientation National Science Foundation Engineering Research Center

Recognizing Emotions in Speech Natural Human-computer interaction: need for machines that can recognize “what, who and how” Call center applications, Teleconferencing, Tutoring by machines, Entertainment Algorithms (ASRU’ 01, ICME’ 02, ICSLP’ 02) Support Vector Machine, Linear discriminant and k-NN classifier for acoustic information Novel information-theoretic measure of emotional salience in language (word sequences) Acoustic and linguistic information fusion Performance evaluated on real call center data rather than feigned data from actors Non-negative Negative (From the same person and dialog) National Science Foundation Engineering Research Center

Immersive Audio 10. 2 Video Files Mac: audio_10. 2_channel. mov PC: audio_10. 2_channel. avi Immersive Audio Environment 3 D Virtual audio field at the location of the ears at all times National Science Foundation Engineering Research Center

Haptics Research Movie File: PC: haptic_1. avi MAC: haptic_1. mov Cyber. Grasp Movie File: PC: haptic_2. avi MAC: haptic_2. mov PHANTo. M (3 D Mouse) Force control for haptics rendering Collision detection Compression of haptic data Haptic playback and analysis Application Research Project: Haptic Museum National Science Foundation Engineering Research Center

Haptics Experiment Mutual Touch INTERNET PHANTo. M Cyber. Grasp Virtual “touching” through the internet Virtual “handshaking” Distributed Haptic World National Science Foundation Engineering Research Center

Media Communications High Bandwidth Networks: synchronization, latency, bandwidth Wireless Systems Optical Communication Compression Encryption and Network Security Photonic Networks Qo. S Management of IP Networks National Science Foundation Engineering Research Center

Managing Immersidata Avatar Haptic Application Control Immersidata 3 D Model Immersive Audio Video Text Real-time acquisition and playback Efficient query and analysis Integration of disparate information sources National Science Foundation Engineering Research Center

“User Centered Sciences” Research Integrates Multiple Methodologies: Human Factors Human Computer Interaction General Experimental Psychology Usability Engineering Communication User-Centered Design Cognitive Science Ethnography With the aim to improve: Effectiveness Efficiency Enjoyment Safety National Science Foundation Engineering Research Center

Systems Engineering Similarly, unique systems engineering and integration are hallmarks of IMSC RMI testbed for streaming media immersion Showcase for IMSC research and integration May 2002 demonstration (NY Times) Flow Scheduling Framework of the Media Immersion Environment - open source release Bio. SIGHT authoring tools and web-based education Testing and assessment in K-12 classrooms National Science Foundation Engineering Research Center

Applications RMI A fusion of internet browsing with a theater-like immersive experience Immersed in a college football game HD Video at up to 45 Mbits/sec 10. 2 ch Immersive audio (16 Mbits/sec) Steaming over the Internet on-demand to a mouse click (20 min prog) Doctors assisting in a remote procedure RMI demonstrates IMSC innovations in several critical technical areas: Streaming media servers Immersive audio capture and rendering Network protocols for error correction Business people negotiating like they are in the same room Synchronization Students visiting an aquarium a thousand miles away National Science Foundation Engineering Research Center

RMI Challenges Immersive, high-quality video and audio acquisition and rendering High Definition video 1080 i and 720 p (~40 Mb/s) 10. 2 channels of uncompressed audio (12 Mb/s) Storage and transmission of media streams across networks Streaming media servers/clients Error correction via selective retransmission Synchronization between streams A/V, A/A, V/V National Science Foundation Engineering Research Center

RMI Architecture National Science Foundation Engineering Research Center

Storage, Streaming & Rendering Focus: End-to-end streaming architecture Requirements: A streaming platform that can scale and handle synchronized, highbandwidth streams Distributed Server Storage Scheduling Scalability Clients Multi-stream Synchronized playback Transmission Robust VBR Flow control [ACM Computer’ 02] National Science Foundation Engineering Research Center

Scalability: Multi-Node, Multi-Disk Data and control network traffic can be routed with different logical topologies Yima-1: single data path (high inter-node traffic) Yima-2: multiple data paths (low inter-node traffic) Yima-1 Yima-2 National Science Foundation Engineering Research Center

Robust Stream Delivery Variable bit rate (VBR) media encoders allocate more bits to complex scenes and less bits to simple ones Smoothing of VBR media traffic has the following quality benefits: Better resource utilization (less bursty) More streams with the same network capacity Multi-Threshold Flow Control (MTFC) algorithm objectives: Online operation Content independence Minimizing feedback control signaling Rate smoothing National Science Foundation Engineering Research Center

Premier Demonstration May 9, 2002 Cross-country streaming from ISI -East in Virginia NY Times coverage in “Circuits” section NBC-TV and KTLA http: //imsc. usc. edu/rmi/ http: //www. east. isi. edu/NGI-S/ National Science Foundation Engineering Research Center

Future Impact of RMI High-quality audio/video experience streamed ondemand over the internet (live or recorded) Everyone is a content producer, distributor, theater Currently only broadcast, cable, theater, or satellite operators can distribute content… high entry cost and regulation barriers RMI technology removes distribution barriers CEO NVidia: “the end of TV as we know it” LIVE concerts, sporting events, alternative news coverage, … RECORDED movie archives, news archives, education programs, … “More than TV” experience – powerful human motivation Playback quality and options are flexible – retain all the capabilities of computing on the internet National Science Foundation Engineering Research Center

Distributed Immersive Performance Next generation of Remote Media Immersion (RMI) Vision – interactive peer-to-peer streaming Create seamless immersive environment for distributed high-quality audio/video interaction Impact – widespread telepresence Distributed performances, meetings, social gatherings, education, Technical barriers Precise timing and synchronization of streams Low latency, error correction – high quality-of-service network protocols National Science Foundation Engineering Research Center

National Science Foundation Engineering Research Center

Summary IMSC’s vision of Immersipresence and Integrated Media Systems aims for fundamental breakthroughs in technology, industry partnerships and education IMSC is carrying out Highly cross-disciplinary cutting edge research Aggressive industry collaboration and technology transfer Unique educational experience and teamwork skills for students IMSC is eager to expand its Collaborations National Science Foundation Engineering Research Center

Acknowledgments National Science Foundation Engineering Research Center program http: //www. eng. nsf. gov/eec/erc. htm IMSC Industry partners. . . More information about IMSC http/: imsc. usc. edu IMSC PARTNERS Airborne 1 Corporation Compression Science Corporation Concept. Labs Eastman Kodak F-X Palo Alto Laboratory Geometrix Hewlett-Packard HRL IBM Informix Institute for Information Industry Intel Inter. Video JPL Litton Lockheed Martin Los Angeles Times Lucent Technologies/Bell Labs NCR Panoram Technologies Philips Multimedia Center Prentice Hall PTR Raytheon STMicroelectronics Sun Microsystems TMH Corporation TRW