COST follow-up Action Content Understanding for Smart Multimedia

COST follow-up Action Content Understanding for Smart Multimedia Systems Prof. Jurij F. Tasič

Introduction Smart multimedia communication system technologies n n n Media like text, audio, graphics, animation, video, and simulations, which convey information. Intelligent mechanisms for understanding the content, the user and the current service conditions Human-friendly virtual multimedia spaces Research focus n n Automatic content recognition Descriptor creation Adaptive user modelling Content adaptation. Research goals n n n Framework for new services based on content recognition and user modelling, driven by friendly interfaces. Unification of content representation and identification Standardization of user modelling procedures Iintuitive natural communication Improved overall user satisfaction. 15/03/2018

Motivation: To avoid flooding of the user with data, the most relevant information for a given situation in terms of user’s preferences, situation and terminal equipment should be prioritised in an automated way. Communication services based on content recognition and understanding over heterogeneous systems should be based on individual user models in order to provide service operation with reduced interaction requests. Intelligent gathering of standardised descriptors (MPEG-4, MPEG-7, MPEG-21) is foreseen. The work will be focused on methods of content understanding that provide relations among digital items, structured descriptors and subjects in an automated service flow. 15/03/2018

Objectives and Scientific Programme: Development and integration of advanced processing, management and communication technologies for automated content uderstanding by nodes and terminals enabling multimedia services and applications Targeted problems: n n n n Bridging the semantic gap Detection and recognition of meaning Building retrieval mechanisms based on multiple audio-visual features Building context aware retrieval mechanisms Using adaptive user modelling The unification of content representation and identification Integrating the activities of the communication chain Content understanding technologies will increase the efficiency of services and will lead to new multimedia experiences and applications. The social benefits of Cross–media Multidisciplinary nature of the proposed research activity is ideal for a COST approach 15/03/2018

Organisation, Timetable, and European and Economic Dimensions: The work on the proposed Action has been split among the following working groups: WG 1: Knowledge management in heterogeneous multimedia systems WG 2: Media processing in intelligent environment WG 3: Multimodal interfaces Transversal WG: Demonstrators and low-power implementations 15/03/2018

WG 1: Knowledge management in heterogeneous multimedia systems Knowledge management issues n n n Structured data in distributed environments Mechanisms for data distribution Heterogeneous types of media terminals Intelligent environments n n Minimal user interaction content analysis, recognition and understanding Advanced user models Seamless operation of new services Delivery methods n n Service quality Distributed content sharing (peer-to-peer) Universal Multimedia Access Content adaptation, scalability and transcoding 15/03/2018

WG 2: Media processing in intelligent environment Basic multimedia signal and data processing n n Enhance the signal quality Shape signals for optimal storage, distribution and display Data embedding algorithms Real time hardware and software implementations. Content analysis n n Segmentation and clustering in still images and video Image feature extraction Motion estimation and analysis Shape description techniques and tools for the definition of new descriptors Content recognition n n Object detection and tracking in videos Object representation Identification of speakers, speech and music, Image recognition Content classification Content description n Content declaration Description languages for objects and their relations Description of digital items. Understanding of content n Object descriptors and relations among objects 15/03/2018

WG 3: Multimodal interfaces Convergence of multi- and hyper-media technologies n n n Personal mobile terminals, television, video and computer technologies Control services and applications without keyboard and keyword search Automation of content recognition and content handling techniques to maintain and update large-scale interactive applications. Multimodal interactive systems n n n Adaptive user modelling Personalised content access Image retrieval systems Interactive multi-modal systems Voice communication interfaces. Personalised service response in heterogeneous environments 15/03/2018

Transversal WG: Demonstrators and low-power implementations Practical issues of smart multimedia systems based on content understanding n n service demonstrators specific hardware implementations Examples: n n n voice and gesture driven home entertainment devices cross-platform access for community-generated multimedia content massive distributed multimedia storage and retrieval systems automated compilation of course materials image restoration applications document reconstruction systems medical applications personalized interfaces in smart cars distance education databases/archives of old material ubiquitous fruition of documents of high cultural value. 15/03/2018

European and economic dimension European dimension: n The scientific involvement of the research topic and the complementary disciplines requested clearly ask for an international cooperation framework. Interactions are foreseen with COST 292 and former COST 282 regarding certain core methods, and with COST 219 ter concerning applications in assistive technologies. Economic dimension: n Participating entities per participant could be estimated as: 1 scientist (60, 000 EUR) + 1/2 technician (20, 000 EUR) + 2 Ph. D. students (60, 000 EUR) + 1/5 secretary (10, 000 EUR) = 150, 000 EUR. The full cost of all 25 expected Action participants estimates to 3, 750, 000 EUR (pa). 15/03/2018