FHG Fraunhofer Institute for Digitale Media Technology Uwe

FHG Fraunhofer Institute for Digitale Media Technology Uwe Kühhirt Christian Weigel Patrick Aichroth Marcel Sieland © Fraunhofer IDMT

Fraunhofer-Gesellschaft n Fraunhofer is the international leader of applied research n 67 institutes and independent research units n 24. 000 staff at more than 40 locations n More than 2. 1 billion euros annual research budget totaling; 1. 8 billion euros from contract research n Roughly two thirds of this sum is generated through contract research on behalf of industry and publicly funded research projects n Roughly one third is contributed by the German federal and Länder governments in the form of base funding n International cooperations © Fraunhofer IDMT

Fraunhofer IDMT n Foundation of the working group of the Fraunhofer IIS in Erlangen in 2000 n Foundation of the Fraunhofer IDMT on January 1 st, 2004 n Director of the institute: Prof. Dr. Karlheinz Brandenburg n 160 employees, apprentices, Ph. D students, interns and student assistants n Head office in Ilmenau, branch labs in Erfurt, and Oldenburg (Lower Saxony) © Fraunhofer IDMT

Business Unit Media Management & Delivery – M 2 D n Core Competencies n digital signal processing, n machine learning, n security and privacy, n evaluation of technologeis, components, and systems n Skills n software engineering and development: C++, JAVA, Python, … n metadata modelling, n cryptography, privacy enhancement technologies, n requirements analysis and system design, n system integration, databases, … © Fraunhofer IDMT

Business Unit Media Management & Delivery – M 2 D n Service offers n Development of Technologies: methods, components, and systems n Solution architecture & Consulting n Evaluation & Testing of Technologies n Markets & Customers n Broadcast Industry n Archives n Multimedia n Research institutions n Manufacturing industry n… © Fraunhofer IDMT

Business Unit Media Management & Delivery – M 2 D Head of BU / Biz. Dev: Steffen Holly Head of Department: Uwe Kühhirt n Audio-visual Systems AVS Uwe Kühhirt 3 researcher/developer, 1 developer n Semantic Music Technologies SMT Hanna Lukashevich 4 researcher/developer, 3 Ph. D students n Media Distribution & Security MDS Patrick Aichroth 3 researcher/developer, 3 Ph. D students http: //www. idmt. fraunhofer. de/m 2 d © Fraunhofer IDMT

Media Management & Delivery – M 2 D Public Projects n THESEUS: Evaluation of Multimedia technologies n SAISBECO: identification of great apes based on face recognition n DIOMEDES: DIstribution Of Multi-view Entertainment using content aware DElivery Systems n CUBRIK: human-enhanced multimedia search n Sync. Global: global music search and cross-modal synchronization of music and video n REWIND: Reverse engineering of A/V content data n Mi. CO: Media in Context n VALCRI: visual analytics for sense-making in criminal intelligence analysis n Audio. Trust n Quo. Vi. Tra: quality optimised video transcoder n CAD-SE: Search Engine for 3 D-models © Fraunhofer IDMT

Face Recognition SAISBECO – Identification of apes © Fraunhofer IDMT A Semi-Automated Audiovisual Species and Individual Identification System for Behavioral Ecological Research and Conservation

Video Identification News Content History: GUI http: //www. cubrikproject. eu/ © Fraunhofer IDMT

Semantic Classification Visual Concepts Familiy Partylife City Beach Snow Indoor / Outdoor Night Sunset Day Mood: Calm, happy, aggressive, melancholic © Fraunhofer IDMT Appeal / Aesthetics + -

Mi. CO: Media in Context Showcase: Crowd Sourcing © Fraunhofer IDMT http: //www. mico-project. eu/

Mi. CO: Media in Context Extractors n Face Detection n Animal Detection https: //www. ffmpeg. org/ n OCR n Media Quality http: //opencv. org/ n Video Segmentation n Diarization libccv http: //www. imagemagick. org/ http: //libccv. org/ mediainfo n Speech 2 Text https: //mediaarea. net/de/Media. Info n Speech-Music-Discrimination https: //opennlp. apache. org/ n Audio Editing Detection n Text Analysis n Sentiment Analysis http: //www-lium. univ-lemans. fr/diarization/ n Emptiness Detection n Nudity detection © Fraunhofer IDMT http: //kaldi. sourceforge. net/ http: //www. mico-project. eu/ http: //nlp. stanford. edu/

IMC: Persons @ FHG n Uwe Kühhirt and Patrick Aichroth: n Coordination and deputy coordination within FHG n Christian Weigel, Alexander Loos (group AVS): n A/V extractors (WP 7) n Patrick Aichroth, Marcel Sieland, Luca Cuccocillo (group MDS): n modeling, orchestration, access control © Fraunhofer IDMT

Role in the Project I-Media-Cities n Technical partner n close cooperation with CINECA and CINTMATEQUE n Support requirement analysis n Support metadata modelling & md analysis n lead WP 7: A/V annotation n provide extraxtors n Annotator ochestration n Support architecture design and system integration © Fraunhofer IDMT

FHG: Contributions n WP 1 Coordination (1 PM) n WP 2 User requirements (3 PM) n Contributions to requirements analysis, especially from a technical perspective („bottom-up“) n WP 4 Research & Content (1 PM) n Contributions to content and annotation preparation, especially from a WP 7 perspective (crucial for extractor development and testing) n WP 5 Visual interfaces of research back office and public portals (3 PM) n Contributions to UI and data visualization from a WP 7 perspective n WP 6 Metadata Analysis and Metadata Repository (7 PM) n Contributions to metadata modelling and metadata repository implementation © Fraunhofer IDMT

FHG: Contributions n WP 7 Components for annotation of moving images (22, 5 PM, M 1 -24) n T 7. 1 Sot. A analysis (2, 5 PM) n T 7. 2 Technology specification and implementation / adaptation (15 PM) n T 7. 3 Annotator orchestration and semi-automatic analysis (5 PM): exploit synergies between annotators n WP 8 Architecture Design and System Integration (7 PM) n Contributions to system design / architecture n Contributions to integration (incl. crawling, pre-processing, access control for storage) © Fraunhofer IDMT

Responsibilities n D 7. 1 Report on State of the Art for moving image analysis Report Public, M 4 n D 7. 2 Video analysis components Websites, patents filling, etc. Confidential, only for members of the consortium (including the Commission Services), M 24 n MS 7 Video analysis API and orchestration approach M 20: Prototype release of the video analysis components. © Fraunhofer IDMT

FHG: Extractor Components n Temporal video segmentation (TVS): detection of shots and keyframes in videos n Quality Control (QC): detection of errors and quality issues in a/v content n video segment matching: matching of video segments e. g. for cleanup and tracking purposes n audio segment matching: matching of audio segments e. g. for cleanup and tracking purposes n semantic scene classification and visual concept detection enhance/adapt 3 rd party components (see Mi. CO) © Fraunhofer IDMT

Thank you! Contact: Uwe Kühhirt uwe. kuehhirt@idmt. fraunhofer. de +49 3677 467 -205 http: //www. idmt. fraunhofer. de/m 2 d © Fraunhofer IDMT