Multimedia Digital Libraries Mohamed Kholief 4 8 99 Overview

Multimedia Digital Libraries Mohamed Kholief (4/8/99)

Overview z. Introducing Multimedia Digital Libraries. z. Problems with video digital libraries. z. Informedia project approach as an example. z. Introducing my research work.

What? z Information storage/retrieval systems that can both store a large amount of multimedia information (audio, video, images, text) in digital form, and distribute this information across global information networks. z Generally consist of media conversion subsystem, storage and retrieval sub-system, and location service sub-system. z Examples: Informedia digital video library, ISLIP digital library, Hitachi Library, . . .

Why? z Vast amount of historic literature, document archives, pictures, and films. z In the near future, computer generated graphics and simulation results will also be stored. z Useful for education and training, medical diagnostics, manufacturing and distributed publishing. z Moving images are primary source material for many disciplines, e. g. political events, psychology, literature, history.

Introduction to video libraries z. A library is not effective without some understanding of the contents. z. The providers should augment the information with rich keywords and descriptors, provide links to related information, and allow the contents of their pages to be searched and indexed. z. How does one represent video content to support content-based retrieval and manipulation?

Introduction to video libraries z The inaccessibility of the information to our primarily text-based information retrieval mechanisms is the problem. z The video likely contains an overabundance of information, conveyed in both the video signal and the audio signal. z A common practice today is to log or tag the video with keywords and other forms of structured text to identify its contents.

Introduction to video libraries z Limitations of keywords: y Manual processes are time consuming. y Manual processes are seriously incomplete. y Transcripts are inaccurate. y Text descriptors are biased by whatever predetermined structures are used to classify the video contents. y Cinematic information is complex and difficult to describe, especially for non-experts. y Text descriptors are biased by the ambiguity of natural language. z We have to go beyond this keyword approach to index the video content.

Reducing Size z. Video downloads are very time consuming. z. Review of these downloads is no better. z. We need: yfaster retrieval, and yfaster viewing. z. There must be: yvideo partitioning, and yalternative representations.

Reducing Size z. Video paragraphing: yscenes, shots, camera motions, and transitions are analogous to chapters, sections. . y. The video is broken into sequences sharing the same spatial location. y. This must be an automated process.

Reducing Size z Alternate representations for video clips: y. Creating layers of information y. Users could then cheaply review a given layer before deciding upon continuing for more expensive layers or not. y. Titles, abstracts, transcript, single image, skim. y. Problems similar to the indexing in text: xhow should they be generated? x. How can they be complete and accurate? xcan tools be used in their creation?

Improving Precision and Recall z. The utility of the DVL is in the ability of the users to get the information they need easily and efficiently. z. Recall is the proportion of relevant documents that are actually retrieved. z. Precision is the proportion of retrieved documents that are actually relevant. z. There are difficulties in indexing a video database to be general purpose.

Improving Precision and Recall z They sacrifice precision to recall z Large retrieval sets. Fast filtering is essential. z Issues with respect to searching for information are how to let the user: yquickly skim the video objects to locate sections of interest yadjust the size of the video objects returned yidentify desired video clips when multiple objects are returned.

Improving Precision and Recall z. Collapsing playback rate. y. Jumping frames may skip information. y. Accelerating playback is not suitable. y. Video and audio are a constant rate, continuous time media, the information in them is not. y. Skim contains only the important information. y. A quick and understandable review.

Improving Precision and Recall z. Returning small pieces: ycontinuous play of audio and video is a totally unacceptable browsing mechanism. y. If the optimally sized chunk of digital video is returned the review time by the user will be reduced considerably. ythe video contents need to be indexed well and sized appropriately.

Improving Precision and Recall z. Information visualization: yto enable better filtering and browsing the important features should be emphasized. y. How? the information in digital libraries should not just be retrieved but should allow for rich interaction, so that users can tailor the information into effective and memorable renderings appropriate to their needs.

The Informedia project z. Creating a digital library of text, images, videos and audio data. z. Full content retrieval. z. Integrates technologies from the fields of natural language understanding, image processing, speech recognition and video compression z. Allows a user to explore multimedia data through interactive queries.

Informedia Approach z A video's contents are conveyed in both the speech and the image. z Collaborative interaction of image, speech and natural language understanding technology is needed to successfully populate, segment, index, and search video with satisfactory recall and precision. z The sound track of each video is converted to a textual transcript using a high-quality speech recognizer.

Informedia Approach z. A language understanding system analyzes and organizes the transcript and stores it in a full-text information retrieval system. z. It also generates brief text abstracts for the videos. z. Image understanding techniques are used for segmenting video sequences by automatically locating boundaries of shots, scenes, and conversations.

Informedia Approach z. Example, text displayed in the video can be located via image processing and then added to the body of text for natural language processing. z. Example, having both a visual scene change and a change in the narrative increases the likelihood of a segment boundary.

Informedia Approach z The user browse through parallel presentations of video clips. z Image understanding techniques are used to select important segments of video to create the skim. z Scene changes, camera motion, object detection, and text detection are used in the heuristics determining what should be included in the skim. z Natural language processing selects appropriate audio.

What have we said so far? z. Introduced multimedia digital libraries. z. Introduced video digital libraries. z. Introduced the Informedia project.

My research work A Digital Library for Interactive Multimedia Collaborative Educational Applications

Interactive Multimedia Collaborative Applications z. The Internet is the communication medium. z. Participants are normally spatially distributed. z. They posses explicit interaction through the use of audio and video as well as implicit interaction through the use of some shared tools like shared whiteboards. z. Synchronous.

Interactive Multimedia Collaborative Applications z. Useful for many scientific and engineering cooperative efforts, distance learning and training, Tele-meetings, and the Internet games. z. Examples: IRI, CINECOM®, etc.

Motivation z. Increase in distance education. z. Increased popularity of IMC approach for distance education. z. Users need to review the IMC-based classes. Recording needed. z. Full playback is usually not acceptable. z. Playback starting from a certain time is not the best idea.

Motivation z. People tend to remember by some event or some content. z. Example: what did the teacher say when he opened Netscape? z. Example: what did he say when he displayed the slide titled … z. Conclusion: Event-based retrieval and Content-based retrieval are desirable.

Motivation z. Digital Libraries are useful for education. z. The current contents of educational digital libraries are traditional curricular material + documentaries. z. New research suggests that recording previous experience and the support for collaboration and re-use are important components of a digital library in education.

Motivation z. Having few jump-start collections for the students to find something quickly will maintain their motivation and engagement. z. A simple way to present information to students is always required instead of letting them search in digital libraries that might be prepared for experts.

Motivation z. Another important application of digital libraries is that they promote the re-use of the class material by the teachers. z. Saving recorded IMC educational applications and supporting event- and content-based retrieval in a digital library satisfies all the arguments stated above.

Motivation z. Supporting a set of tools that will help the different parties of the education community to search the digital library for different goals is thought to be very important. z. Using this library to compare the IMCapproach to other technology-based education schemes can also be done.

My Research Objective z Studying different issues involved in building a digital library that contains recorded IMC educational application sessions. z Developing various sets of retrieval tools for this library. These tools will use the new concept of event-based retrieval as well as content-based retrieval to facilitate the different uses of different users.

My Research Objective z. Studying the effectiveness of this digital library as an educational tool and using it to compare the IMC-approach to different technology-based education approaches like WEB-based courses.

Basic Block Diagram Teachers Re-use Tools Saved search results Researchers Special Tools Search Results Basic Search Tools IMC session Recording Digital Library Recorded IMC sessions Formatting + Metadata generation Students