Скачать презентацию Multimedia Signal Processing Content Based Image Retrieval Скачать презентацию Multimedia Signal Processing Content Based Image Retrieval

c7dd1d59ae9593c67f14cdff447eaf19.ppt

  • Количество слайдов: 67

Multimedia Signal Processing & Content. Based Image Retrieval Anastasios N. Venetsanopoulos University of Toronto Multimedia Signal Processing & Content. Based Image Retrieval Anastasios N. Venetsanopoulos University of Toronto Contact: [email protected] toronto. edu http: //www. ece. toronto. edu

OUTLINE l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) OUTLINE l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES

INTRODUCTION-1 l l l WHAT IS MULTIMEDIA? WHAT IS MULTIMEDIA PROCESSING? GOALS OF MULTIMEDIA INTRODUCTION-1 l l l WHAT IS MULTIMEDIA? WHAT IS MULTIMEDIA PROCESSING? GOALS OF MULTIMEDIA PROCESSING

INTRODUCTION-2 WHAT IS MULTIMEDIA? l l DIFFICULT TO DEFINE GENERALLY CONSISTS OF: l l INTRODUCTION-2 WHAT IS MULTIMEDIA? l l DIFFICULT TO DEFINE GENERALLY CONSISTS OF: l l l MULTIMEDIA DATA: l l MULTIMEDIA DATA INTERACTION SET MULTI-SOURCE, MULTI-TYPE, MULTI-FORMAT INTERACTION SET: l WITHOUT INTERACTIONS BETWEEN MULTIMEDIA COMPONENTS, MULTIMEDIA IS MERELY A COLLECTION OF DATA

INTRODUCTION-3 EXAMPLE: AUGMENTED REALITY CONFERENCE REAL OBJECTS VIRTUAL OBJECTS REAL SPEECH Mutimedia Data Components INTRODUCTION-3 EXAMPLE: AUGMENTED REALITY CONFERENCE REAL OBJECTS VIRTUAL OBJECTS REAL SPEECH Mutimedia Data Components COMPLEX INTERACTIONS BETWEEN COMPONENTS IN THE SCENE MAKE VIRTUAL COMPONENTS SEEM MORE REALISTIC

INTRODUCTION-4 WHAT IS MULTIMEDIA PROCESSING? l MULTIMEDIA PROCESSING l APPLY SIGNAL PROCESSING TOOLS TO INTRODUCTION-4 WHAT IS MULTIMEDIA PROCESSING? l MULTIMEDIA PROCESSING l APPLY SIGNAL PROCESSING TOOLS TO MULTIMEDIA DATA TO ENABLE: l l REPRESENTATION INTERPRETATION ENCODING DECODING

INTRODUCTION-5 GOALS OF MULTIMEDIA PROCESSING l EFFECTIVE & EFFICIENT l l ACCESS MANIPULATION EXCHANGE INTRODUCTION-5 GOALS OF MULTIMEDIA PROCESSING l EFFECTIVE & EFFICIENT l l ACCESS MANIPULATION EXCHANGE STORAGE OF MULTIMEDIA CONTENT

CONTINUING… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) CONTINUING… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES

MULTIMEDIA APPLICATIONS-1 SCALABLE VIDEO STREAMING GPS NAVIGATION MULTIMEDIA APPLICATIONS-1 SCALABLE VIDEO STREAMING GPS NAVIGATION

MULTIMEDIA APPLICATIONS-2 TELEPRESENCE CELLULAR E-COMMERCE MULTIMEDIA APPLICATIONS-2 TELEPRESENCE CELLULAR E-COMMERCE

MULTIMEDIA APPLICATIONS-3 l MORE SPECIFIC EXAMPLES v MPEG-4, 7, 21 v JPEG-2000 v MP MULTIMEDIA APPLICATIONS-3 l MORE SPECIFIC EXAMPLES v MPEG-4, 7, 21 v JPEG-2000 v MP 3 & PERCEPTUAL CODING l v MULTIMEDIA STORAGE v VIDEO-ON-DEMAND v DIGITAL CINEMA v AUTHENTICATION MULTIMEDIA APPLICATION GOALS l l IMPROVE INTERPERSONAL COMMUNICATION PROMOTE UNDERSTANDING OF IDEAS ALLOW INTERACTIVITY WITH MEDIA INCREASE ACCESSIBILITY TO DATA

GOING ON… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL GOING ON… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES

IMPACT OF MULTIMEDIA-4 WORLD INTERNET USAGE (July 23, 2005) CURRENT USERS % WORLD USERS IMPACT OF MULTIMEDIA-4 WORLD INTERNET USAGE (July 23, 2005) CURRENT USERS % WORLD USERS GROWTH (2000 -2005) PENETRATION North America 223, 392, 807 23. 8% 106. 7% 68. 0% Europe 269, 036, 096 28. 7% 161. 0% 36. 8% Asia 323, 756, 956 34. 5% 183. 2% 8. 9% Middle East 21, 770, 700 2. 3% 311. 9% 8. 3% Africa 16, 174, 600 1. 7% 258. 3% 1. 8% Latin America & Caribbean 68, 130, 804 7. 3% 277. 1% 12. 5% Oceania & Australia 33, 448 1. 8% 115. 9% 49. 2% 938, 810, 929 100% 160. 0% 14. 6% COUNTRY WORLD

IMPACT OF MULTIMEDIA-2 l USERS (S 0 CIETY) DEMAND l l l INCREASED MOBILITY IMPACT OF MULTIMEDIA-2 l USERS (S 0 CIETY) DEMAND l l l INCREASED MOBILITY EASE-OF-USE PERSONAL CUSTOMIZATION DEVICE FLEXIBILITY HIGH LEVEL OF COLLABORATION WITH PEERS DEVICES MUTATE AND BECOME l l l MULTI-FUNCTIONAL, NOT SPECIALIZED EFFORTLESSLY PORTABLE, NOT STATIONARY UBIQUITOUSLY NETWORKED, NOT ISOLATED

IMPACT OF MULTIMEDIA-3 l MULTI-FUNCTIONAL DEVICES MUST l l l FACILITATE MANY TYPES OF IMPACT OF MULTIMEDIA-3 l MULTI-FUNCTIONAL DEVICES MUST l l l FACILITATE MANY TYPES OF WORKFLOW MANAGE USER’S TIME CUSTOMIZATION l l BROWSE INTERNET ENTERTAIN BE EASY-TO-USE PERSONALIZATION (THEMES, PREFERENCES) NETWORKED l CAPABLE OF CONNECTING TO MANY DIFFERENT NETWORKS (INTERNET, P. O. T. S. , LAN, CELLULAR, BLUETOOTH, 802. 11 b, GPS)

IMPACT OF MULTIMEDIA-4 CONVERGENCE TECHNOLOGIES WHICH WERE TOTALLY UNRELATED 10 YEARS AGO ARE NOW IMPACT OF MULTIMEDIA-4 CONVERGENCE TECHNOLOGIES WHICH WERE TOTALLY UNRELATED 10 YEARS AGO ARE NOW UNIFIED UNDER THE CONCEPT OF MULTIMEDIA

IMPACT OF MULTIMEDIA-5 l EXAMPLE: CELLULAR PHONES PRIMARY CONSUMER USE: l WIRELESS TELEPHONY CONVERGED IMPACT OF MULTIMEDIA-5 l EXAMPLE: CELLULAR PHONES PRIMARY CONSUMER USE: l WIRELESS TELEPHONY CONVERGED USES l PERSONAL ORGANIZER l INTERNET BROWSER/EMAIL l ENTERTAINMENT (MP 3, RADIO) l l PAGER/MESSAGING (SMS) VIDEO/STILL CAMERA

IMPACT OF MULTIMEDIA-6 OVERALL l DEMANDS l l l FUNCTIONALITY CONSUMPTION OF MANY MEDIA IMPACT OF MULTIMEDIA-6 OVERALL l DEMANDS l l l FUNCTIONALITY CONSUMPTION OF MANY MEDIA TYPES CONNECTIVITY PORTABILITY, ETC. RESULT l l HIGHLY COMPLEX DEVICES PUSH TOWARDS DENSE CIRCUITRY MULTIMEDIA DEVICES BECOME UBIQUITOUS DEVICES GENERATE MULTIMEDIA DATA (INCLUDING IMAGES, VIDEO, AUDIO)

MOVING ALONG… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL MOVING ALONG… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES

CBIR OVERVIEW l l l MOTIVATION & GOALS WHAT IS CBIR? CONTRIBUTING DISCIPLINES APPLICATION CBIR OVERVIEW l l l MOTIVATION & GOALS WHAT IS CBIR? CONTRIBUTING DISCIPLINES APPLICATION SCENARIOS SOME SPECIFIC ISSUES TYPICAL CAPABILITIES

CBIR MEDIA FLOODING EXAMPLE: GENERAL PHOTOGRAPHY l POLAROID FILED FOR BANKRUPTCY l HAS DIGITAL CBIR MEDIA FLOODING EXAMPLE: GENERAL PHOTOGRAPHY l POLAROID FILED FOR BANKRUPTCY l HAS DIGITAL KILLED FILM? IF SO, WHY? Ø Ø MEMORY Ø l SNAPSHOT PREVIEWS PRINTER TECHNOLOGY Ø REUSABLE EASY SHARING VIA INTERNET Ø CHEAP & DENSE STORAGE Ø EFFECTS & PROCESSING RESULT: DIGITAL MEDIA FLOOD l HOW DO WE COPE, TRACK, ORGANIZE IT ALL?

CBIR MOTIVATION l DEVICE FUNCTION CONVERGENCE l l DATA RAPIDLY GENERATED BY MANY DEVICES CBIR MOTIVATION l DEVICE FUNCTION CONVERGENCE l l DATA RAPIDLY GENERATED BY MANY DEVICES INTERNET ACTS AS GLOBAL TRANSPORT DATA CONSUMED BY DEVICES ON DEMAND MULTIMEDIA DATA NEEDS TO BE l l l EFFICIENTLY STORED INDEXED ACCURATELY EASILY RETRIEVED

CBIR IS… l l CONTENT BASED IMAGE RETRIEVAL PART OF MULTIMEDIA INDEXING l l CBIR IS… l l CONTENT BASED IMAGE RETRIEVAL PART OF MULTIMEDIA INDEXING l l l l IMAGES (2 -D SPACE-DEPENDENT SIGNALS) VIDEO (TIME-VARYING IMAGE SET) AUDIO (1 -D TIME-DEPENDENT SIGNALS) TEXT (e. g. BOOK INDEX, SEARCH ENGINES) COMPUTER BASED HIGHLY AUTOMATED DIFFICULT TO DO PROPERLY

CBIR SIMPLE EXAMPLE l FOR A GIVEN QUERY… l l l EXAMPLE IMAGE ROUGH CBIR SIMPLE EXAMPLE l FOR A GIVEN QUERY… l l l EXAMPLE IMAGE ROUGH SKETCH EXPLICIT DESCRIPTION CRITERIA …RETURN ALL ‘SIMILAR’ IMAGES RETRIEVAL SYSTEM QUERY IMAGE RETRIEVAL RESULTS BASED ON COLOR CONTENT

CBIR QUERY TYPES COLOR SKETCH SHAPE EXAMPLE TEXTURE MORE COMPLEX TYPES EXIST YET ABOVE CBIR QUERY TYPES COLOR SKETCH SHAPE EXAMPLE TEXTURE MORE COMPLEX TYPES EXIST YET ABOVE ARE MOST FUNDAMENTAL & MOST REGULARLY USED

CBIR CONTRIBUTORS l COMBINES HIGH-TECH ELEMENTS l MULTIMEDIA/SIGNAL/IMAGE PROCESSING l COMPUTER VISION/PATTERN RECOGNITION l CBIR CONTRIBUTORS l COMBINES HIGH-TECH ELEMENTS l MULTIMEDIA/SIGNAL/IMAGE PROCESSING l COMPUTER VISION/PATTERN RECOGNITION l COMPUTER SCIENCES (I. E. HUMAN-COMPUTER INTERACTION) l AND MORE TRADITIONAL CONCEPTS l PSYCHOLOGY/HUMAN PERCEPTION l INFORMATION SCIENCES (I. E. LIBRARY)

CBIR SCENARIOS l SOME CBIR APPLICATION AREAS l MEDICAL IMAGING ART/CULTURAL HERITAGE l a CBIR SCENARIOS l SOME CBIR APPLICATION AREAS l MEDICAL IMAGING ART/CULTURAL HERITAGE l a l a DESIGN/VISUAL ARTS ENTERTAINMENT (FILM, TV) l l INDUSTRY (LOGO MANAGEMENT) GOVERNMENT (E. G. MUGSHOTS)

CBIR VERSUS TEXT l IMPORTANT QUESTION ARISES: “WHY NOT SIMPLY INDEX USING TEXT? ” CBIR VERSUS TEXT l IMPORTANT QUESTION ARISES: “WHY NOT SIMPLY INDEX USING TEXT? ” (YAHOO! HAS HAD SOME SUCCESS WITH THIS) l INTUITIVE, YET USING TEXT IS l SIMPLE BUT SIMPLISTIC l TIME CONSUMING – CAN’T AUTOMATE l HIGHLY SUBJECTIVE & USER-DEPENDENT l SUSCEPTIBLE TO TRANSLATION PROBLEMS

CBIR BASIC STRUCTURE l FEATURE EXTRACTION l FEATURE DESCRIPTIONS I N D E X CBIR BASIC STRUCTURE l FEATURE EXTRACTION l FEATURE DESCRIPTIONS I N D E X SIMILARITY CALCULATION l MANY DESCRIPTORS SIMILAR RESULTS MPEG-7 IS ISO STANDARD l GENERATION OF RESULTS USER INTERFACE COLOR, TEXTURE, SHAPE l l QUERY 3 BASIC FEATURES REALLY A DESIGN CHOICE SIMILARITY l l OPEN TO RESEARCH LITTLE PERCEPTUAL CONSIDERATION

CBIR (DIS)SIMILARITY? SIMILARITY IS NOT SO SIMPLE l CONSIDER THREE IMAGES l ON WHAT CBIR (DIS)SIMILARITY? SIMILARITY IS NOT SO SIMPLE l CONSIDER THREE IMAGES l ON WHAT BASIS ARE THEY SIMILAR? l l COLOR CONTENT? SHAPE CONTENT? HIGH LEVEL IDEAS (‘MASKS’, ‘GENDER’)? PERCEPTION IS ALWAYS AN ISSUE

CBIR SIMILARITY l DOMAIN [0, 1] l CAN BE CALCULATED MANY WAYS l GENERALIZED CBIR SIMILARITY l DOMAIN [0, 1] l CAN BE CALCULATED MANY WAYS l GENERALIZED MINKOWSKI l CANBERRA l PERCEPTUAL MEASURE

CBIR TYPICAL ABILITIES l EFFECTIVE QUERIES IN l l SIMPLE HYBRID QUERIES l l CBIR TYPICAL ABILITIES l EFFECTIVE QUERIES IN l l SIMPLE HYBRID QUERIES l l l COLOR, TEXTURE, SHAPE DESCRIPTOR SUPERVECTORS WEIGHTED AVERAGE OF (DIS)SIMILARITIES RELEVANCE FEEDBACK l l l USER PLACED IN LOOP GIVES BETTER RESULTS STATISTICAL APPROACHES APPLY/ADJUST FEATURE WEIGHTS TO RELEVANT/IRRELEVANT ELEMENTS

CBIR SUMMARY l l l BORN FROM MULTIMEDIA FLOOD TEXT TOO SIMPLE AND LABORIOUS CBIR SUMMARY l l l BORN FROM MULTIMEDIA FLOOD TEXT TOO SIMPLE AND LABORIOUS SYSTEMS WORK DECENTLY IN VITRO l l SHORTCOMINGS l l QUERY BY SHAPE, COLOR, TEXTURE, EXAMPLE NEED RELEVANCE FEEDBACK & PERCEPTUAL HYBRID QUERIES DIFFICULT TO CREATE SEMANTIC GAP NEEDS TO BE BRIDGED MPEG-7: IMPORTANT DEVELOPMENT

GOING FORWARD… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL GOING FORWARD… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES

MPEG l l l MOTION PICTURES EXPERT GROUP MPEG-1 MPEG-2 MPEG-4 MPEG-7: ISO/IEC 15938 MPEG l l l MOTION PICTURES EXPERT GROUP MPEG-1 MPEG-2 MPEG-4 MPEG-7: ISO/IEC 15938 l l MULTIMEDIA CONTENT DESCRIPTION INTERFACE MPEG-21

MPEG-1 & MPEG-2 l MPEG-1 (c. 1992) l l BASIC VIDEO CODING USING DPCM MPEG-1 & MPEG-2 l MPEG-1 (c. 1992) l l BASIC VIDEO CODING USING DPCM & DCT TARGET: CD-BASED VIDEO & MULTIMEDIA USE I, B & P-FRAMES IN YUV SPACE MPEG-2 (c. 1994) l l l SUPERSET OF MPEG-1 GOAL: DTV/DSS OR ATM TRANSPORT MINIMUM OF NTSC/PAL QUALITY MORE ERROR RESILIENT SCALABLE – GRACEFUL DEGRADATION

MPEG-4 & MPEG-21 l MPEG-4 (c. 1998) l l l TOOLS TO AUTHOR MULTIMEDIA MPEG-4 & MPEG-21 l MPEG-4 (c. 1998) l l l TOOLS TO AUTHOR MULTIMEDIA CONTENT TRAFFIC AWARE, ERROR RESILIENT OBJECT-BASED CODING VERY EFFICIENT FOR LOW BIT-RATES MPEG-21 (STARTED JUNE 2000) l l l AN OPEN “MULTIMEDIA FRAMEWORK” IDEA ADDRESSES DIGITAL RIGHTS MANAGEMENT ENHANCED DELIVERY & ACCESS OF DATA FOR DEVICES ON HETEROGENEOUS NETWORKS

MPEG-7 NEW PARADIGM l UNLIKE MPEG-1, MPEG-2, & MPEG-4 l l APPLICABLE TO l MPEG-7 NEW PARADIGM l UNLIKE MPEG-1, MPEG-2, & MPEG-4 l l APPLICABLE TO l l l DOESN’T REPRESENT CONTENT ITSELF MPEG-7 ONLY DESCRIBES CONTENT DIFFICULT CONCEPT FOR SOME TO GRASP IMAGES VIDEO l l AUDIO & SPEECH TEXT INDEPENDENT OF l l STORAGE ARCHITECTURE l l TRANSPORT CODING

MPEG-7 HOW IT DIFFERS l MPEG-1 l l TAKES INPUT FRAMES AND REPRESENTS AS MPEG-7 HOW IT DIFFERS l MPEG-1 l l TAKES INPUT FRAMES AND REPRESENTS AS AN BINARY ENCODED VIDEO BITSTREAM MPEG-7 l TAKES VIDEO FRAMES (SAY MPEG-1 FORMAT) AND DESCRIBES CONTENTS OF EACH FRAME 1: COLOR CONTENT: 20% WHITE, 14% BLUE, SHAPES: BRIDGE, etc. FRAME 2: COLOR CONTENT: 20% WHITE, 15% BLUE, SHAPES: BRIDGE, etc. FRAME 3: COLOR CONTENT: 21% WHITE, 14% BLUE, SHAPES: BRIDGE, etc.

MPEG-7 SCOPE MULTIMEDIA DATA FEATURE EXTRACTION ALGORITHM CONTENT DESCRIPTION MPEG-7 SCOPE CODING SCHEME OTHER MPEG-7 SCOPE MULTIMEDIA DATA FEATURE EXTRACTION ALGORITHM CONTENT DESCRIPTION MPEG-7 SCOPE CODING SCHEME OTHER ELEMENTS. . .

MPEG-7 GOALS l DESCRIBE MULTIMEDIA CONTENT l l RELATIONS BETWEEN DESCRIPTORS l l SET MPEG-7 GOALS l DESCRIBE MULTIMEDIA CONTENT l l RELATIONS BETWEEN DESCRIPTORS l l SET OF DESCRIPTION SCHEMES (DS) LANGUAGE DEFINING D’s & DS’s l l SET OF DESCRIPTORS (D) DESCRIPTION DEFINITION LANGUAGE (DDL) BASED ON XML (e. Xtensible Markup Language) USED TO BUILD UP NEW D’s & DS’s ENCODING OF D’s FOR EFFICIENCY

MPEG-7 SUMMARY-1 l l STANDARDIZED DESCRIPTIONS APPLIES TO ALL DIGITAL MEDIA l l DOES MPEG-7 SUMMARY-1 l l STANDARDIZED DESCRIPTIONS APPLIES TO ALL DIGITAL MEDIA l l DOES NOT REPRESENT DATA ITSELF l l CBIR IS CASE FOR STILL IMAGES DESCRIBES WHAT DATA REPRESENTS SETS THE BAR FOR SYSTEMS l MULTIMEDIA/IMAGE RETRIEVAL SYSTEMS NEED AT LEAST MPEG-7 CONFORMANCE

MPEG-7 SUMMARY-2 l DOES NOT ADDRESS l l l SIMILARITY RELEVANCE FEEDBACK FEATURE EXTRACTION MPEG-7 SUMMARY-2 l DOES NOT ADDRESS l l l SIMILARITY RELEVANCE FEEDBACK FEATURE EXTRACTION HYBRID QUERY GENERATION ARCHIVE ORGANIZATION THE ABOVE ISSUES HAVE BEEN PURPOSEFULLY LEFT OPEN FOR INNOVATION

FORGING AHEAD… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL FORGING AHEAD… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES

RESEARCH ISSUES l SHORTCOMINGS OF CBIR SYSTEMS l ONGOING RESEARCH l l RELEVANCE FEEDBACK RESEARCH ISSUES l SHORTCOMINGS OF CBIR SYSTEMS l ONGOING RESEARCH l l RELEVANCE FEEDBACK HYBRID QUERY GENERATION DISTRIBUTED MULTIMEDIA INDEXING OPEN RESEARCH AVENUES

CBIR SHORTCOMINGS-1 l COLOR l l SHAPE l l l USUALLY GLOBAL HIGH DIMENSIONALITY CBIR SHORTCOMINGS-1 l COLOR l l SHAPE l l l USUALLY GLOBAL HIGH DIMENSIONALITY GAMMA NONLINEARITIES CAUSE PROBLEMS COMPLICATED & DIFFICULT OCCLUSION ISSUES DURING EXTRACTION TEXTURE l l COMPLICATED & UNINTUITIVE USER-SYSTEM RIFT FOR QUERY CREATION

CBIR SHORTCOMINGS-2 l PERCEPTUAL ISSUES l l l SIMILARITY MEASURES l l l SUBTLE CBIR SHORTCOMINGS-2 l PERCEPTUAL ISSUES l l l SIMILARITY MEASURES l l l SUBTLE DIFFERENCES BETWEEN VIEWERS COLOR-BLIND USERS NEED TO BE TUNED TO DESCRIPTORS e. g. EUCLIDEAN DISTANCE NOT APPLICABLE IN NON-EUCLIDEAN DESCRIPTION SPACE RELEVANCE FEEDBACK l l PERFORMED AT GLOBAL (IMAGE) LEVEL NEED TO ADDRESS SPECIFIC IMAGE ELEMENTS

ONGOING RESEARCH -2 RELEVANCE FEEDBACK l ITERATIVE QUERY REFINEMENT l l PLACE USER IN ONGOING RESEARCH -2 RELEVANCE FEEDBACK l ITERATIVE QUERY REFINEMENT l l PLACE USER IN LOOP TO ITERATIVELY IMPROVE RETRIEVAL RATES HIGH-DIMENSIONAL SPACE NEEDS PRUNING EMPHASIZED FEATURE(S) MUST BE FOUND TYPICAL APPROACHES l l STATISTICAL METHODS FEATURE WEIGHTING

ONGOING RESEARCH -2 RELEVANCE FEEDBACK l FEATURE SELECTIVE INTERFACE l l WHY CHOOSE IMAGES ONGOING RESEARCH -2 RELEVANCE FEEDBACK l FEATURE SELECTIVE INTERFACE l l WHY CHOOSE IMAGES ON WHOLE? REQUIRES PROCESSING/STATS TO FIND GOOD FEATURES USER CAN EXPLICITLY INDICATE ELEMENTS OF IMAGE WHICH ARE GOOD: NO GUESSWORK RELEVANT SHAPE EXPLICIT FEATURES TO R. F. ENGINE RELEVANT COLOR

ONGOING RESEARCH -3 SIMILARITY AGGREGATION/HYBRID QUERIES l TYPICALLY USED APPROACHES l l BOOLEAN (AND, ONGOING RESEARCH -3 SIMILARITY AGGREGATION/HYBRID QUERIES l TYPICALLY USED APPROACHES l l BOOLEAN (AND, OR & NOT OPERATORS) EUCLIDEAN (MINKOWSKI W/ r=1) WEIGHTED AVERAGE (WA) i. e. SUPERVECTORS DISADVANTAGES l l EUCLIDEAN: FCN OF DESCRIPTORS – CHANGE DESCRIPTOR, DRASTICALLY ALTER MEASURE WA: INFLEXIBLE FOR HIGH LEVEL QUERIES, SUPERVECTORS IMPOSE CERTAIN STRUCTURE BOOLEAN: HARD LIMITED TO LOGIC FCNs ALL LACK PERCEPTUAL CONSIDERATIONS

ONGOING RESEARCH-4 SIMILARITY AGGREGATION/HYBRID QUERIES l FUZZY AGGREGATION OF DECISIONS l USE MEMBERSHIP FUNCTION ONGOING RESEARCH-4 SIMILARITY AGGREGATION/HYBRID QUERIES l FUZZY AGGREGATION OF DECISIONS l USE MEMBERSHIP FUNCTION TO ‘FUZZIFY’ DISTANCES & GENERATE A ‘FUZZY DECISION’ l EXPONENTIAL MODELS HUMAN PERCEPTION d SIMILARITY DISTANCE FUZZY MEMBERSHIP FUNCTION m FUZZY DISTANCE DECISION

ONGOING RESEARCH-5 DISTRIBUTED MULTIMEDIA INDEXING l INDEXES USUALLY CENTRALIZED l l ENTIRE SYSTEM FAILS ONGOING RESEARCH-5 DISTRIBUTED MULTIMEDIA INDEXING l INDEXES USUALLY CENTRALIZED l l ENTIRE SYSTEM FAILS IF COMPONENT FAILS NO GRACEFUL PERFORMANCE DEGRADATION HIGH DATA VOLUME = HIGH SYSTEM REQ’S DISTRIBUTED INDEXES l l SPREAD WORKLOAD OVER MANY SUBSYSTEMS INCREASE REDUNDANCY P 2 P SYSTEMS LACK CENTRALIZED ELEMENTS P 2 P SYSTEMS RESEMBLE SOCIAL NETWORKS

ONGOING RESEARCH-6 DISTRIBUTED MULTIMEDIA INDEXING l l SMALL WORLD INDEXING MODEL 1 SOCIOLOGICAL PEER ONGOING RESEARCH-6 DISTRIBUTED MULTIMEDIA INDEXING l l SMALL WORLD INDEXING MODEL 1 SOCIOLOGICAL PEER DESCRIPTIONS l l l WE ARE NOT BLIND TO WHO OUR PEERS ARE PEOPLE KEEP MEMORY OF THEIR PEERS WE ARE NOT BLIND TO HOW OUR PEERS ARE l l WE REFER OTHERS TO OUR PEERS EXAMPLE [1] P. Androutsos, D. Androutsos and A. N. Venetsanopoulos, “A distributed fault-tolerant MPEG-7 retrieval scheme based on small world theory”, Distributed Media Technologies and Applications Special Issue of IEEE Transactions on Multimedia, under review.

ONGOING RESEARCH-7 DISTRIBUTED MULTIMEDIA INDEXING l INDEX AND ARCHIVE BECOME ONE l l SWIM ONGOING RESEARCH-7 DISTRIBUTED MULTIMEDIA INDEXING l INDEX AND ARCHIVE BECOME ONE l l SWIM DATA STORED IN ARCHIVE OBJECTS EACH DATA OBJECT BEHAVES AS OWN AGENTS ARE EFFECTIVE IN HIGHLY NETWORKED ENVIRONMENTS (SWIM) RETRIEVALS l l l AGENT BASED RETRIEVAL USE OF REFERRAL BASED TECHNIQUE SIMILAR TO ‘SIX DEGREES OF SEPARATION’ CURRENTLY PERFORMED WITH IMAGES

ONGOING RESEARCH-8 DISTRIBUTED MULTIMEDIA INDEXING 2 [2] P. Androutsos, D. Androutsos and A. N. ONGOING RESEARCH-8 DISTRIBUTED MULTIMEDIA INDEXING 2 [2] P. Androutsos, D. Androutsos and A. N. Venetsanopoulos, “Graceful image retrieval performance degradation using small world distributed indexing”, International Conference on Image Processing ICIP 2005, Genoa, Italy.

RESEARCH AVENUES-1 l HYBRID QUERIES & AGGREGATION l l WHAT DO WEIGHTS MEAN? HOW RESEARCH AVENUES-1 l HYBRID QUERIES & AGGREGATION l l WHAT DO WEIGHTS MEAN? HOW TO CHOOSE? ALTERNATIVE AGGREGATIONS METHODS ADAPTIVE SCHEMES USING REL. FEEDBACK USER INTERFACE l l BRIDGE SEMANTIC GAP BETWEEN USER’S IDEA, AND ABILITY TO EXPRESS AS A QUERY ALTERNATIVE INTERFACES–ICONIC, SEMANTIC

RESEARCH AVENUES-2 l PERCEPTUAL ISSUES l l l EMPHASIS OF DOMINATING FEATURES FEATURE MASKING RESEARCH AVENUES-2 l PERCEPTUAL ISSUES l l l EMPHASIS OF DOMINATING FEATURES FEATURE MASKING EMOTIONAL INDEXING/ ALL USERS DIFFERENT–CUSTOMIZED PROFILE ARCHIVE DEPENDENCE l l SYSTEMS USUALLY SPECIALIZED ADAPTIVE INDEXING – MOST APPROPRIATE SYSTEM USED BASED ON PRELIMINARY SURVEY OF CANDIDATE DATABASE

RESEARCH AVENUES-3 l DISTRIBUTED INDEXING l l l INCORPORATE TEXT METHODS l l l RESEARCH AVENUES-3 l DISTRIBUTED INDEXING l l l INCORPORATE TEXT METHODS l l l DISTRIBUTED INDEXES & RETRIEVAL INDEX SYNCHRONIZATION RESULTS ORGANIZATION & RANKING SWIM OVERHEAD ESTIMATION EXTENSION OF SWIM TO OTHER DATA TYPES TEXT-INDEXING USING LIMITED VOCABULARY DON’T REJECT BUT USE INTELLIGENTLY EXTEND TO MPEG-21 & METADATA

SUMMARY-1 l MULTIMEDIA PROCESSING l l RESULTS FROM MULTIMEDIA EXPLOSION USERS DEMANDING MORE FROM SUMMARY-1 l MULTIMEDIA PROCESSING l l RESULTS FROM MULTIMEDIA EXPLOSION USERS DEMANDING MORE FROM DEVICES ARE CONVERGING CONTENT BASED IMAGE RETRIEVAL l l l NECESSARY TO TRACK VISUAL SEA OF DATA GOOD CAPABILITIES, BUT W/ SHORTCOMINGS PERCEPTUAL/SUBJECTIVE ISSUES RELEVANCE FEEDBACK DISTRIBUTED CONCEPTS BECOMING CRITICAL

SUMMARY-2 l MPEG-7 l l l AIMED AT STANDARDIZING DESCRIPTIONS RADICALLY DIFFERENT THAN PREVIOUS SUMMARY-2 l MPEG-7 l l l AIMED AT STANDARDIZING DESCRIPTIONS RADICALLY DIFFERENT THAN PREVIOUS MPEGs DDL IS AN EXTENSION OF XML SCHEMA APPLICABLE TO ALL MULTIMEDIA DATA ALWAYS MORE TO DO l l l MPEG-7 HAS LEFT MANY ISSUES OPEN CBIR NEEDS TO ADDRESS USERS, PERCEPTION, HYBRID QUERIES, DISTRIBUTED SYSTEMS, ETC VIBRANT RESEARCH COMMUNITY

THANK YOU THANK YOU

IMPACT OF MULTIMEDIA l HIGH FLEXIBILITY RESULTS IN l l MANY TYPES OF NETWORKS IMPACT OF MULTIMEDIA l HIGH FLEXIBILITY RESULTS IN l l MANY TYPES OF NETWORKS CAUSE l l l RISE IN DATA GENERATION & STORAGE INCREASE IN BANDWIDTH NEEDS ONE TOOL DOING WORK OF MANY COMPLEX HARDWARE COMBINATIONS ONE DEVICE CONNECTING TO ALL NETWORKS SMALL, PORTABLE DEVICES l l MINIATURIZATED WITH HUGE CAPABILITIES ONE DEVICE REPLACES MANY

CBIR WHO’S WHO CBIR WHO’S WHO

MPEG-7 D, DS, & DDL BUILDING MORE Ds & DSs USING THE DDL DS MPEG-7 D, DS, & DDL BUILDING MORE Ds & DSs USING THE DDL DS DS D D D DEFINED IN MPEG-7 STANDARD DEFINED VIA DDL

MPEG-7 COMPONENTS l l l SYSTEMS DDL VISUAL l l l PRIMARY CONCERN FOR MPEG-7 COMPONENTS l l l SYSTEMS DDL VISUAL l l l PRIMARY CONCERN FOR THIS PRESENTATION AUDIO MULTIMEDIA DESCRIPTION SCHEMES EXPERIMENTATION MODEL (XM) CONFORMANCE

MPEG-7 VISUAL COMPONENT HIGHLIGHTED DESCRIPTORS USED BY Uof. T l BASIC DESCRIPTORS l l MPEG-7 VISUAL COMPONENT HIGHLIGHTED DESCRIPTORS USED BY Uof. T l BASIC DESCRIPTORS l l l COLOR SPACE COLOR QUANTIZATION DOMINANT COLOR SCALABLE COLOR STRUCTURE COLOR LAYOUT Go. F/Go. P COLOR OTHER l FACE RECOGNITION TEXTURE DESCRIPTORS l l l l REGION-BASED CONTOUR-BASED 3 D SHAPE MOTION DESCRIPTORS l l l EDGE HISTOGRAM HOMOGENEOUS TEXTURE BROWSING SHAPE DESCRIPTORS l COLOR DESCRIPTORS l l GRID LAYOUT 2 D/3 D VIEW TIME SERIES SPATIAL 2 D COORDS TEMPORAL INTERPOLATION l CAMERA MOTION TRAJECTORY PARAMETRIC MOTION ACTIVITY LOCALIZATION l l SPATIO-TEMPORAL REGION LOCATOR

ONGOING RESEARCH SIMILARITY AGGREGATION/HYBRID QUERIES l FUZZY AGGREGATION OF DECISIONS l l AGGREGATE DECISIONS ONGOING RESEARCH SIMILARITY AGGREGATION/HYBRID QUERIES l FUZZY AGGREGATION OF DECISIONS l l AGGREGATE DECISIONS USING LOGIC USE COMPENSATIVE OPERATOR PARAMETER g CONTROLS DEGREE OF ANDNESS (max) & ORNESS (min) RESULT IS A SINGLE VALUE IN [0, 1] INDICATING OVERALL IMAGE SIMILARITY