Multimedia Signal Processing & Content. Based Image Retrieval Anastasios N. Venetsanopoulos University of Toronto Contact: anv@dsp. toronto. edu http: //www. ece. toronto. edu
OUTLINE l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES
INTRODUCTION-1 l l l WHAT IS MULTIMEDIA? WHAT IS MULTIMEDIA PROCESSING? GOALS OF MULTIMEDIA PROCESSING
INTRODUCTION-2 WHAT IS MULTIMEDIA? l l DIFFICULT TO DEFINE GENERALLY CONSISTS OF: l l l MULTIMEDIA DATA: l l MULTIMEDIA DATA INTERACTION SET MULTI-SOURCE, MULTI-TYPE, MULTI-FORMAT INTERACTION SET: l WITHOUT INTERACTIONS BETWEEN MULTIMEDIA COMPONENTS, MULTIMEDIA IS MERELY A COLLECTION OF DATA
INTRODUCTION-3 EXAMPLE: AUGMENTED REALITY CONFERENCE REAL OBJECTS VIRTUAL OBJECTS REAL SPEECH Mutimedia Data Components COMPLEX INTERACTIONS BETWEEN COMPONENTS IN THE SCENE MAKE VIRTUAL COMPONENTS SEEM MORE REALISTIC
INTRODUCTION-4 WHAT IS MULTIMEDIA PROCESSING? l MULTIMEDIA PROCESSING l APPLY SIGNAL PROCESSING TOOLS TO MULTIMEDIA DATA TO ENABLE: l l REPRESENTATION INTERPRETATION ENCODING DECODING
INTRODUCTION-5 GOALS OF MULTIMEDIA PROCESSING l EFFECTIVE & EFFICIENT l l ACCESS MANIPULATION EXCHANGE STORAGE OF MULTIMEDIA CONTENT
CONTINUING… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES
MULTIMEDIA APPLICATIONS-1 SCALABLE VIDEO STREAMING GPS NAVIGATION
MULTIMEDIA APPLICATIONS-2 TELEPRESENCE CELLULAR E-COMMERCE
MULTIMEDIA APPLICATIONS-3 l MORE SPECIFIC EXAMPLES v MPEG-4, 7, 21 v JPEG-2000 v MP 3 & PERCEPTUAL CODING l v MULTIMEDIA STORAGE v VIDEO-ON-DEMAND v DIGITAL CINEMA v AUTHENTICATION MULTIMEDIA APPLICATION GOALS l l IMPROVE INTERPERSONAL COMMUNICATION PROMOTE UNDERSTANDING OF IDEAS ALLOW INTERACTIVITY WITH MEDIA INCREASE ACCESSIBILITY TO DATA
GOING ON… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES
IMPACT OF MULTIMEDIA-4 WORLD INTERNET USAGE (July 23, 2005) CURRENT USERS % WORLD USERS GROWTH (2000 -2005) PENETRATION North America 223, 392, 807 23. 8% 106. 7% 68. 0% Europe 269, 036, 096 28. 7% 161. 0% 36. 8% Asia 323, 756, 956 34. 5% 183. 2% 8. 9% Middle East 21, 770, 700 2. 3% 311. 9% 8. 3% Africa 16, 174, 600 1. 7% 258. 3% 1. 8% Latin America & Caribbean 68, 130, 804 7. 3% 277. 1% 12. 5% Oceania & Australia 33, 448 1. 8% 115. 9% 49. 2% 938, 810, 929 100% 160. 0% 14. 6% COUNTRY WORLD
IMPACT OF MULTIMEDIA-2 l USERS (S 0 CIETY) DEMAND l l l INCREASED MOBILITY EASE-OF-USE PERSONAL CUSTOMIZATION DEVICE FLEXIBILITY HIGH LEVEL OF COLLABORATION WITH PEERS DEVICES MUTATE AND BECOME l l l MULTI-FUNCTIONAL, NOT SPECIALIZED EFFORTLESSLY PORTABLE, NOT STATIONARY UBIQUITOUSLY NETWORKED, NOT ISOLATED
IMPACT OF MULTIMEDIA-3 l MULTI-FUNCTIONAL DEVICES MUST l l l FACILITATE MANY TYPES OF WORKFLOW MANAGE USER’S TIME CUSTOMIZATION l l BROWSE INTERNET ENTERTAIN BE EASY-TO-USE PERSONALIZATION (THEMES, PREFERENCES) NETWORKED l CAPABLE OF CONNECTING TO MANY DIFFERENT NETWORKS (INTERNET, P. O. T. S. , LAN, CELLULAR, BLUETOOTH, 802. 11 b, GPS)
IMPACT OF MULTIMEDIA-4 CONVERGENCE TECHNOLOGIES WHICH WERE TOTALLY UNRELATED 10 YEARS AGO ARE NOW UNIFIED UNDER THE CONCEPT OF MULTIMEDIA
IMPACT OF MULTIMEDIA-5 l EXAMPLE: CELLULAR PHONES PRIMARY CONSUMER USE: l WIRELESS TELEPHONY CONVERGED USES l PERSONAL ORGANIZER l INTERNET BROWSER/EMAIL l ENTERTAINMENT (MP 3, RADIO) l l PAGER/MESSAGING (SMS) VIDEO/STILL CAMERA
IMPACT OF MULTIMEDIA-6 OVERALL l DEMANDS l l l FUNCTIONALITY CONSUMPTION OF MANY MEDIA TYPES CONNECTIVITY PORTABILITY, ETC. RESULT l l HIGHLY COMPLEX DEVICES PUSH TOWARDS DENSE CIRCUITRY MULTIMEDIA DEVICES BECOME UBIQUITOUS DEVICES GENERATE MULTIMEDIA DATA (INCLUDING IMAGES, VIDEO, AUDIO)
MOVING ALONG… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES
CBIR OVERVIEW l l l MOTIVATION & GOALS WHAT IS CBIR? CONTRIBUTING DISCIPLINES APPLICATION SCENARIOS SOME SPECIFIC ISSUES TYPICAL CAPABILITIES
CBIR MEDIA FLOODING EXAMPLE: GENERAL PHOTOGRAPHY l POLAROID FILED FOR BANKRUPTCY l HAS DIGITAL KILLED FILM? IF SO, WHY? Ø Ø MEMORY Ø l SNAPSHOT PREVIEWS PRINTER TECHNOLOGY Ø REUSABLE EASY SHARING VIA INTERNET Ø CHEAP & DENSE STORAGE Ø EFFECTS & PROCESSING RESULT: DIGITAL MEDIA FLOOD l HOW DO WE COPE, TRACK, ORGANIZE IT ALL?
CBIR MOTIVATION l DEVICE FUNCTION CONVERGENCE l l DATA RAPIDLY GENERATED BY MANY DEVICES INTERNET ACTS AS GLOBAL TRANSPORT DATA CONSUMED BY DEVICES ON DEMAND MULTIMEDIA DATA NEEDS TO BE l l l EFFICIENTLY STORED INDEXED ACCURATELY EASILY RETRIEVED
CBIR IS… l l CONTENT BASED IMAGE RETRIEVAL PART OF MULTIMEDIA INDEXING l l l l IMAGES (2 -D SPACE-DEPENDENT SIGNALS) VIDEO (TIME-VARYING IMAGE SET) AUDIO (1 -D TIME-DEPENDENT SIGNALS) TEXT (e. g. BOOK INDEX, SEARCH ENGINES) COMPUTER BASED HIGHLY AUTOMATED DIFFICULT TO DO PROPERLY
CBIR SIMPLE EXAMPLE l FOR A GIVEN QUERY… l l l EXAMPLE IMAGE ROUGH SKETCH EXPLICIT DESCRIPTION CRITERIA …RETURN ALL ‘SIMILAR’ IMAGES RETRIEVAL SYSTEM QUERY IMAGE RETRIEVAL RESULTS BASED ON COLOR CONTENT
CBIR QUERY TYPES COLOR SKETCH SHAPE EXAMPLE TEXTURE MORE COMPLEX TYPES EXIST YET ABOVE ARE MOST FUNDAMENTAL & MOST REGULARLY USED
CBIR CONTRIBUTORS l COMBINES HIGH-TECH ELEMENTS l MULTIMEDIA/SIGNAL/IMAGE PROCESSING l COMPUTER VISION/PATTERN RECOGNITION l COMPUTER SCIENCES (I. E. HUMAN-COMPUTER INTERACTION) l AND MORE TRADITIONAL CONCEPTS l PSYCHOLOGY/HUMAN PERCEPTION l INFORMATION SCIENCES (I. E. LIBRARY)
CBIR SCENARIOS l SOME CBIR APPLICATION AREAS l MEDICAL IMAGING ART/CULTURAL HERITAGE l a l a DESIGN/VISUAL ARTS ENTERTAINMENT (FILM, TV) l l INDUSTRY (LOGO MANAGEMENT) GOVERNMENT (E. G. MUGSHOTS)
CBIR VERSUS TEXT l IMPORTANT QUESTION ARISES: “WHY NOT SIMPLY INDEX USING TEXT? ” (YAHOO! HAS HAD SOME SUCCESS WITH THIS) l INTUITIVE, YET USING TEXT IS l SIMPLE BUT SIMPLISTIC l TIME CONSUMING – CAN’T AUTOMATE l HIGHLY SUBJECTIVE & USER-DEPENDENT l SUSCEPTIBLE TO TRANSLATION PROBLEMS
CBIR BASIC STRUCTURE l FEATURE EXTRACTION l FEATURE DESCRIPTIONS I N D E X SIMILARITY CALCULATION l MANY DESCRIPTORS SIMILAR RESULTS MPEG-7 IS ISO STANDARD l GENERATION OF RESULTS USER INTERFACE COLOR, TEXTURE, SHAPE l l QUERY 3 BASIC FEATURES REALLY A DESIGN CHOICE SIMILARITY l l OPEN TO RESEARCH LITTLE PERCEPTUAL CONSIDERATION
CBIR (DIS)SIMILARITY? SIMILARITY IS NOT SO SIMPLE l CONSIDER THREE IMAGES l ON WHAT BASIS ARE THEY SIMILAR? l l COLOR CONTENT? SHAPE CONTENT? HIGH LEVEL IDEAS (‘MASKS’, ‘GENDER’)? PERCEPTION IS ALWAYS AN ISSUE
![CBIR SIMILARITY l DOMAIN [0, 1] l CAN BE CALCULATED MANY WAYS l GENERALIZED](https://present5.com/presentation/c7dd1d59ae9593c67f14cdff447eaf19/image-31.jpg "CBIR SIMILARITY l DOMAIN [0, 1] l CAN BE CALCULATED MANY WAYS l GENERALIZED")
CBIR SIMILARITY l DOMAIN [0, 1] l CAN BE CALCULATED MANY WAYS l GENERALIZED MINKOWSKI l CANBERRA l PERCEPTUAL MEASURE
CBIR TYPICAL ABILITIES l EFFECTIVE QUERIES IN l l SIMPLE HYBRID QUERIES l l l COLOR, TEXTURE, SHAPE DESCRIPTOR SUPERVECTORS WEIGHTED AVERAGE OF (DIS)SIMILARITIES RELEVANCE FEEDBACK l l l USER PLACED IN LOOP GIVES BETTER RESULTS STATISTICAL APPROACHES APPLY/ADJUST FEATURE WEIGHTS TO RELEVANT/IRRELEVANT ELEMENTS
CBIR SUMMARY l l l BORN FROM MULTIMEDIA FLOOD TEXT TOO SIMPLE AND LABORIOUS SYSTEMS WORK DECENTLY IN VITRO l l SHORTCOMINGS l l QUERY BY SHAPE, COLOR, TEXTURE, EXAMPLE NEED RELEVANCE FEEDBACK & PERCEPTUAL HYBRID QUERIES DIFFICULT TO CREATE SEMANTIC GAP NEEDS TO BE BRIDGED MPEG-7: IMPORTANT DEVELOPMENT
GOING FORWARD… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES
MPEG l l l MOTION PICTURES EXPERT GROUP MPEG-1 MPEG-2 MPEG-4 MPEG-7: ISO/IEC 15938 l l MULTIMEDIA CONTENT DESCRIPTION INTERFACE MPEG-21
MPEG-1 & MPEG-2 l MPEG-1 (c. 1992) l l BASIC VIDEO CODING USING DPCM & DCT TARGET: CD-BASED VIDEO & MULTIMEDIA USE I, B & P-FRAMES IN YUV SPACE MPEG-2 (c. 1994) l l l SUPERSET OF MPEG-1 GOAL: DTV/DSS OR ATM TRANSPORT MINIMUM OF NTSC/PAL QUALITY MORE ERROR RESILIENT SCALABLE – GRACEFUL DEGRADATION
MPEG-4 & MPEG-21 l MPEG-4 (c. 1998) l l l TOOLS TO AUTHOR MULTIMEDIA CONTENT TRAFFIC AWARE, ERROR RESILIENT OBJECT-BASED CODING VERY EFFICIENT FOR LOW BIT-RATES MPEG-21 (STARTED JUNE 2000) l l l AN OPEN “MULTIMEDIA FRAMEWORK” IDEA ADDRESSES DIGITAL RIGHTS MANAGEMENT ENHANCED DELIVERY & ACCESS OF DATA FOR DEVICES ON HETEROGENEOUS NETWORKS
MPEG-7 NEW PARADIGM l UNLIKE MPEG-1, MPEG-2, & MPEG-4 l l APPLICABLE TO l l l DOESN’T REPRESENT CONTENT ITSELF MPEG-7 ONLY DESCRIBES CONTENT DIFFICULT CONCEPT FOR SOME TO GRASP IMAGES VIDEO l l AUDIO & SPEECH TEXT INDEPENDENT OF l l STORAGE ARCHITECTURE l l TRANSPORT CODING
MPEG-7 HOW IT DIFFERS l MPEG-1 l l TAKES INPUT FRAMES AND REPRESENTS AS AN BINARY ENCODED VIDEO BITSTREAM MPEG-7 l TAKES VIDEO FRAMES (SAY MPEG-1 FORMAT) AND DESCRIBES CONTENTS OF EACH FRAME 1: COLOR CONTENT: 20% WHITE, 14% BLUE, SHAPES: BRIDGE, etc. FRAME 2: COLOR CONTENT: 20% WHITE, 15% BLUE, SHAPES: BRIDGE, etc. FRAME 3: COLOR CONTENT: 21% WHITE, 14% BLUE, SHAPES: BRIDGE, etc.
MPEG-7 SCOPE MULTIMEDIA DATA FEATURE EXTRACTION ALGORITHM CONTENT DESCRIPTION MPEG-7 SCOPE CODING SCHEME OTHER ELEMENTS. . .
MPEG-7 GOALS l DESCRIBE MULTIMEDIA CONTENT l l RELATIONS BETWEEN DESCRIPTORS l l SET OF DESCRIPTION SCHEMES (DS) LANGUAGE DEFINING D’s & DS’s l l SET OF DESCRIPTORS (D) DESCRIPTION DEFINITION LANGUAGE (DDL) BASED ON XML (e. Xtensible Markup Language) USED TO BUILD UP NEW D’s & DS’s ENCODING OF D’s FOR EFFICIENCY
MPEG-7 SUMMARY-1 l l STANDARDIZED DESCRIPTIONS APPLIES TO ALL DIGITAL MEDIA l l DOES NOT REPRESENT DATA ITSELF l l CBIR IS CASE FOR STILL IMAGES DESCRIBES WHAT DATA REPRESENTS SETS THE BAR FOR SYSTEMS l MULTIMEDIA/IMAGE RETRIEVAL SYSTEMS NEED AT LEAST MPEG-7 CONFORMANCE
MPEG-7 SUMMARY-2 l DOES NOT ADDRESS l l l SIMILARITY RELEVANCE FEEDBACK FEATURE EXTRACTION HYBRID QUERY GENERATION ARCHIVE ORGANIZATION THE ABOVE ISSUES HAVE BEEN PURPOSEFULLY LEFT OPEN FOR INNOVATION
FORGING AHEAD… l l l INTRODUCTION MULTIMEDIA APPLICATIONS IMPACT OF MULTIMEDIA CONTENT-BASED IMAGE RETRIEVAL (CBIR) MPEG-7 RESEARCH ISSUES
RESEARCH ISSUES l SHORTCOMINGS OF CBIR SYSTEMS l ONGOING RESEARCH l l RELEVANCE FEEDBACK HYBRID QUERY GENERATION DISTRIBUTED MULTIMEDIA INDEXING OPEN RESEARCH AVENUES
CBIR SHORTCOMINGS-1 l COLOR l l SHAPE l l l USUALLY GLOBAL HIGH DIMENSIONALITY GAMMA NONLINEARITIES CAUSE PROBLEMS COMPLICATED & DIFFICULT OCCLUSION ISSUES DURING EXTRACTION TEXTURE l l COMPLICATED & UNINTUITIVE USER-SYSTEM RIFT FOR QUERY CREATION
CBIR SHORTCOMINGS-2 l PERCEPTUAL ISSUES l l l SIMILARITY MEASURES l l l SUBTLE DIFFERENCES BETWEEN VIEWERS COLOR-BLIND USERS NEED TO BE TUNED TO DESCRIPTORS e. g. EUCLIDEAN DISTANCE NOT APPLICABLE IN NON-EUCLIDEAN DESCRIPTION SPACE RELEVANCE FEEDBACK l l PERFORMED AT GLOBAL (IMAGE) LEVEL NEED TO ADDRESS SPECIFIC IMAGE ELEMENTS
ONGOING RESEARCH -2 RELEVANCE FEEDBACK l ITERATIVE QUERY REFINEMENT l l PLACE USER IN LOOP TO ITERATIVELY IMPROVE RETRIEVAL RATES HIGH-DIMENSIONAL SPACE NEEDS PRUNING EMPHASIZED FEATURE(S) MUST BE FOUND TYPICAL APPROACHES l l STATISTICAL METHODS FEATURE WEIGHTING
ONGOING RESEARCH -2 RELEVANCE FEEDBACK l FEATURE SELECTIVE INTERFACE l l WHY CHOOSE IMAGES ON WHOLE? REQUIRES PROCESSING/STATS TO FIND GOOD FEATURES USER CAN EXPLICITLY INDICATE ELEMENTS OF IMAGE WHICH ARE GOOD: NO GUESSWORK RELEVANT SHAPE EXPLICIT FEATURES TO R. F. ENGINE RELEVANT COLOR
ONGOING RESEARCH -3 SIMILARITY AGGREGATION/HYBRID QUERIES l TYPICALLY USED APPROACHES l l BOOLEAN (AND, OR & NOT OPERATORS) EUCLIDEAN (MINKOWSKI W/ r=1) WEIGHTED AVERAGE (WA) i. e. SUPERVECTORS DISADVANTAGES l l EUCLIDEAN: FCN OF DESCRIPTORS – CHANGE DESCRIPTOR, DRASTICALLY ALTER MEASURE WA: INFLEXIBLE FOR HIGH LEVEL QUERIES, SUPERVECTORS IMPOSE CERTAIN STRUCTURE BOOLEAN: HARD LIMITED TO LOGIC FCNs ALL LACK PERCEPTUAL CONSIDERATIONS
ONGOING RESEARCH-4 SIMILARITY AGGREGATION/HYBRID QUERIES l FUZZY AGGREGATION OF DECISIONS l USE MEMBERSHIP FUNCTION TO ‘FUZZIFY’ DISTANCES & GENERATE A ‘FUZZY DECISION’ l EXPONENTIAL MODELS HUMAN PERCEPTION d SIMILARITY DISTANCE FUZZY MEMBERSHIP FUNCTION m FUZZY DISTANCE DECISION
ONGOING RESEARCH-5 DISTRIBUTED MULTIMEDIA INDEXING l INDEXES USUALLY CENTRALIZED l l ENTIRE SYSTEM FAILS IF COMPONENT FAILS NO GRACEFUL PERFORMANCE DEGRADATION HIGH DATA VOLUME = HIGH SYSTEM REQ’S DISTRIBUTED INDEXES l l SPREAD WORKLOAD OVER MANY SUBSYSTEMS INCREASE REDUNDANCY P 2 P SYSTEMS LACK CENTRALIZED ELEMENTS P 2 P SYSTEMS RESEMBLE SOCIAL NETWORKS
ONGOING RESEARCH-6 DISTRIBUTED MULTIMEDIA INDEXING l l SMALL WORLD INDEXING MODEL 1 SOCIOLOGICAL PEER DESCRIPTIONS l l l WE ARE NOT BLIND TO WHO OUR PEERS ARE PEOPLE KEEP MEMORY OF THEIR PEERS WE ARE NOT BLIND TO HOW OUR PEERS ARE l l WE REFER OTHERS TO OUR PEERS EXAMPLE [1] P. Androutsos, D. Androutsos and A. N. Venetsanopoulos, “A distributed fault-tolerant MPEG-7 retrieval scheme based on small world theory”, Distributed Media Technologies and Applications Special Issue of IEEE Transactions on Multimedia, under review.
ONGOING RESEARCH-7 DISTRIBUTED MULTIMEDIA INDEXING l INDEX AND ARCHIVE BECOME ONE l l SWIM DATA STORED IN ARCHIVE OBJECTS EACH DATA OBJECT BEHAVES AS OWN AGENTS ARE EFFECTIVE IN HIGHLY NETWORKED ENVIRONMENTS (SWIM) RETRIEVALS l l l AGENT BASED RETRIEVAL USE OF REFERRAL BASED TECHNIQUE SIMILAR TO ‘SIX DEGREES OF SEPARATION’ CURRENTLY PERFORMED WITH IMAGES
![ONGOING RESEARCH-8 DISTRIBUTED MULTIMEDIA INDEXING 2 [2] P. Androutsos, D. Androutsos and A. N.](https://present5.com/presentation/c7dd1d59ae9593c67f14cdff447eaf19/image-55.jpg "ONGOING RESEARCH-8 DISTRIBUTED MULTIMEDIA INDEXING 2 [2] P. Androutsos, D. Androutsos and A. N.")
ONGOING RESEARCH-8 DISTRIBUTED MULTIMEDIA INDEXING 2 [2] P. Androutsos, D. Androutsos and A. N. Venetsanopoulos, “Graceful image retrieval performance degradation using small world distributed indexing”, International Conference on Image Processing ICIP 2005, Genoa, Italy.
RESEARCH AVENUES-1 l HYBRID QUERIES & AGGREGATION l l WHAT DO WEIGHTS MEAN? HOW TO CHOOSE? ALTERNATIVE AGGREGATIONS METHODS ADAPTIVE SCHEMES USING REL. FEEDBACK USER INTERFACE l l BRIDGE SEMANTIC GAP BETWEEN USER’S IDEA, AND ABILITY TO EXPRESS AS A QUERY ALTERNATIVE INTERFACES–ICONIC, SEMANTIC
RESEARCH AVENUES-2 l PERCEPTUAL ISSUES l l l EMPHASIS OF DOMINATING FEATURES FEATURE MASKING EMOTIONAL INDEXING/ ALL USERS DIFFERENT–CUSTOMIZED PROFILE ARCHIVE DEPENDENCE l l SYSTEMS USUALLY SPECIALIZED ADAPTIVE INDEXING – MOST APPROPRIATE SYSTEM USED BASED ON PRELIMINARY SURVEY OF CANDIDATE DATABASE
RESEARCH AVENUES-3 l DISTRIBUTED INDEXING l l l INCORPORATE TEXT METHODS l l l DISTRIBUTED INDEXES & RETRIEVAL INDEX SYNCHRONIZATION RESULTS ORGANIZATION & RANKING SWIM OVERHEAD ESTIMATION EXTENSION OF SWIM TO OTHER DATA TYPES TEXT-INDEXING USING LIMITED VOCABULARY DON’T REJECT BUT USE INTELLIGENTLY EXTEND TO MPEG-21 & METADATA
SUMMARY-1 l MULTIMEDIA PROCESSING l l RESULTS FROM MULTIMEDIA EXPLOSION USERS DEMANDING MORE FROM DEVICES ARE CONVERGING CONTENT BASED IMAGE RETRIEVAL l l l NECESSARY TO TRACK VISUAL SEA OF DATA GOOD CAPABILITIES, BUT W/ SHORTCOMINGS PERCEPTUAL/SUBJECTIVE ISSUES RELEVANCE FEEDBACK DISTRIBUTED CONCEPTS BECOMING CRITICAL
SUMMARY-2 l MPEG-7 l l l AIMED AT STANDARDIZING DESCRIPTIONS RADICALLY DIFFERENT THAN PREVIOUS MPEGs DDL IS AN EXTENSION OF XML SCHEMA APPLICABLE TO ALL MULTIMEDIA DATA ALWAYS MORE TO DO l l l MPEG-7 HAS LEFT MANY ISSUES OPEN CBIR NEEDS TO ADDRESS USERS, PERCEPTION, HYBRID QUERIES, DISTRIBUTED SYSTEMS, ETC VIBRANT RESEARCH COMMUNITY
THANK YOU
IMPACT OF MULTIMEDIA l HIGH FLEXIBILITY RESULTS IN l l MANY TYPES OF NETWORKS CAUSE l l l RISE IN DATA GENERATION & STORAGE INCREASE IN BANDWIDTH NEEDS ONE TOOL DOING WORK OF MANY COMPLEX HARDWARE COMBINATIONS ONE DEVICE CONNECTING TO ALL NETWORKS SMALL, PORTABLE DEVICES l l MINIATURIZATED WITH HUGE CAPABILITIES ONE DEVICE REPLACES MANY
CBIR WHO’S WHO
MPEG-7 D, DS, & DDL BUILDING MORE Ds & DSs USING THE DDL DS DS D D D DEFINED IN MPEG-7 STANDARD DEFINED VIA DDL
MPEG-7 COMPONENTS l l l SYSTEMS DDL VISUAL l l l PRIMARY CONCERN FOR THIS PRESENTATION AUDIO MULTIMEDIA DESCRIPTION SCHEMES EXPERIMENTATION MODEL (XM) CONFORMANCE
MPEG-7 VISUAL COMPONENT HIGHLIGHTED DESCRIPTORS USED BY Uof. T l BASIC DESCRIPTORS l l l COLOR SPACE COLOR QUANTIZATION DOMINANT COLOR SCALABLE COLOR STRUCTURE COLOR LAYOUT Go. F/Go. P COLOR OTHER l FACE RECOGNITION TEXTURE DESCRIPTORS l l l l REGION-BASED CONTOUR-BASED 3 D SHAPE MOTION DESCRIPTORS l l l EDGE HISTOGRAM HOMOGENEOUS TEXTURE BROWSING SHAPE DESCRIPTORS l COLOR DESCRIPTORS l l GRID LAYOUT 2 D/3 D VIEW TIME SERIES SPATIAL 2 D COORDS TEMPORAL INTERPOLATION l CAMERA MOTION TRAJECTORY PARAMETRIC MOTION ACTIVITY LOCALIZATION l l SPATIO-TEMPORAL REGION LOCATOR
ONGOING RESEARCH SIMILARITY AGGREGATION/HYBRID QUERIES l FUZZY AGGREGATION OF DECISIONS l l AGGREGATE DECISIONS USING LOGIC USE COMPENSATIVE OPERATOR PARAMETER g CONTROLS DEGREE OF ANDNESS (max) & ORNESS (min) RESULT IS A SINGLE VALUE IN [0, 1] INDICATING OVERALL IMAGE SIMILARITY
Скачать презентацию Multimedia Signal Processing Content Based Image Retrieval
c7dd1d59ae9593c67f14cdff447eaf19.ppt