Video Analysis Content Extraction VACE Executive Brief for MLMI Dennis Moellman, VACE Program Manager
Video Analysis Content Extraction • • • Briefing Outline Introduction Phase II Evaluation Technology Transfer Phase III Conclusion
Video Analysis Content Extraction Introduction What is ARDA/DTO/VACE? • ARDA – Advanced Research and Development Activity – A high-risk/high-payoff R&D effort sponsored by US Do. D/IC • ARDA taking a new identity – In FY 2007 under the DNI • Report to: ADNI(S&T) – Renamed: Disruptive Technology Office • VACE – Video Analysis and Content Extraction – Three Phase initiative begun in 2000 and ending 2009 • Winding down Phase II • Entering into Phase III
Context Video Analysis Content Extraction Video Exploitation Barriers • Problem Creation: – Video is an ever expanding source of imagery and open source intelligence such that it commands a place in the all-source analysis. • Research Problem: – Lack of robust software automation tools to assist human analysts: • Human operators are required to manually monitor video signals • Human intervention is required to annotate video for indexing purposes • Content based routing based on automated processing is lacking • Flexible ad hoc search and browsing tools do not exist • Video Extent: – Broadcast News; Surveillance; UAV; Meetings; and Ground Reconnaissance
Video Analysis Content Extraction Research Approach Video Exploitation • Research Objectives: – – Basic technology breakthroughs Video analysis system components Video analysis systems Formal evaluations: procedures, metrics and data sets • Evaluate Success: – Quantitative Testing Metric Current Need Accuracy <Human >>Human Speed >Real time <<Real time – Technology Transition • Over 70 technologies identified as deliverables • 50% have been delivered to the government • Over 20 undergoing government evaluation
Video Analysis Content Extraction Management Approach Geared for Success Management Philosophy – NABC • N – Need • A – Approach • B – Benefit • C – Competition
Interests Video Analysis Content Extraction System View Source Video Enhancement Filters 01101010111 110100101011 01101011 011010 0110 Reference Understanding Engine Recognition Engine Extraction Engine Intelligent Content Services Concept Applications Language/User Technology Visualization
VACE Interests Video Analysis Content Extraction Technology Roadmap Phase 1 Phase 2 Phase 3 Future In t Co ellig Se nt en rv en t ice t s C Ex ont tra en ct t io n Object Detection & Tracking Object/Scene Classification Object Recognition Object Modeling Simple Event Detection Event Recognition Complex Event Detection Scene Modeling Event Understanding Mensuration Indexing Video Browsing Summarization Filtering Advanced query/retrieval using Q&A technologies Content-based Routing Video Mining Change Detection Te Ena ch bli no ng log ies Video Monitoring Image Enhancement/Stabilization Camera Parameter Estimation Multi-modal fusion Integrity Analysis Motion Analysis Event Ontology Event Expression Language Automated Annotation Language Evaluation
Funding Video Analysis Content Extraction Commitment to Success FY 06 Allocations FY 07 Allocations 4% 4% 10% 12% 11% 20% 39% 64% 36%
Video Analysis Content Extraction Phase II Programmatics • Researcher Involvement: – Fourteen contracts – Researchers represent a cross section of industry and academia throughout the U. S. partnering to reach a common goal • Government Involvement: – Tap technical experts, analysts and COTRs from Do. D/IC agencies – Each agency is represented on the VACE Advisory Committee, an advisory group to the ARDA/DTO Program Manager
Phase II Video Analysis Content Extraction Demographics Univ. of Washington Univ. of Illinois. Urbana-Champaign (2) Chicago Urbana-Champaign Boeing Phantom Works Carnegie IBM T. J. Mellon Watson Center Univ. (2) (Robotics Inst. ) (Informedia) . Virage TASC Wright State Univ. Purdue Univ. AFIT MIT BBN SRI Salient Stills Alphatech Columbia Univ. of Southern California Sarnoff Corp (2) Univ. of Maryland (2) Univ. of Southern California / Info. Science Inst. Prime Contractors (14) Sub Contractors (14) Telcordia Technologies Georgia Inst. Of Tech. Univ. of Central Florida
Phase II Video Analysis Content Extraction Projects Title Organization Principal Investigator Foreign Broadcast News Exploitation ENVIE: Extensible News Video Information Exploitation Carnegie Mellon University Howard Wactlar Reconstructing and Mining of Semantic Threads Across Multiple Video Broadcast News Sources using Multi. Level Concept Modeling IBM T. J. Watson Research Center / Columbia Univ. John Smith / Prof. Shih-Fu Chang Formal and Informal Meetings From Video to Information: Cross-Modal Analysis of Planning Meetings Wright State University Va. Tech/AFIT / Univ. of Chicago / Purdue Univ. / Univ. of Illinois. Urbana-Champaign Event Recognition from Video of Formal and Informal Meetings Using Behavioral Models and Multi-modal Events BAE Systems/ MIT / Univ. of Maryland / Virage Francis Quek / Ronald Tuttle / David Mc. Neill & Bennett Bertenthal / Thomas Huang / Mary Harper Victor Tom/ William Freeman & John Fisher / Yaser Yacoob & Larry Davis / Andy Merlino
Phase II Video Analysis Content Extraction Projects Title Organization Principal Investigator Abstraction and Inference about Surveillance Activities Video Event Awareness Sarnoff Corporation / Telcordia Technologies Integrated Research on Visual Surveillance University of Maryland Rafael Alonso / Dimitrios Georgakopoulos Larry Davis, Yiannis Aloimonos & Rama Chellappa UAV Motion Imagery Adaptive Video Processing for Enhanced Object and Event Recognition in UAV Imagery Boeing Phantom Works (Descoped to UAV data collection) Robert Higgins Task and Event Driven Compression (TEDC) for UAV Video Sarnoff Corporation Hui Cheng Ground Reconnaissance Video Content and Event Extraction from Ground Reconnaissance Video TASC, Inc. / Univ. of Central Florida / Univ. of California-Irvine Sadiye Guler / Mubarak Shah / Ramesh Jain
Phase II Video Analysis Content Extraction Projects Title Organization Principal Investigator Cross Cutting / Enabling Technologies Probabilistic Graphical Model Based Tools For Video Analysis University of Illinois, Urbana - Champaign Thomas Huang Automatic Video Resolution Enhancement Salient Stills, Inc. John Jeffrey Hunter Robust Coarse-to-Fine Object Recognition in Video CMU/Pittsburgh Pattern Recognition Henry Schneiderman & Tsuhan Chen Multi-Lingual Video OCR BBN Technologies / SRI International John Makhoul / Greg Myers Model-based Object and Video Event Recognition USC - Institute for Robotics and Intelligent Systems / USC - Information Sciences Institute Ram Nevatia, Gerard Medioni & Isaac Cohen / Jerry Hobbs
Video Analysis Content Extraction Evaluation Goals • Programmatic: – Inform ARDA/DTO management of progress/challenges • Developmental: – Speed progress via iterative self testing – Enable research and evaluation via essential data and tools – build lasting resources • Key is selecting the right tasks and metrics – Gear evaluation tasks to research suite – Collect data to support all research
Evaluation Video Analysis Content Extraction The Team NIST USF Video Mining
Video Analysis Content Extraction Evaluation NIST Process Planning Products Results Evaluation Plan Determine Sponsor Requirements Task Definitions Protocols/Metrics Dry-Run shakedown Rollout Schedule Assess required/existing resources Develop detailed plans with researcher input Data Identification Formal Evaluation Resources Training Data Development Data Evaluation Data Technical Workshops and reports Ground Truth and other metadata Scoring and Truthing Tools Recommendations
Video Analysis Content Extraction Evaluation NIST Mechanics Algorithms System Output Evaluation Video Data Ground Truth Measures Annotation Results
Evaluation Video Analysis Content Extraction 2005 -2006 Evaluations Detection Evaluation Type Tracking Recognition P F V T Meeting Room x x Broadcast News x x x x x UAV x x Surveillance x x x Ground Recon Domain P = Person; F = Face; V = Vehicle; T = Text
Video Analysis Content Extraction Evaluation Quantitative Metrics • Evaluation Metrics: – Detection: SFDA (Sequence Frame Detection Accuracy) • Metric for determining the accuracy of a detection algorithm with respect to space, time, and the number of objects – Tracking: STDA (Sequence Tracking Detection Accuracy) • Metric for determining detection accuracy along with the ability of a system to assign and track the ID of an object across frames – Text Recognition: WER (Word Error Rate) and CER (Character Error Rate) • In-scene and overlay text in video • Focused Diagnostic Metrics (11)
Video Analysis Content Extraction Evaluation Phase II Best Results
Video Analysis Content Extraction Evaluation Face Detection: BNews (Score Distribution)
Video Analysis Content Extraction Evaluation Text Detection: BNews (SFDA Score distribution)
Video Analysis Content Extraction Evaluation Open Evaluations and Workshops -- International • Benefit of open evaluations – Knowledge about others’ capabilities and community feedback – increased competition -> progress • Benefit of evaluation workshops – Encourage peer review and information exchange, minimize “wheel reinvention”, focus research on common problems, venue for publication • Current VACE-related open evaluations – – – VACE: Core Evaluations CLEAR: Classification of Events, Activities, and Relationships RT: Rich Transcription TRECVID: Text Retrieval Conference Video Track ETISEO: Evaluation du Traitment et de l’Interpretation de Sequences Video
Evaluation Video Analysis Content Extraction Expanded Task Source Data Conference Meetings Sub-condition Seminar Meetings Surveillance Broadcast News Studio Poses UAV 3 D Sing Per Track Video CHIL Audio+Video CHIL 2 D Multi Per Detect VACE 2 D Multi Per Track VACE CHIL VACE Person ID Video CHIL Audio + Video CHIL 2 D Face Det VACE 2 D Face Track VACE CHIL VACE Head Pose Est CHIL Hand Detect VACE Hand Track VACE Text Detect VACE Text Track VACE Text Recognition VACE Vehicle Detect VACE Vehicle Track VACE Feature Extract TRECVID Shot Boundary Detect TRECVID Search TRECVID Acoustic Event Detection CHIL Environment Class CHIL
Video Analysis Content Extraction Evaluation Schedule
Video Analysis Content Extraction TECH TRANSFER DTO Test and Assessment Activities Purpose: Move technology from lab to operation • Technology Readiness Activity – – An independent repository for test and assessment Migrate technology out of lab environment Assess technology maturity Provide recommendations to DTO and researchers
Video Analysis Content Extraction TECH TRANSFER Do. D Technology Readiness Levels (TRL) Level Definitions Entry Condition Contractor Activity 1 Basic principles observed and reported Some peer review of ideas Reporting on basic idea 2 Technology concept and/or application formulated Target applications are proposed Speculative work; invention 3 Analytical and experimental critical function and/or characteristic proof of concept Algorithms run in contractor labs and basic testing is possible (internal, some external may be possible) Doing Analytical studies with weakly integrated components Component/breadboard validation in lab Proof of concept exists; test plans exist; external testing is possible Low fidelity integration of components 5 Component/breadboard validation in relevant environment Integrated system functions outside contractor lab; some TRA tests completed Working with realistic situations 6 System/subsystem model or prototype in demonstration in relevant environment IC/Do. D users identified; target environment defined; simulated testing possible Demonstrating engineering (software qualities) feasibility 7 System prototype demo in operational environment Test lab trials in simulated environment completed; installed in operational environment Completing the product 8 Actual system completed and qualified through test and demonstration Product completed; Test lab trial completed successfully Releasing the product; Repairing minor bugs 9 Actual system proven through successful mission operations Proven value-added in an operational environment Repairing minor bugs; noting proven operational results 4
Technology Transfer Video Analysis Content Extraction Applying TRL DOD Technology Risk Scale RISK HIGH LOW DTO Control DTO Influence UNCLASSIFIED Contractor Test Facility 8 9 Production UNCLASSIFIED Info-X Test Facility 6 7 UNCLASSIFIED IC/DOD Test Facility(s) 4 5 Use in assessing project’s 1 2 3 • Technology maturity • Risk level • Commercialization potential
Video Analysis Content Extraction Technology Transfer TRA Maturity Assessments
Video Analysis Content Extraction Phase III BAA Programmatics • Contracting Agency: DOI, Ft. Huachuca, AZ – DOI provides COR – ARDA/DTO retain Do. D/IC agency COTR’s and add more • Currently in Proposal Review Process – Span 3 FY’s and 4 CY’s – Remains open thru 6/30/08 • Funding objective: $30 M over program life – Anticipate to grow in FY 07 and beyond • Address the same data source domains as Phase II • Will conduct formal evaluations • Will conduct maturity evaluations and tech transfer
Video Analysis Content Extraction Phase III BAA Programmatics • Emphasis on technology and system approach – Move up technology path where applicable – Stress ubiquity • Divided into two tiers: – Tier 1: One year base with option year • Technology focus • Open to all – US and international • More awards for lesser funding – Tier 2: Two year base with option year(s) • Comprehensive component/system level initiative • Must be US prime • Fewer awards for greater funding
Video Analysis Content Extraction Phase III BAA Schedule
Video Analysis Content Extraction Summary Take-Aways • VACE is interested in: – Solving real problems with risky, radical approaches – Processing multiple data domains and multimodal data domains – Developing technology point solutions as well as component/system solutions – Evaluating technology process – Transferring technology into user’s space
Conclusion Video Analysis Content Extraction Potential DTO Collaboration • Invitations: – Welcome to participate in VACE Phase III Evaluations
Video Analysis Content Extraction Contacts Dennis Moellman, Program Manager Phones: 202 -231 -4453 443 -479 -4365 301 -688 -7092 800 -276 -3747 (Dennis Moellman) (Paul Matthews) (DTO Office) FAX: 202 -231 -4242 301 -688 -7410 (Dennis Moellman) (DTO Office) E-Mail: dennis. moellman@dia. mil (Internet Mail) pmmatth@nsa. gov Location: Room 12 A 69 NBP #1 Suite 6644 9800 Savage Road Fort Meade, MD 20755 -6644
Скачать презентацию Video Analysis Content Extraction VACE Executive Brief for
400368c1017fed930df85f2f804a35ad.ppt