Скачать презентацию Data collection and Multimodal Annotation Tools Dagstuhl 2001 Скачать презентацию Data collection and Multimodal Annotation Tools Dagstuhl 2001

8de224a978ed04b2a49a436f4bffd649.ppt

  • Количество слайдов: 27

Data collection and Multimodal Annotation Tools Dagstuhl 2001 Workgroup 2 Data collection and Multimodal Annotation Tools Dagstuhl 2001 Workgroup 2

Members Permanents • Lisa Harper • Michael Kipp • Emiel Krahmer • Jean-Claude Martin Members Permanents • Lisa Harper • Michael Kipp • Emiel Krahmer • Jean-Claude Martin • Dagmar Schmauks Visiting Scientists • Harry Bunt • Kioto Hasida • John Lee • Thomas Rist • Laurent Romary

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Multimodal Corpus • Sound – human speech (e. g. MPEG 7) • • transcription, Multimodal Corpus • Sound – human speech (e. g. MPEG 7) • • transcription, (morphology) part-of-speech syntax (linguistic DS) binary relations – thematic roles – rhetorical relations – co-reference – computer voice, sound, music – environmental sounds

Multimodal Corpus (2) • Vision – head: movement, gaze, facial expression – gesture: hands/arms Multimodal Corpus (2) • Vision – head: movement, gaze, facial expression – gesture: hands/arms • basic phases • formal features (handshape, trajectories, direction, location etc. ) • encode qualities (Laban efforts? ) • functional/semiotic categories (emblem, iconic, deictic, selfadaptors etc. ) – posture: including feet/legs – computer graphics (charts/tables), characters – static/dynamic environment (people/objects): • moving camera

Multimodal Corpus (3) • Haptic – pressure of feet/hands/back/on seat, texture – force feedback Multimodal Corpus (3) • Haptic – pressure of feet/hands/back/on seat, texture – force feedback • Biometric – heartrate, eye dilation, skin sensitivity, eyebrow movement, breathing • Smell & taste (VR) • Balance (VR) • Thermal (VR) – body/object temperature, conduit properties

Multimodal Corpus (4) • Within-modality/cross-modality relations – mirror behavior, synchronized behavior, repeated behavior, postural Multimodal Corpus (4) • Within-modality/cross-modality relations – mirror behavior, synchronized behavior, repeated behavior, postural congruence – distance and touch • Behavioral/Social units? often across modalities!

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Existing Corpora: Meta Survey • Existing surveys: – ISLE and NIMM (D 8, EU Existing Corpora: Meta Survey • Existing surveys: – ISLE and NIMM (D 8, EU & US) – ELRA (EU) – COCOSDA (Japan) – LDC (US) – Talk. Bank (US)

Existing Corpora: Dagstuhl 2001 • Survey with Dagstuhl participants • Collected 28 questionnaires • Existing Corpora: Dagstuhl 2001 • Survey with Dagstuhl participants • Collected 28 questionnaires • From 24 different institutes number of corpora number of participants 0 6 1 12 2 -9 8 10+ 2

Questionnaire • annotated modalities – – – speech: 20 gestures: 17 facial expression: 5 Questionnaire • annotated modalities – – – speech: 20 gestures: 17 facial expression: 5 gaze: 3 posture: 3 • file format – analogue: 4 – digital: 12 – I don't know: 4 • tool – – own tool: 9 other tool: 3 no tool: 8 I don't know: 1 • application areas – tourism/navigation (10), consumer electronics, info kiosk, realty, storytelling, instruction, cinema, graphical design, everyday gestures, education, car, face guessing, games, talk shows

Questionnaire (2) • Languages – – – English: 11 German: 5 French: 2 Japanese: Questionnaire (2) • Languages – – – English: 11 German: 5 French: 2 Japanese: 3 Italian: 2 Dutch, Swedish, Finnish: 1 • Planning to collect: 21

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Data Collection: Methodology? • Legal issues: – ethical – commercial – country dependent legislature Data Collection: Methodology? • Legal issues: – ethical – commercial – country dependent legislature • Practical guidelines (best practice) – technical setup for recording – field-specific coder training, models for coding manuals • Specify meta-data

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Coding Schemes • Survey on existing schemes: ISLE D 9 • Guidelines for developing Coding Schemes • Survey on existing schemes: ISLE D 9 • Guidelines for developing schemes: – encoding vs. inference – can scheme accommodate semantics or generation languages for MM players (MPEG) • Standardization – partial standards like in speech – standards for computer output log files (graphics output, locations, xml, trajectories, time-stamping, granularity)

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Tools • Surveys of existing tools: – ISLE D 11 – (Bigbee, Loehr, Harper Tools • Surveys of existing tools: – ISLE D 11 – (Bigbee, Loehr, Harper 2001) – Talk. Bank proposal • Underlying frameworks: – track-based – annotation graphs – spatial annotation?

Tools: Checklists • Checklist for coding support – fast and efficient annotation – efficient Tools: Checklists • Checklist for coding support – fast and efficient annotation – efficient view, search & find, customizable – extensibility of annotation – easy access to scheme definitions (online) – automatic extraction of modality-specific specimen (images, sound bits, transcription sequences) • Checklist for multi-coder support – update/merge, concurrent coding, reliability • Checklist for Import/Export

Tools: Visions • Bootstrapping (semi-automatic or fully automatic annotation) • Use MM techniques for Tools: Visions • Bootstrapping (semi-automatic or fully automatic annotation) • Use MM techniques for coding tools (3 D, haptic, VR) • Standardized analysis (e. g. metrics) and visualization (metaphors) • Modular generic framework for – tools – schemes

Tools (4) ML classifier parser data viewer general analysis Coding Scheme Coding Tool specific Tools (4) ML classifier parser data viewer general analysis Coding Scheme Coding Tool specific analysis Logical Layer Annotation Framework (Tracks, types, objects etc. )

ML classifier gaze parser speech gesture analysis module scheme framework Coding Tool Annotation Framework ML classifier gaze parser speech gesture analysis module scheme framework Coding Tool Annotation Framework (Tracks, types, objects etc. )

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Organizations • Initiatives: – EAGLES/ISLE – ATLAS, MATE/NITE – Talk. Bank, Childes • International: Organizations • Initiatives: – EAGLES/ISLE – ATLAS, MATE/NITE – Talk. Bank, Childes • International: – ELRA/ELDA, US? Asia? • National agencies (Eurospeech): – BAS, LDC, MPI Nijmegen

Needs • • What is a multimodal corpus? What corpora do exist? How to Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Future • Data collection project – sample videos with illustrative MM data – pre-coded Future • Data collection project – sample videos with illustrative MM data – pre-coded minimal data (speech transcription) • Comparison/integration of schemes • Encourage collaborative coding?

Future (2) • Workshop on LREC Language Resources and Evaluation Canary Islands! May 2002 Future (2) • Workshop on LREC Language Resources and Evaluation Canary Islands! May 2002 – deadline: 20 Nov 2001 – paper on Dagstuhl and follow-ups – coding excercise based on data coll. – questionnaire based on Dagstuhl survey