Data collection and Multimodal Annotation Tools Dagstuhl 2001

Data collection and Multimodal Annotation Tools Dagstuhl 2001 Workgroup 2

Members Permanents • Lisa Harper • Michael Kipp • Emiel Krahmer • Jean-Claude Martin • Dagmar Schmauks Visiting Scientists • Harry Bunt • Kioto Hasida • John Lee • Thomas Rist • Laurent Romary

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Multimodal Corpus • Sound – human speech (e. g. MPEG 7) • • transcription, (morphology) part-of-speech syntax (linguistic DS) binary relations – thematic roles – rhetorical relations – co-reference – computer voice, sound, music – environmental sounds

Multimodal Corpus (2) • Vision – head: movement, gaze, facial expression – gesture: hands/arms • basic phases • formal features (handshape, trajectories, direction, location etc. ) • encode qualities (Laban efforts? ) • functional/semiotic categories (emblem, iconic, deictic, selfadaptors etc. ) – posture: including feet/legs – computer graphics (charts/tables), characters – static/dynamic environment (people/objects): • moving camera

Multimodal Corpus (3) • Haptic – pressure of feet/hands/back/on seat, texture – force feedback • Biometric – heartrate, eye dilation, skin sensitivity, eyebrow movement, breathing • Smell & taste (VR) • Balance (VR) • Thermal (VR) – body/object temperature, conduit properties

Multimodal Corpus (4) • Within-modality/cross-modality relations – mirror behavior, synchronized behavior, repeated behavior, postural congruence – distance and touch • Behavioral/Social units? often across modalities!

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Existing Corpora: Meta Survey • Existing surveys: – ISLE and NIMM (D 8, EU & US) – ELRA (EU) – COCOSDA (Japan) – LDC (US) – Talk. Bank (US)

Existing Corpora: Dagstuhl 2001 • Survey with Dagstuhl participants • Collected 28 questionnaires • From 24 different institutes number of corpora number of participants 0 6 1 12 2 -9 8 10+ 2

Questionnaire • annotated modalities – – – speech: 20 gestures: 17 facial expression: 5 gaze: 3 posture: 3 • file format – analogue: 4 – digital: 12 – I don't know: 4 • tool – – own tool: 9 other tool: 3 no tool: 8 I don't know: 1 • application areas – tourism/navigation (10), consumer electronics, info kiosk, realty, storytelling, instruction, cinema, graphical design, everyday gestures, education, car, face guessing, games, talk shows

Questionnaire (2) • Languages – – – English: 11 German: 5 French: 2 Japanese: 3 Italian: 2 Dutch, Swedish, Finnish: 1 • Planning to collect: 21

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Data Collection: Methodology? • Legal issues: – ethical – commercial – country dependent legislature • Practical guidelines (best practice) – technical setup for recording – field-specific coder training, models for coding manuals • Specify meta-data

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Coding Schemes • Survey on existing schemes: ISLE D 9 • Guidelines for developing schemes: – encoding vs. inference – can scheme accommodate semantics or generation languages for MM players (MPEG) • Standardization – partial standards like in speech – standards for computer output log files (graphics output, locations, xml, trajectories, time-stamping, granularity)

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Tools • Surveys of existing tools: – ISLE D 11 – (Bigbee, Loehr, Harper 2001) – Talk. Bank proposal • Underlying frameworks: – track-based – annotation graphs – spatial annotation?

Tools: Checklists • Checklist for coding support – fast and efficient annotation – efficient view, search & find, customizable – extensibility of annotation – easy access to scheme definitions (online) – automatic extraction of modality-specific specimen (images, sound bits, transcription sequences) • Checklist for multi-coder support – update/merge, concurrent coding, reliability • Checklist for Import/Export

Tools: Visions • Bootstrapping (semi-automatic or fully automatic annotation) • Use MM techniques for coding tools (3 D, haptic, VR) • Standardized analysis (e. g. metrics) and visualization (metaphors) • Modular generic framework for – tools – schemes

Tools (4) ML classifier parser data viewer general analysis Coding Scheme Coding Tool specific analysis Logical Layer Annotation Framework (Tracks, types, objects etc. )

ML classifier gaze parser speech gesture analysis module scheme framework Coding Tool Annotation Framework (Tracks, types, objects etc. )

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Organizations • Initiatives: – EAGLES/ISLE – ATLAS, MATE/NITE – Talk. Bank, Childes • International: – ELRA/ELDA, US? Asia? • National agencies (Eurospeech): – BAS, LDC, MPI Nijmegen

Needs • • What is a multimodal corpus? What corpora do exist? How to collect a corpus? How to develop/choose a coding scheme? What tool to develop/choose? What is the organizational infrastructure? What is the future?

Future • Data collection project – sample videos with illustrative MM data – pre-coded minimal data (speech transcription) • Comparison/integration of schemes • Encourage collaborative coding?

Future (2) • Workshop on LREC Language Resources and Evaluation Canary Islands! May 2002 – deadline: 20 Nov 2001 – paper on Dagstuhl and follow-ups – coding excercise based on data coll. – questionnaire based on Dagstuhl survey