Situation Models and Embodied Language Processes Franz Schmalhofer

Situation Models and Embodied Language Processes Franz Schmalhofer University of Osnabrück / Germany 1) Memory and Situation Models 2) Computational Modeling of Inferences 3) What Memory and Language are for 4) Neural Correlates 5) Integration of Behavioral Experiments and Neural Correlates (ERP; f. MRI) by Formal Models

Acknowledgements Charles A. Perfetti, Michal Balass, Jessica Nelson, Chin-Lung Yang & Edward Wlotko University of Pittsburgh funded by Alexander von Humboldt Foundation Uwe Friese, Markus Raabe, Karin Pietruska Ho-Ming Chow Niki Vavatzanidis Anke Karabanov University of Osnabrück Mark Greenlee Roland Rutschmann University of Oldenburg

Outline • Behavioral Data and Computational Modeling on Inferencing in Text Comprehension • Event-related potentials (ERP) on the reading of inference-related words • An f. MRI-experiment on inference processes and the verification of inference related statements

Text comprehension and Inferencing • Mary heard the ice-cream van coming. • She remembered the pocket money. • She rushed into the house.

A blueprint of the reader (Perfetti, 1999) Comprehension Processes Inferences Situation Model Text Representation General Knowledge Linguistic System Phonology Syntax Morphology Parser Meaning and Form Selection Lexicon Meaning Morphology Syntax -argument structure Word Representation Word Identification Orthographic Units Phonological Units Writing System Orthography Mapping to phonology Visual Input

The Ki. Wi-model (Schmalhofer, 1998) common sense related domain knowledge situation model text repres. text Sensory encoding direct experience

A mapping of mental and brain processes • Beeman, Bowden & Gernsbacher (2000) – Information supporting predictive inferences initially activated in RH • Long & Baines (2002) – Hemispheric differences in representations • Tapiero & Fillon: Emotional inferences and hemispheric differences • Mason & Just (2004) – Inferencing by right hemisphere language network and reasoning network (dorsolateral prefrontal cortex) • Ferstl (2003, HC); Ferstl & von Cramon (2001) – Shift from local to global aspects precuneus – Inferencing – Situation model (dep. on content) fronto-median cortex (FMC) v. FMC

Predictions from the KIWi-model (explicit and paraphrase)

Predictions from the KIWi-model (inference and control conditions)

How many nodes and links had to be newly constructed Nodes Links Explicit 1 1 Paraphrase 1 1 Inference 2 2 Control 3 3

Text Materials in ERP experiment

ERP study of inferencing • Use ERPs to examine inference processes. • Vary the accessibility of referents required by the integration (new construction versus preexisting traces) referent previously introduced, possibly inferred or control condition • Measurements taken on a single word that occurs early in the second sentence

The N 400 component in ERP-studies • N 400: A negativity shift around 400 ms is associated with an incongruent meaning. • “He smeared the bread with socks. ” • A marker for semantic processing; sensitive to semantic congruence effects in sentence contexts (Van Petten & Kutas, 1990) • Sensitive to global comprehensibility effects in text (St. George, Mannes, & Hoffman, 1994)

General Procedure of ERP-study Participants: • 32 adult readers Procedure: • Slow-SVP presentation • 600 ms SOA word-to-word • 300 ms exposure duration • 300 ms interval • Last word of sentence followed by additional 300 ms interval Materials: • 120 two-sentence passages for each participant; 30 in each condition; Comprehension Probes (T-F) at random after 25% of trials

Summary (ERP) • A common set of assumptions about construction and integration processes in text comprehension is useful for accounting for behavioral data as well as neural correlates (e. g. ERP) • Inference possibilities yield a cognitive preparation for target concept, but different from explicit and control conditions • Distinction between text and situation representation accounts for behavioral data (priming) and ERP-data quite well

Text Materials of f. MRI Experiment

f. MRI-Lab

f. MRI Lab • 1. 5 T Siemens Sonata f. MRI-Scanner (Siemens Medical Solutions) • 8 Channel Head Coil • Lumitouch optical response device • Visual Stimulation and recording of responses were controlled by E-Prime and PC

Behavioral Results • Results in Milliseconds • Differences are significant except the following pairings: – Explicit – Paraphrase – Filler – Predictive

f. MRI: General Procedure • 13 adult skilled readers • 108 (72 + 36) trials; 18 trials per condition • Experiment is divided into 3 Sessions à 36 trials (6 per condition) • Pseudorandom order of trials • Conditions and domains (themes) were counterbalanced by a Latin Square Design • Different response delays were equalized at the end of each trial to receive a constant trial length

Time sequence of one trial

f. MRI - Procedure BOLD sensitive T 2 weighted functional sequence (TR = 3 s, TE = 50 ms, FOV = 192 mm) T 1 weighted structural sequence (TR= 1900 ms, TE=3. 93 ms, FOV = 256 mm) Rotated app. 10° relating to AC-PC line, covering prefrontal, parietal and temporal regions in full and the majority of the occipital cortex Acquisition at the end of the experiment (app. 10 Minutes) 3 Sessions à 324 continuously acquired scans (app. 48 Minutes)

f. MRI – Data Analysis • Data were preprocessed analyzed using The SPM 2 Software Package (Functional Imaging Laboratory, Welcome Department of Imaging Neuroscience, London) • Statistical Analysis was conducted by calculating a Random Effects Analysis • Referring to the problem of multiple comparisons only those clusters with a corrected p < 0. 05 were considered as significantly activated

f. MRI – Data Analysis • The last third of reading – modeled as a block Reading • Sentence versus Pseudoword Reading • Sentence Reading in • Statement Verification Task Inference Condition versus Sentence Reading in Explicit – modeled as a block Condition Statement Verification • Inference vs Explicit • Inference vs Paraphrase • Control vs Explicit

Word-Reading vs Pseudoword-Reading

Summary (f. MRI-Experiment) • fronto-median wall: – reasoning, inferencing, situation model integrations (Ferstl, Mason & Just; Robertson & Gernsbacher) • Precuneus, Cuneus, occipital lobe: – Imagery, perception tasks, memory related imagery operations • Posterior Cingulate: – Memory retrieval • Prefrontal: Inferencing • BA 9/10: theory of mind

Integration of Fields and Methods Philosophy Linguistics Language sytems Computer Science Electronic machines Psychology Cognitive Science How the mind works? Human Behavior Medicine - Physiology Brain Research Components of the brain, How does it work? Cognitive Neuroscience How the brain enables the mind?

What the course was about: Sth. Old and Std. New • • • Memory and Situation Models Computational Modeling of Inferences What Memory and Language are for Neural Correlates (ERP, f. MRI) Integration of Behavioral Experiments and Neural Correlates (ERP; f. MRI) by Formal Models