Embodied Coding through Perspective Brian Mac Whinney CMU

Embodied Coding through Perspective Brian Mac. Whinney, CMU Perspective 1

Embodiment • The problem of the mental code. Something from nothing. • Mentalese vs. bodyese w The perception-motor cycle w Seeing things through our actions on them w The brain connects to the body w Even true on the level of the saccade w The mirror system connects the body to others Perspective 2

What does the code do? • It shapes memory and retrieval. • It permits resonance. • It structures mental models w Long-term w Current conversation • It gives meaning through the homunculus. • So, meaning is based in the end on the body. Perspective 3

Bodyese and Perspective • When we act, we have w A stance (balanced, sitting, stretched) w A spatial position (in front, behind) w A temporal flow (repeat, single, end) w Goals and plans (push, open, turn, fall) w A social position (giver, brother, trust) • All of these actions situate us in a perspective Perspective 4

Processing • Incrementalism w In comprehension, we build competing perspectives and select w Picture-sentence matching, sometimes with mental pictures w In production, we enact a walk-through of our past experiences • Tip of the iceberg theory w Enactive-depictive • Speech-gesture linkage theory Perspective 5

Mac. Whinney & Pleh 1987 • # switches • SS: The dog that chased the cat kicked the horse. 0 • OS: The dog chased the cat that kicked the horse. 1 - • OO: The dog chased the cat the horse kicked. 1+ • SO: The dog the cat chased kicked the horse. 2 • SS > OS = OO > SO • The dog the cat the boy liked chased snarled. • (dog -> cat -> boy -> cat -> dog) Perspective 4+

Counting Shifts • Advantage of first mention - Gernsbacher • Reflexive shift w Mary combed herself. w The desert trains the young to be tough. • Complex NP shift -- The communist farmers hated died. • Shift to object, implicit causer w Bill chased the cat. w Coming towards him, John saw five bicycles. w Bill criticized Mary, because she/he w Bill apologized to Mary, because she/he Perspective 7

Principles • • • Pronouns must be referential First mention maintains unless … Missing mentions involve costs Objects prepare shifts Verbs code perspectives Perspective 8

Perspective across Cognition unified embodied image language as a functional neural circuit perspective direct experience perspective space/time deixis perspective plans Perspective perspective social 9

How language links to levels • • • Words - abandon, beloved Linkers - deixis, anaphora Phrases - (my (little doll)) Clauses - grammar Discourse - characters on stage Perspective

Words • • Banana －affordances，actions Navajo table and chair Miller and Johnson-Laird’s “table” Verbs as links to motor actions -- give, walk, talk, push, run • Huge schemas -- promise, gravity, a certiorari Perspective

Coding and Resonance • • • Local organization Somatotopic Coding through the whole system Even crazy resonances facilitate learning Is there hard evidence? Perspective 12

Clauses: An example • “The cat licks itself” Perspective

Depictive Imagery Perspective

Enactive Imagery • • • take stance of cat find area to lick enact licking shift attention to area being licked perceive touch through licking Perspective

Two pathways enactive depictive Perspective

Compatible Images • He hammered the nail into the wall. • He hammered the nail into the floor. w Nail pointing up. w Nail pointing across. w Tasks: verify, name w Zwaan, Schwartz, Tversky, Klatzky, Glenberg …. Perspective

Mirror Neurons - Rizzolatti E grasps M grasps E with pliers Perspective M grasps

Monkey grasps in dark Perspective

Ungrounding • Identification • Imagery • Dreaming Perspective

Identification • Imitation and identification require “mapping” of body image • This requires an accessible body image that can be dynamically remapped • Once mapped, we can follow another’s motions • Johnson (1998), Meltzoff and Moore, Csibra et al. § Eyes and movement trigger identification § Identification is basic, not cue-driven • Mapping to other § Birds “have no hands” (Menn) § Child looks for its tail. (Mervis) § Child opens mouth for scissors (Fincham) Perspective

Three spatial perspectives • Ego-centered -- egocentric w Here, go up, go left • Object-centered -- allocentric w On the table, left of the tower w Come, go, arrive w Ego takes object perspective • Frame-centered -- geocentric w Toward the Bay, north Perspective

Three temporal perspectives • Speech time (ego centered) w Now, today • Reference time (object centered) w Thursday, graduation, when he arrived … • Event time against reference w Before he arrived, John had lost his wallet w After he arrived, John lost his wallet • We take the perspective of the reference time Perspective

Aspect • Aspects are contours of actions or events w Terminative: I stop chopping. w Inceptive: I began chopping. w Iterative: I kept on chopping. • The whole action is the background and the aspect involves approaching the action with a particular involvement Perspective

Causal Roles • • Predicates are directly grounded Case is routinized perspective Grammar arises to encode perspective shifts Constructions serve to express perspective modulation Perspective

Case is routinized perspective • Arg 1 – The initial perspective or starting point • Arg 2 – Recipient of shifted perspective • Arg 3 – External perspective • Three roles are enough for an item-based grammar (Construction Grammar, Word Grammar) Perspective

Item-based Sockets • Phrasal constructions provide sockets for extension and pushy polysemy (Mac. Whinney 1989) w Give me another sand. w A wiser Reagan returned from Rejkjavik. • Verbs provide sockets for social expectations w criticize vs. amazed Perspective

Constructions • • Passive Double Object Relativization Appositive Fictive Agent Conflation Comparison Complementation Adverbialization Attachment Dislocation Clefting Topicalization Possession Ellipsis Coordination …. Perspective

Ambiguity • Tim saw the Grand Canyon flying to New York. • Visiting relatives can be a nuisance. • The women discussed the dogs on the beach. • I ordered her pancakes. • Although John always jogs, a mile seems long. • The horse raced past the barn fell. Perspective

Scope • • • Many arrows didn’t hit the target. The target wasn’t hit by many arrows. Three students read two books. Two books were read by three students. Every man who owns a Porsche loves it. Perspective 30

C-Command Perspective

C-command as Perspective • Starting points must be referentially committed (Mac. Whinney, 1977) • Pronouns immediately seek their referents w Mc. Donald and Mac. Whinney (1996) § Mary criticized Billi, because hei. . . § Maryi amazed Bill, because shei … • Perspective flows to referents online Perspective

Centrality forces commitment • • • S > DO > IO > Poss * Hei says Billi came. * John told himi that Billi was crazy. ? John said to himi that Billi was crazy. John told hisi mother that Billi was crazy. • C-Command blocking depends on how you draw the tree Perspective

Backgrounding • Releasing commitment: w After hei came, Johni drank three beers. w Near himi, Johni saw a snake. • Yes, we can backtrack, but it is marked: w *Near Johni, hei saw a snake. w ? Near John’si computer desk, hei placed a printer. w ? After Johni came, hei drank three beers. w ? Hei came and then Johni drank three beers. Perspective 34

Reflexives maintain Perspective • • • Igori hid the book behind himi/himselfi. Johni kicked *himi/himselfi. Johni ignored the oil on himi/*himselfi. Johni heard a story about himi/himselfi. Maxi told a story about *himi/himselfi. Perspective

Perspective interruption • *Jessiei stole a photo of heri out of the archives. • Jessiei stole me a photo of heri out of the archives. Perspective

Shifts • Jessie stole a photo of *her/herself out of the archives. • Jessie stole a silly photo of her/herself out of the archives. • Anna hid a snapshot of *her/herself under the linoleum. • Anna hid the snapshot of *her/herself under the linoleum. • Lucie talked about the operation on *her/herself. • Lucie talked about the operation on her/herself that Dr. Edward performed. Perspective

Clitic Assimilation § Why do you want to go? § Why do you wanna go? § Who(m) do you want to go? § * Who(m) do you wanna go? § I get ta go. (Privilege) § I gotta go. (Obligation) Perspective

An example A cyclone hammered the Bangladesh coast Monday with the force of "hundreds of demons" leveling entire villages of mud and thatch huts, flooding crops, and killing at least six people. Three men and two children were crushed under collapsed buildings or hit by flying pieces of tin roofs in the southern port of Chittagong. One man died in Teknaf, about 110 miles down the coast, when he was blown off his roof, while trying to secure it. The storm roared in from the Bay of Bengal with wind gusts of 125 mph, forcing a half-million people to flee their huts and huddle in concrete shelters. Many power and telephone lines were down, so a full account of casualties and damage was not available. Perspective 39

Mental Spaces • A contemporary philosopher, leading a seminar (From Fauconnier and Turner) § I claim that reason is a self-developing capacity. Kant disagrees with me on this point. He says it’s innate, but I answer that’s begging the question, to which he counters, in Critique of Pure Reason, that only innate ideas have power. But I say to that, what about neuronal group selection? And he gives no answer. • One space for the teacher, one for Kant, and a comparison space where the debate occurs. Perspective

Put the bottom block below the apple Perspective

Linguistic Devices • • • Your beloved cat -- Your damned cat. John criticized Bill, because he. . John apologized to Bill, because he … Japanese speaker’s territory Evidentials, deontic-evidential Perspective

Perspectival Domains • The bicyclist appears to have escaped injury • Did the bicyclist appear to have escaped injury? • The reporter said that the bicyclist appeared to have escaped injury. • The reporter asked if the bicylcist appeared to have escaped injury. • John said that Marco hates Sarah’s beloved cousin. • The adults in the picture are facing away from us, with the children hidden behind them. (Cantrall) Perspective

Logophoricity • John said, “The bicycle is here. ” • John said that the bicycle is here. • Unfortunately, John said that the bicycle is behind the damn wall. • Did John say that the bicycle is evidently behind the damn wall? Perspective

Frontal support • Inferior frontal cortex supports code generation through perceptual-motor images for memory lists • Linkage of frontal areas to temporoparietal areas supports reenactment (simulation) • Interconnections of frontal motor and attentional areas • Linkage of prefrontal to subcortical and cingulate gyrus Perspective

Evolution • Apes can shift perspective (Call & Tomasello) • Gestural communication required further perspective taking • Creativity revolution: language or symbol? • Charismatic leaders and priests could use language to motivate the people Perspective 46

Classic Issues • • • Formalism vs. Functionalism in Grammar Language and Consciousness Grounding in Language Development Symbol Grounding Robotic Grounding Language as a Special Gift Perspective 47

Assumptions that don’t change • Basic activation/inhibition framework w Power Law w Neural Network modelling • Mac. Whinney-Bates Competition Model w Cue validity, cue strength w Lexical grammar w Incrementalism • Barsalou makes mental models safe for AI • Dual code theory still works Perspective

Conclusions • Cognition is grounded on perceptual and motoric simulations, triggered by language and gesture • The forms of language are shaped by the process of perspective-taking • The brain provides mechanisms to implement simulation and perspective Perspective