Embodied Language Cognition and Communication E LCC Spontaneous

Embodied Language, Cognition, and Communication (E LCC): Spontaneous gesture with speech Alan Cienki and Irene Mittelberg Engelse taalkunde Afdeling Taal en Communicatie Faculteit der Letteren LCC presentation, 6 November 2008

Some background: gesture studies • Spoken language studied in fields such as anthropology, sociology (e. g. , CA), ethnography of communication • Gesture sometimes studied in these contexts • The study of “non-verbal communication” in clinical psychology

Gesture studies as its own field • E. g. , Adam Kendon’s work since 1970 s – Gesture: Visible action as utterance (2004) • Landmark – Hand mind: What gestures reveal about thought (David Mc. Neill 1992) • International Society for Gesture Studies (ISGS) • ISGS conference series • Founding of the journal Gesture in 2001

What are we calling a gesture? • A visible, distinct, effortful movement of part of the body (Kendon) • Special status of manual gestures

Background on gesture • Range of conventionality of form-meaning associations – from highly idiosyncratic to highly conventionalized “Kendon’s continuum” (Kendon 1982; Mc. Neill 1992) gesticulation - pantomime - emblems - sign language

Gesturing is not usually just handwaving • Not random, structureless or unsystematic • Basic structuring principles of gesture at work: – schematic patterns – abstractions from physical actions, objects, and relations – roles of metonymy and metaphor

On the coding of gestural forms • Four parameters are often used (based on research on sign languages): – hand shape – palm orientation – location in gesture space – movement

Notation of form features in gestures • status of fingers differentiated (Bressem 2006, in prep. )

Notation of form features in gestures • form class “flat hand” (Bressem 2006, in prep. )

Models of gesture space etc. -1 0 1 2 3 upper extreme periphery center cc lower right left (Fricke 2005/ in prep. ) (Mc. Neill 1992)

Some functions of gestures • Discourse structuring function • Pragmatic/interactional function • Referential function • These can overlap – Gestures can have multiple functions

Some functions of gestures • Referential function – Concrete reference – Abstract reference

Some functions of gestures Concrete reference Oh, an essay. You have to `write fast! {video clip} (video clips have not been included in this Power. Point since we do not have permission to publish them)

Some functions of gestures • Abstract reference: Representation of the abstract (non-physical) in terms of the concrete e. g. , an idea as a space, form, or motion Metaphor

Example: metaphors in gestures that, there’s never a situation, that is, ideally r- -where there’s, an ideal, <@well@>, <at least there isn’t in my life>, where there’s something right, and something absolutely wrong. What you have to do is draw your line, and figure out on which side of it you fall.

Example: metaphors in gestures {video clip}

Example: metaphors in gestures Possible analysis might include: • SITUATIONS ARE LOCATIONS • MAKING A DECISION IS DRAWING A LINE • EVALUATING YOUR BEHAVIOR IS POSITIONING YOURSELF • Note timing of onset of gestures with respect to speech

Metaphor • Metaphor as not just a matter of language, but of mapping one conceptual domain (with imagery!) onto another (Lakoff & Johnson 1980, 1999; and many other works…) • Therefore, we should see evidence of conceptual metaphors in different forms of human expression • Potential for multimodal expression of metaphor in language use (Cienki & Müller 2008; Forceville & Urios-Aparisi, forthcoming; Müller 2008)

Side note: perspective in gesture studies • If studying gesture can help in meaningmaking, whose meaning-making are we talking about? – the speaker’s? – the addressee’s? – another observer’s? (the researcher’s? )

Interaction of metaphor and metonymy in co-verbal gestures Irene Mittelberg

points of departure: How do speech and gesture share the semiotic work of rendering abstract knowledge domains more graspable? Previous research has shown that spontaneous co-speech gestures may reveal: • embodied and situated aspects of abstract reasoning • materializations of spatial metaphor • interaction of iconicity and metaphor (also in signed languages) Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; Mc. Neill 1992, 2005; Müller 1998, 2003, 2008; Nuñez 2004; Nuñez & Sweetser 2006; Sweetser 1998, 2007; Taub 2001; P. Wilcox 2000, 2004; S. Wilcox 2004; Williams 2008; inter alia

While metaphor is central to accessing abstract domains and the gestures in the data of this study are essentially metaphorical in nature metaphorical modes do not suffice to account for the semantic / pragmatic processes in the multimodal data. => not necessarily a direct iconic link between the form of the gesture and the form of its referent. example: palm-up open hand gesture on mention of “noun” as sitting on palm-up open hand (Müller 2004) contiguity relation between hand imaginary object

Grammatical categories (noun, verb) contiguity relations between hands & imaginary objects {video clip}

cognitive-semiotic approach to co-speech gesture Charles Sanders Peirce’s (1931, 1955) pragmaticist theory of signs, focus on interpretation perspective for analysis: addressee -------Roman Jakobson’s (1956, 1960 1961, 1963, 1966) theory of metaphor & metonymy -------Cognitive linguistics: metaphor, metonymy, image schemas (Dirven 2002; Gibbs 1994, 2006; Johnson 1987; Lakoff 1987, 1993; Lakoff & Johnson 1980, 1999; Radden 2000; Panther & Thornburg 2003, 2004; Sweetser 1990; Taub 2001; Wilcox 2004) Gesture research (Calbris 1991; Cienki 1998; Kendon 2000, 2004; Mc. Neill 1992, 2005; Müller 1998, 2004, 2007; Webb 1996; Streeck 2003; Sweetser 1998; reseach project “Towards a grammar of gesture, “ Berlin / Viadrina FFO) ----Data: genre: academic discourse / introductory linguistics courses subject matter: linguistic form, grammar, & linguistic theory gesture types: “referential gestures” of abstract concepts and structures (Müller 1998) meta-linguistic & meta-grammatical co-speech gestures

inventory of gestural sign carriers in meta-linguistic & meta-grammatical gestures data-driven approach resulted in a set of prominent hand configurations & motion patterns recurrent forms and recurrent form-meaning mappings – manipulation of imaginary objects of different sizes and dimensions (solid objects and containers with an inside) – geometrical shapes basic shapes: squares, triangles, regtangles, (semi-) circles, etc. lines: • straight, curved, wavy along horizontal, vertical und diagonal axes other patterns evoke some of image-schematic patterns suggested in the cognitive linguistics literature => ‘material’ basis for metaphorical and metonymic projections Mittelberg (2006, 2008, fc)

Johnson‘s original definition of “image schemas”: “recurring, dynamic patterns of our perceptual interactions and motor programs that give coherence and structure to our experience. ” (Johnson 1987, XIV) Johnson (2005, 31): “But let us not forget that the truly significant work done by image schemas is tied to the fact that they are not merely skeletons or abstractions. They are recurring patterns of organism-environment interactions that exist in the felt qualities of our experience, understanding and thought. ”

image schemata instantiated in the gesture data SUPPORT CONTAINMENT OBJECT SOURCE-PATH-GOAL BALANCE SCALE CYCLE ITERATION FRONT-BACK FORCE ADJACENCY CONTACT PART-WHOLE (“puoh-tray, ” “puoh-cup”) (“puoh-cup, ” “fist, ” “pcoh-box”) (“puoh-tray, ” “puoh-cup, ” “pcoh-box, ”, “fist”) (“hori-trace, ” “vert-trace, ” “diag-trace; ) (“puoh-tray-bh, ” “puoh-cup-bh, ” “fist-bh, ” “sym-offshoot”) (“scale”) (“circle-bh, ” “wrist-rotation, ” “rotation lateral”) (“push, ” “pull”) (“push, ” “pull”, “hori-join, ” “sym-offshoot”) (Johnson 1987; Gibbs 2005; Hampe 2005; Lakoff & Johnson 1999; Mandler 1996; Talmy 1988) Gesture research: iconic and metaphoric gestures (Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; Mittelberg 2008, fc. ; Müller 1998, 2007, 2008, fc. ; Sweetser 1998; Williams 2004, 2008; Zlatev 2005; inter alia)

metonymic pathways as emphasized by relevance theorists (e. g. , Sperber & Wilson 2002) and cognitive linguists (Gibbs 1994; Lakoff 1987; Fauconnier & Turner 2002), “metonymic pathways” are part of the cognitive competence of speakers and listeners => provide natural inference schemata (Panther & Thornburg 2003, 2004) One definition of metonymy: “Metonymy is a cognitive process in which one conceptual entity, the vehicle, provides mental access to another conceptual entity, the target, within the same domain or ICM. ” “Proust is on the top shelf. ” (Kövecses and Radden 1998: 39)

Jakobson (1956): “Two types of language and two types of aphasic disturbances” inspired by Peirce’s (1955) concepts of similarity & contiguity theory of metaphor & metonymy as two major modes of association & signification Jakobson § devotes equal attention to both tropes and those who use his work posit a continuum between the two. (cf. Croft 1993; Dirven 2002; Lodge 1977; Radden 2000; Waugh 1998) § no absolute categories: they interact in any process of meaning-making to various degrees § hierarchy: the function of a given sign depends on the preponderance of one mode over the others. The aim here is to show metonymic principles lead into the interpretation of metaphoric gestures: metonymy first, metaphor second

Jakobson further distinguished between internal metonymy: contiguity relationships between sign & object synecdoche (pars pro toto) “all hands on deck” (hands stand for entire bodies) external metonymy: contiguity relationships between signs or between entities in extra-linguistic, discursive, or conceptual space “metonymy proper” (adjacency, contact) “The White House remains silent. ” “The White House” stands for U. S. president / spokesperson (place for person)

I. internal metonymy (within metaphoric gestures) the formation of the gestural sign gestures exhibit the basic principle of semiotic representation => that is, that all semiotic representations are partial these abstraction processes are driven by synecdoche: by picking out the locally pragmatically salient aspects of an object / action e. g. , shape, dimensions, texture, manner of movement for example, a gesture representing a picture frame (Bouvet 2001; Cienki & Müller 2006; Gibbs 1994; Müller 1998, fc. ; Sweetser 2001; Taub 2001; Wilcox 2004; Wilcox & Morford 2007) most work on metonymy in co-speech gesture has focused on this kind of metonymic motivation

internal metonymy (within metaphoric gestures) a horizontal line stands for a sentence “a sentence is a string of words”

external metonymy (within metaphoric gestures) space extending between fingers is assigned meaning (adjacency/contact) here: the word “Diana” in a sentence

external metonymy / metonymy of place “subcategory” {video clip}

Conotiguity relations between body and objects “there is” pointing gesture (“shifters”; Jakobson) “the main verb” CONTAINMENT (cup) metaphor [object manipulation & external metonymy] “there is what‘s called the main verb”

Contiguity relations holding between hands & object “sub-category” imaginary OBJECT seemingly being held between two hands [external metonymy] comparably low position in gesture space & in relation to body [metonymy of place]

metonymy first, metaphor second (Mittelberg & Waugh fc. ) 1. Metonymy (external): source: hands target: object (physical) (imaginary) 2. Metaphor: source: OBJECT target: IDEA (imaginary) (conceptual) co-speech: “sub-category“ (technical) metonymic target = metaphoric source (imaginary object) => PLACE FOR FUNCTION “sub-category”

some conclusions The assumption that some metaphors are grounded in metonymy seems to hold in the gestures discussed. (Barcelona 2000; Dirven 2002; Fauconnier & Turner 2002; Geeraerts 2002; Goossens 1995; Jakobson 1956, 1960; Lakoff 1987; Lodge 1977; Radden 2000; inter alia) figures of thought may manifest themselves in gesture even if the concurrent speech is nonfigurative (“noun”, “Diane”, “teach-”, “infinitive”, “sentence”, etc. ) (cf. Cienki 1998, 2008; Cienki & Müller 2008, fc. ; Müller 2008)

Gesture and cognitive linguistics Alan Cienki

• Given: – the goal in cognitive linguistics to characterize language in a way that is coherent with what is known about cognitive processing in general – the cog. ling. focus on actual linguistic behavior – the universality of gesture with speech Gesture can provide additional insight into conceptualizations underlying semantic and grammatical structures

Gesture in relation to the study of several topics in cognitive linguistics • • • metaphor cognitive models mental spaces metonymy and cognitive reference points image schemas

Metaphor in words and gestures (Cienki 1998, 2008 a) • Gestures can provide evidence of activation (on some level) of an image being used to characterize a given topic • (a mapping from one domain to another, i. e. , [conceptual] metaphor, on some level) • The inclusion of gesture data provides a way out of the criticism of circularity of conceptual metaphor theory • (that is: language provides evidence of conceptual metaphors, and we find evidence of conceptual metaphors in language)

Metaphor in words and gestures (Cienki 1998, 2008 a) • Metaphor can appear in words without accompanying gestures • Metaphor can appear in gestures without accompanying metaphorically used words • The ‘same’ metaphor can appear in words and gestures in an utterance • Different metaphors can be expressed simultaneously in words and gestures

Cognitive models • A cognitive model as the way an individual characterizes knowledge for a given domain (compare schema, script, frame, etc. ) • Cultural models as shared cognitive models • These models can be constituted by metaphors to varying degrees

Russian model of chestnost’ Dlia menia chestnost’ eto nekaia abso`liutnaia kategoriia For me chestnost’ is a kind of absolute category. Kogda vot iest’ situatsiia, When there’s this situation, seichas postupit’ chestno `tak. then [you need] to act honestly like this. {video clip}

Cognitive models and gesture (Cienki 1999) • Metaphors can highlight aspects of cognitive/cultural models • Gesture data can provide evidence of the speaker’s online use of a cognitive model • This can be valuable for the study of (aspects of) cognitive models which are not expressed in words

Mental spaces • Developed from work by Gilles Fauconnier (1985) • Refers to “small conceptual ‘packets’ constructed as we think and talk” • Linguistic forms – reveal the speaker’s use of various mental spaces, – and they provide cues to the listener for how to construct the mental spaces used by the speaker • Examples: hypothetical constructions (if…, then…)

Mental spaces: gestural example (Cienki 2008 b) It depends on the student, but it also depends on the teacher. {video clip}

Mental spaces, metaphor, and gesture • IDEAS AS SPACES (a kind of ontological metaphor, i. e. , ABSTRACT AS CONCRETE)

Metonymy and reference points • Langacker (1993) notes how we regularly make use of one (known) entity to invoke mental contact with another (less well-known) entity: – > cognitive reference point • Metonymy as one means of doing this

Metonymy and reference points (Cienki 2007) • Appears in gesture in many ways: – The hand enacts a function it would actually perform (possibly with an object) (e. g. , you have to write fast!) – Or the hand represents (embodies) part of an object (viz. Darstellungsweisen [modes of representation], Müller 1998) – Or the hand points to a reference point

Metonymy in a referential gesture – hand points `That was dishonest, ’cause she didn’t… • abstract target grounded on concrete referent {video clip}

Image schemas “An image schema is a recurring, dynamic pattern of our perceptual interactions and motor programs that gives coherence and structure to our experience. ” (Johnson 1987: xiv) E. g. : CONTAINER PATH CYCLE PART-WHOLE SURFACE BALANCE COUNTERFORCE LINK CONTACT CENTER-PERIPHERY

If we accept that. . . • image schemas constitute significant patterns in our embodied experience • most research on image schemas has been based on linguistic data • manual gesture is a behavior involving a different kind of embodied experience than speech

• What role might image schemas play in our production, or comprehension, of gestures?

Image schemas and gesture (Cienki 2005) • compared the categorization of two types of gestures in terms of image schemas: – Abstract Referential (metaphoric) gestures – combined group of discourse-structuring and performative gestures (= “Other” gestures) • two conditions: – gestures with no sound – gestures with speech

Image schemas and gesture (Cienki 2005) • Can people reliably characterize the forms of gestures using image schemas as descriptors? • Will this vary if they hear the accompanying speech or not?

Image schemas and gesture: findings (Cienki 2005) • both groups of gestures were characterized with reliable agreement using a set of image schemas as descriptors – (this was true with and without sound) • Abstract Referential (metaphoric) gestures were categorized with a slightly greater degree of agreement than the group of “Other” gestures (in both conditions)

Conclusions • Cognitive linguistics claims to be a usage-based theory of language; gesture with speech is part of spoken language usage • Taking this into account can lead to multimodal analyses of grammar (e. g. , Fricke 2008)

Conclusions • The study of gesture with speech prioritizes research on the dynamic processes of language and cognitive processing, including mental simulation • Can help move cognitive linguistics beyond its focus on the individual speaker and connect it to research on interaction and intersubjectivity

Looking ahead • Creativity in gesture (IM & AC) • Open-class and closed-class systems in co-speech gesture (IM) • Spoken language semantics (AC)

• Creativity in gesture (IM & AC)

Creativity in gesture: forms – use of more extended gesture space – greater dynamicity – coordinated with other conspicuous embodied expressions (e. g. , body shifts, facial expression) • Differences in scale between non-creative and creative gestures

Creativity in gesture: functions – graphic representation of something concrete – or graphic representation of an (abstract) idea via a physical image –> metaphoric gesturing – or clearly performing a comment reflecting the speaker’s attitude toward the topic • Creative gestures appear to serve discrete functions (versus greater multifunctionality of less creative gestures)

Creativity in gesture: contexts of production • Impetus to express a complex idea concisely –> High communicative pressure – e. g. , pedagogic context, or shared problem solving • ongoing creative work • the “object” being presented needs to be presented

• Open-class and closed-class systems in co-speech gesture (IM)

open-class & closed-class systems in co-speech gesture In language, there is a fundamental difference between content words (e. g. , nouns, verbs, adjectives); open-class system (Talmy 2000) function words (propositions, personal & demonstrative pronouns, conjunctions, etc. ) expressing spatial, temporal, logical and functional relations; closed-class system How does the difference between substance and relations (structure) play out in co-speech gesture?

open-class & closed-class systems in co-speech gesture two lines of inquiry: A) Do gestures accompanying linguistic expressions of either kind differ in form and function? B) Is there a difference between gestures depicting contents and gestures depicting relations - without considering the concurrent speech. Since gestures are inherently indexical a focus will be on how they support or function as “shifters” (Jakobson 1960), e. g. those linguistic forms that take on a different meaning in a given moment of speech

• Spoken language semantics (AC)

Spoken language semantics • Premises: – Inherent bias in formal linguistic research towards language as it is written – Cognitive linguistic approach to semantics as conceptualization – Different processes of conceptualization during speaking versus writing, listening versus reading • Thesis: The semantics of spoken language is different in form and content than that of written language

The semantics of spoken language as different in form and content than that of written language • Spoken language functions differently than written language in terms of – – – time frame grammatical structures used embodied aspects of use involvement of imagery dynamic processes of expressing meaning • Proposal: This calls for a distinct approach to analyzing the semantics of spoken language

Thank you!