Watching the eyes when talking about size An

Watching the eyes when talking about size: An investigation of message formulation and utterance planning Sarah Brown-Schmidt, Michael K. Tanenhaus Presentation by Olatz Mendiola Ling 5555 November 17, 2010

Introduction n How do speakers plan and update messages that are translated into utterances during language production? How are message formulation and utterance planning coordinated? Speakers often update messages on the fly based on new information. q Message formulation and utterance planning are mixed in time and must communicate with one another at a relatively fine temporal grain.

Introduction n Levelt & Maassen (1981) q n Meyer (1996) q q q n Latency difference evidence that the unit of advance planning includes each of the lemmas in the subject noun phrase. Dutch speakers perform a naming task with auditory distractors. Naming latencies were delayed by a distractor that was phonologically related to the first noun in NP but not the second word in the NP. Distractors semantically related to either 1 st or 2 nd noun decreased latencies. Schriefers &Teruel (1999) q q q Only 1 st word is prepared before speech onset Presented phonological distractors to German speakers in a naming task. Phonological distractors related to 1 st syllable of 1 st word facilitated naming times, no effect found for distractors related to 2 nd word and distractors related to 2 nd syllable of 1 st word had a weak effect. In some cases, articulation may begin before the 1 st phonological word is entirely planned.

Introduction n The degree to which speakers pre-plan a complex phrase is highly variable. q q Amount of advance planning varies depending on the task and type of utterance Planning is partially under control of speaker. n Little work has addressed how messages are prepared n Griffin and Bock (2000) q q Difficult? Participants described simple events (woman shooting a man…) The sequence of eye movements reflected the order of constituents in the utterance. Speakers looked at picture objects about 800 ms to 1 s before naming them. This eye-voice lag is similar to the time it takes to initiate naming an object in isolation, suggesting that the eye-voice delay reflects word preparation.

Introduction n Timing of eye movements could provide insight into interplay between message formulation and utterance planning. n Work investigating gaze-speech link may typically conflate message formulation with utterance planning. n How disentangle message formulation form utterance planning? q creating situations in which the speaker, while in the process of planning or producing an utterance, encounters new information that requires revising the message. Brown-Schmidt Tanenhaus (2006) q If eye movements can be used to infer when the speaker 1 st encounters that information, then the timing between the uptake of the new information and the form of the utterance might shed light on the interface between message production and utterance planning.

Present Study. Experiment 1 n n n 18 pairs of participants Seated at different computers. 5 feet separation No eye contact Subject X Eye tracker + headset microphone Subject Y Headset microphone

Listener’s perspective Speaker’s perspective

Present Study n Hypothesis 1 q n Hypothesis 2 q n Fixation to the contrast will predict use or non-use of a size adjective. Higher rates of modification when contrast has been fixated. There is a systematic relationship btw timing of 1 st look to the contrast and the form of the utterance. Hypothesis 3 q Timing of looks to the contrast and use of a pre-nominal vs. post-noun repair for descriptions of complex and simple shapes should provide insight into the size of message planning units, and how repairs to these units affect utterance planning.

Results Manipulations successful in eliciting expected forms: n 1. 2. Size adj. typically used only when contrast present in display. Pre-nominal adj. used for Simple Shapes. Adj. in post-nominal PP used for Complex Shapes.

Results 1. Use of size adjectives n n n Contrast : size adj. on 98% of trials No contrast: size adj. on 27% of trials Statistically Significant Contrast/Simple: size adj. on 99% of trials Contrast/Complex: size adj. on 97% of trials No. Contrast/Simple: size adj. on 23% of trials No. Contrast/Complex: size adj. on 31 % of trials Not significant

Results 2. Utterance form q q n Contrast / Size adj. / Simple shape pre-nominal 89% of trials Contrast / Size adj. /Complex shape post-nominal PP 73% of trials Size adj. also occurred in post-noun repairs “The triangle, uh the small one” “The square with triangles, uh small ones” q q 11% of trials for simple shapes 16% of trials for complex shapes

Results. n Hypothesis 1 q q Fixation to the contrast will predict use or non-use of a size adjective. Higher rates of modification when contrast has been fixated. Size contrast / fixated n q Size adj. on 99% of trials Size contrast / not fixated n Size adj. on 68% of trials

Results. New message elements can be added to a planned referential description during or immediately before production n Time of the first fixation to the contrast relative to onset of utterance. Form of referring expression is related to timing btw 1 st look to size contrast and onset of utterance. ANOVA: q Effect of adjective position n q Effect of complexity. n q UNEXPECTED Supports Hypothesis 2 Earlier fixation for pre-nominal adj. Earlier fixation for references to simple shapes Interaction btw complexity and adj. position was not significant

Results n n Simple shapes: Negative relationship between first fixations and adj. onset time. Complex shape: marginal effect When need for a size adjective is noticed after speech onset, preparation of the repair is delayed by preparation of the original utterance. Lag between 1 st fixation to contrast and adjective onset q q Complex: 1220 ms Simple: 440 ms

Results n Simple shape q Prenominal n q Post repair n n 1546 ms after display 1061 ms after display Complex shape q Prenominal n q 1434 ms after display Post repair n 1390 ms after display

Conclusions n n n Fixation on size contrast predicts use of size adj. Timing of fixations predicts the form of the utterance. When preparing a modified referring expression, messages are incrementally prepared and passed onto utterance planning in units smaller than the size needed for an entire referring expression.

Towards experiment 2 n 2 unexpected results q 600 ms lag between fixation to the contrast and onset of utterance. 200 ms faster than literature reports. n q Size adj. were used on more than 25% of trials when there was no contrast. n n Explanation? A small number of shapes were repeatedly used on target trials, which may reduce the demands of lexical encoding Explanation? 25% of trials had targets with size contrast which might make inclusion of size in the description somewhat formulaic Experiment 2 q q Increasing target type Decreasing proportion of target trials

Experiment 2

Results n n Speakers rarely used a size adj. when there was no contrast in display = 1 % Speakers used a size adj. 72% of the references on trials with contrast Proportions consistent with other studies Of the size contrast trials where adj. was used: q q 62% pre-nominal 37% post-noun repairs

Results n Analysis was restricted to trials were there was a size contrast on same side of the river as target. Fluent and disfluent trials analyzed separately. n Fluent trials q Looks to contrast predicted whether the size of target would be mentioned (Hyp 1) = Exp 1 n n q 86% of fluent trials during which speakers looked at contrast, the referring expression included a size modifier. When there was no fixation to the contrast, modifier was only used in 19 % of trials First fixation to the contrast was systematically related to the form of the utterance. (Hyp 2) = Exp 1 n n The mean 1 st fixation to contrast for pre-nominal modifiers was 887 ms before onset of referring expression. The mean 1 st fixation to contrast with post-noun repairs was 14 ms before onset.

Results n Disfluent trials q q q When speakers fixated the contrast adj was used on 91%. When contrast was not fixated ajd was used 34%. First fixation to contrast systematically related to form of utterance. n n First fixations were on average 528 ms before onset of referring expression for utterance with pre-nominal modifiers. First fixations were on average +819 ms for post-noun repair utterances

Results. n n Earliest fixations are associated with fluent prenominal utterance Latest fixations are associated with a post-noun repair. Consistent with H 3: suggests speaker was able to use disfluency to buy time to include size in the noun phrase as a prenominla adj. rather than as a postnoun repair. =Exp 1. Relationship btw time speakers 1 st fixated the size contrast and the timing of the size adj. for postnoun repairs.

Results n Time to begin speaking q Average time between the highlighting of target and NP onset < 2 s q ≠ Exp 1. Utterance onset time was not significantly affected by the form of the utterance q For initially fluent utterances, speakers started pre-nominally modified expressions 345 ms faster than expressions with postnoun repairs (not reliable) q For disfluent utterances pre-nominally modified expressions were only 120 ms faster than utterances with postnoun repairs q Speakers uttered disfluent pre-nominally modified expressions on average 66 ms faster than fluent pre-nominally modified utterances (not reliable)

General Discussion n 3 results q q q 1. Speakers typically used size adj. in a referential expression only when there was a similar object in the referential domain that differed in size 2. Use of size adj was more likely if speaker had fixated on the size contrast. 3. Timing of first fixation to contrast was related to the form of referential expression n n Earlier – prenominal adj Later – postnoun repair

General Discussion n Preliminary support for hypotheses about interplay between message formulation and utterance planning. When an error is detected in the utterance plan after speaking has begun, initiation of a repair may be delayed by preparation of original utterance. Messages that will be mapped onto referring expressions can be constructed and passed onto utterance planning incrementally.

Eskerrik asko!