Speech Reading and the Linguistic Process A conference

Speech, Reading and the Linguistic Process: A conference in honor of Ignatius G. Mattingly 1

Prosody and Reading Janet Dean Fodor Graduate Center, CUNY 2

Collaborative research with Dianne Bradley, Eva Fernández, Yuki Hirose, Yoshihisa Kitagawa, Nenad Lovrić, Deirdre Quinn, Amit Shaked. 3

Ignatius’ synthesis by rule research 1964 Holmes, J. N. , Mattingly I. G. , and Shearme, J. N. Speech synthesis by rule. Language and Speech 7, 127. 1966 Mattingly, I. G. Synthesis by rule of prosodic features. Language and Speech 9, 1 -13. 1968 Mattingly, I. G. Synthesis by rule of General American English. Haskins Laboratories Status Report on Speech Research. Supplement. 4

Ignatius’ synthesis by rule research 1964 Holmes, J. N. , Mattingly I. G. , and Shearme, J. N. Speech synthesis by rule. Language and Speech 7, 127. 1966 Mattingly, I. G. Synthesis by rule of prosodic features. Language and Speech 9, 1 -13. 1968 Mattingly, I. G. Synthesis by rule of General American English. Haskins Laboratories Status Report on Speech Research. Supplement. 5

Some input strings for the JSRU speech synthesis system A ‘BERD in DHA ‘HAAND IZ WERTH ‘TUU IN DHA ‘BOOSH. DID YUU ‘KUM BIE ‘MOATAKAR? Prosodic marks: . falling tone, final pause ? rising tone, final pause + fall-rise tone, non-final pause 6

Realization of these tones Adjust the F 0 (pitch) of the syllable immediately following the stressed syllable of the last prominent word in the sentence 8 steps down, for a falling tone 4 steps up, for a rising tone and extend the fall / rise to end of sentence. 7

The motor car Did you come by motor car? (original, synthesized) Thanks to Matthew Mattingly, Gary Chant 8

The original motorcar Did you co me by mo tor car? 9

The available technology – a computer An excerpt from an interview with Ignatius by Matthew Mattingly in 1999: “The procedure was to make paper tapes at JSRU…you couldn’t see what you were typing at the time. . . you could run the tape through a printer and find your mistakes… When you’d made these tapes, that consisted of your program and your data, you shipped the whole thing to the installation several miles away, where the computer was. They would run the program, and make more paper tapes as output, and ship it all back to you. In the case of the speech synthesis, we’d send a paper tape with the synthesis by rule program, and more paper tapes with the input sentences that we were synthesizing, and they would send back paper tapes that contained groups of values for each parameter in the synthesizer. The value for the pitch, and each of the three formants, and the amplitudes and so on, one such set for every ten milliseconds. We would take this paper tape and mount it on a series of pulleys, and an electric eye read the tape and sent the signals to the actual speech synthesizer. As the tape went round on the pulleys, you heard the output that you’d called for. ” 10

The motor car Did you come by motor car? (original) Did you come by motor car? (JDF high) Did you come by motor car? (JDF low) You came by motor car. (JDF) ? ? 11

JDF Did you come by motorcar? H* vs L* question 12

JDF Did you come by motorcar? Question vs statement 13

Group data by Tanya Viger 15 Ss, 5 items e. g. , Su sa nna has an en e my. (Does) Su sa nna have an en e my? 14

The motor car Did you come by motor car? (original) Did you come by motor car? (JDF high) Did you come by motor car? (JDF low) You came by motor car. (JDF) ? ? 15

Relevance for sentence processing In speech, the prosodic contour can disambiguate some syntactic ambiguities. Many studies since Lehiste 1973. They fed her dog biscuits. They fed her dog-biscuits. Flying planes can be dangerous. 16

At CUNY we study silent prosody n For readers, few prosodic cues in the input. n In reading aloud, the reader computes a prosodic contour and imposes it on the word string. n We claim that this happens in silent reading too. n How can we know this? Why does it matter? 17

Syntactic parsing may be biased by silent prosody For an ambiguous sentence, a reader who assigns a certain prosodic contour may treat that prosody as if it had been part of the input stimulus, and use it to resolve the ambiguity. If so, ambiguity resolution preferences will not be a reliable source of information about syntactic parsing strategies – even in experiments on silent reading. 18

In fact, a crisis in syntactic parsing theory: universal parsing principles in jeopardy 1978 Frazier & Fodor: The Sausage Machine. Hypothesis: The human sentence parsing mechanism is innate (universal); just plug in a grammar. The parser’s task: Take in words; build a syntactic tree. n n Chop the word string into approx 6 -word chunks Minimal Attachment Late Closure ( local / low attachment) Minimal Chain Principle 1988 Cuetos & Mitchell: Late Closure is not universal. Spanish doesn’t obey it. 19

The relative clause attachment ambiguity Someone shot the servant of the actress who was… Who was on the balcony? The servant = HIGH ATTACHMT favored in Spanish The actress = LOW ATTACHMT favored in English * LC 20

the servant of the actress who… NP 2 the N’ 2 N 1 PP servant 2 of NP 2 the N’ 2 N 2 RC actress NP 3 the N’ 3 N’ RC 2 N 1 PP servant 2 of NP 2 the N 2 actress 21

Proposed explanation Prosodic breaks are optimally aligned with syntactic phrase edges (Selkirk 2000) For perceivers, a prosodic discontinuity favors a syntactic discontinuity. Prosody N 1 of N 2 / RC HIGH attachment SPANISH Prosody N 1 / of N 2 RC LOW attachment Prosody N 1 of N 2 RC LOW attachment ENGLISH 22

Prosody could explain the curious grouping of languages Hypothesis: Weaker vs stronger tendency to break before RC. LOW RC-ATTACHMENT TENDENCY American English British English Egyptian Arabic Norwegian Romanian Swedish HIGH RC-ATTACHMENT TENDENCY Afrikaans Croatian Dutch French German Russian Spanish 23

Prosody can explain a universal effect of RC-length n Short RC Less likely break before RC More low attachment …the servant of the actress who smokes twenty cigarettes a day. n n This is reminiscent of the ‘packaging’ effects of the Sausage Machine, which were also length-dependent. HIGH attachment is easier for phrases packaged as a separate package. Package = prosodic phrase? 24

Crisis over. Why do readers of different languages sometimes make different ambiguity resolution choices? Many interesting and plausible explanations, but the evidence now suggests: n n n The parsing routines obey fully universal principles. Any differences are due to differences in the grammar that is applied to input strings of words. The language-specific grammar includes prosodic principles – which are applied even in silent reading. 25

Prosody in silent reading can explain a variety of other parsing facts n n n Effect of preposition in Croatian, Hebrew, German, Greek. Clause boundary placement in Japanese. Effect of focus particles in German. Wh-scope interpretation in Japanese. Not-because scope preference in English. PP-attachment in English questions. 26

Prosody can explain the effect of prepositions n Croatian (Lovric 2003) no preposition between nouns no prosodic break there prosodic break before RC high RC-attachment n preposition between the nouns prosodic break there no prosodic break before RC low RC-attachment Hebrew (Shaked 2004) Similar results but shifted along the scale. Hebrew “construct state” with no preposition is one phonological word. Permits absolutely no prosodic break between the nouns. 27

Prosody can explain preferences in garden-path reanalysis n n Hirose 1998: Japanese clause boundaries are highly ambiguous. Readers tends to locate them where there is a prosodic break for reasons of phrase length. Bader 1998: German focus particles force a particular stress pattern. Gardenpath recovery is harder if the prosody needs correcting as well as the syntax. 28

Effects of focus prosody as well as prosodic phrasing n Kitagawa & Fodor 2004: Japanese wh-questions are ambiguous if there are 2 clauses, each with a potential scope-marker for the wh-phrase. [[……wh-NP……… ka]………ka] n n Scope can be disambiguated by post-focus deprosodification, which extends from the wh-phrase (focus) to its (later) scope-marker. Prosodic pressures (e. g. , retain rhythmicity; align with syntax) predict which scope interpretation readers prefer in different cases. 29

Not-because scope ambiguity (pilot data only, so far) n Koizumi 2004: Based on Frazier & Clifton 1996. Preferred BECAUSE > NOT: Sue doesn’t cry because she realizes life is hard. Inside an if-clause, preferred NOT > BECAUSE: n a. b. c. d. § Sue didn’t cry because she was in public. Was she tearful later? If Sue didn’t cry because she was in public, was she tearful later? Sue didn’t cry because she felt lonely. What else was the matter? If Sue didn’t cry because she felt lonely, what else was the matter? The if-clause context because ▪ reduces the prosodic break before ▪ induces F 0 rise at end of the because-clause This prosody is typical of the NOT>BECAUSE reading (Hirschberg & Avesani ) (But also pragmatics of if? How to dissociate prosody & pragmatics? ) 1997 30

More on NOT-BECAUSE Troseth, Fodor, Koizumi & Fernandez 2004: Force the NOT > BECAUSE reading by using negative polarity item: John didn’t leave the meeting early because he was mad at anyone. Grammaticality judgment task. Readers accepted only 14% Listeners accepted 49% - but they regard the sentence as incomplete. 31

Now – the prosody of questions A current investigation, far from complete n PP-attachment in English questions, e. g. , a. The nanny seated the cranky little child on the swing in Oakwood Park. b. The nanny seated the cranky little child on the swing in his stroller. DIFFICULT! c. Did the nanny seat the cranky little child on the swing in his stroller? NOT SO DIFFICULT? n Hypothesis: final rise in questions is a discontinuity favors a discontinuity in the syntactic tree structure facilitates high attachment of the final PP. Fodor, Bradley & Shaham 2004 32

PP 1 high reanalyze PP 1 to low VP 9 V NP PP 1 seated child on swing PP 2 in stroller! VP 9 V seated NP 1 PP 2 in stroller N PP 1 child on swing 33

Recall the sharp prosodic discontinuity in questions e. g. , Su sa nna has an en e my. (Does) Su sa nna have an en e my? 34

In sum Ignatius’ research encompassed both prosody and reading It seems now that these are even more closely bound together than we knew I am happy to be following (a little bit) in his footsteps. 35