Dialogue mechanisms for mutual understanding Simon Garrod Institute

Dialogue mechanisms for mutual understanding Simon Garrod Institute of Neuroscience and Psychology, Glasgow

Acknowledgement Martin Pickering University of Edinburgh

Overview • What is mutual understanding? • Distributed dialogue control mechanism – Backward Looking – Interactive Alignment – Forward Looking – Joint Prediction • Interactive Alignment & communicative success • Joint Prediction, monitoring & repair

What is Mutual Understanding? • (1) A & B understand utterance u in the same way as each other – Achieved through interactive alignment – Using the same linguistic forms (phonologicalsyntactic-semantic) in the same way as each other • (2) A & B both believe (1) – A&B believe they are well aligned – Alignment monitoring through joint prediction leading to repair when necessary

Distributed control and Mutual Understanding in dialogue Past Dialogue A Representations Future dialogue B’s Prediction of A Interactive Alignment B Representations A’s Prediction of B Mutual Understanding

Alignment & Prediction in dialogue From Schegloff (1996) Bee: Ava: He was handing me the book and he told me twenty dollars I almost dropped it ‘hh thhunh. ‘hhh I said but fer twenty dollars I better hh ‘hh yi’know, [hold onto it] ‘hh! [not drop it ] huhh huh

Overview • What is mutual understanding? • Distributed dialogue control mechanism – Backward Looking – Interactive Alignment – Forward Looking – Joint Prediction • Interactive Alignment & communicative success • Joint Prediction, monitoring & repair

Parity&Priming: +ve feedback system for alignment (Garrod&Pickering, TICS, 2004) Garrod & Pickering, TICS (2004)

Interactive Alignment Model (Pickering&Garrod, BBS, 2004) Automatic alignment channels

Does alignment correlate with communicative success? • Fusaroli et al (2012) – Used Bahrami et al. ’s joint psychophysical judgment task to estimate communicative success • Reitter & Moore (2014) – Used Map Task dialogues with associated path -matching scores to estimate communicative success

Bahrami et al. ’s (Science, 2010) Joint Psychophysical Task

Linguistic alignment predicts joint benefit in Bahrami’s task • Fusaroli et al (Psych. Sci. 2012) • Successful pairs aligned on task relevant expressions (r(14)= 0. 5 -0. 67 alignment predicts joint benefit) • But not for all expressions

Map task path overlap as measure of communicative success

Alignment and communicative success • Reitter & Moore (2014) – Short-term syntactic priming (as decay rate in rule repetition) did not correlate with success – Long-term syntactic priming (rule repetition between early and late in dialogue) did correlate with success – Similarly long-term lexical repetition correlated with success

Conclusion about alignment and Mutual Understanding • Alignment on task relevant language predicts degree of mutual understanding • Long-term alignment (P&G’s 2004 ‘routinization’) enhances mutual understanding

Overview • What is mutual understanding? • Two dynamic dialogue mechanisms: – Backward Looking – Interactive Alignment – Forward Looking – Joint Prediction • Interactive Alignment & communicative success • Joint Prediction & Monitoring – Joint monitoring & repair using forward models

Dialogue is a challenge for accounts of monitoring Joint productions & distributed control (1) Horton & Gerrig(2005) (2) Tannen(1989)

Monitoring and dialogue • • How can we converse so fluently? Requires efficient monitoring Self-monitoring using forward models Other-monitoring through covert imitation using forward models • Joint monitoring combines self- and othermonitoring to support fluent & aligned dialogue

Self-monitoring using forward models • Pickering & Garrod’s (2013) integrated theory of language production & comprehension • Depends heavily on the use of forward modeling and prediction • Self-monitoring involves comparison of predicted utterance and its implementation • By analogy with action control more generally

Pickering & Garrod ‘Integrated theory’ BBS 2013 • • Production is a kind of action Comprehension is a kind of action perception Dialogue is joint action Production and comprehension are interwoven – Effects of comprehension processes on production and vice versa in behavioural experiments – Overlap of brain circuits for production and comprehension – Tight coupling in dialogue • Speakers predict what they intend to produce, and listeners predict what the speaker will say next at many linguistic levels

Forward modeling in action control (e. g. Wolpert, ‘ 97) Action command u(t) Action implementer Act a(t) Perceptual implementer Percept s(t) Predicted percept ŝ(t) Efference copy Forward perceptual model Forward action model Predicted act â(t) comparator

Forward modeling in language production • Action implementer replaced by production implementer • Perceptual implementer replaced by comprehension implementer • Action command is the production command – Drives the production implementer – Efference copy drives the forward model • Monitor compares the utterance percept and the predicted utterance percept

Self-monitoring with forward models Implementer Production i(t) command inverse models Utterance percept Predicted utterance percept cat Efference copy Forward models monitor

Production evidence • Rate of self-monitoring (Hartsuiker et al. 2001) – “Move v- horizontally” (Levelt, 1983) – Message>semantics (175 ms. )>syntax (75 ms. )>phon(205 ms. )>artic(145 ms) – Error detection to correction requires forward model based monitoring • Reafference cancellation during speech – MEG M 100 reduction for undistorted vs distorted speech feedback (Heinks. Maldonado et al. 2006). – ECo. G suppression of auditory cortical activity during speech (Flinker et al. 2010) – Imagining and producing speech lead to same rapid MEG response in auditory cortex (Tian & Poeppel, 2010)

Self -monitoring leads to rapid adaptation (Tourville et al. 2008) • Tourville et al. (2008) distorted feedback by shifting F 1 pitch up or down 30% • Speakers adapted within ≈ 100 msecs • Monitor discrepancy = (predicted F 1)/3 so speakers use inverse model to correct F 1 by - (predicted F 1)/3 • In production small changes corrected internally through inverse model (perhaps unconsciously) • Big changes (e. g. “move v- horizontally” (Levelt, ’ 83) invoke more general conflict resolution & reformulation of production command

Other monitoring • Listeners can monitor speakers’ utterances • Underlies other repair – Parent-child dialogue (Chouniard & Clark, 2003) – Instructional dialogue • Listeners predict other’s utterances as they unfold via covert imitation & forward modeling • Other-monitoring then compares predicted with observed utterances at many linguistic levels

Covert imitation aids speech recognition • D’Ausilio et al. (2009) – Double-pulse TMS to motor cortex • Lip control area > faster & more accurate responses to lip-articulated phonemes (/pa/ /ba/) • Tongue control area >faster & more accurate responses to tongue-articulated phonemes (/ta/ /da/) – This suggests that covert imitation facilitates speech recognition • Möttönen et al(2009) also Rogers et al (2014) – Repetitive TMS or c. TBS disrupting lip control in left primary motor cortex impairs categorical perception of speech sounds involving the lips (e. g. , /ba/-/da/), but not of those involving other articulators (e. g. , /ka/ -/ga/).

Other monitoring depends on predicting ‘what’ others say and ‘when’ it is said. • ‘What’ prediction gives the content of other’s utterance • ‘When’ prediction indicates when the content will appear • ‘what’ + ‘when’ prediction needed to support fluent and seamless turn-taking • Accuracy of turn end detection depends on accuracy of prediction (De Ruiter et al. 2006)

Low frequency oscillatory entrainment predicts ‘when’ Gross et al. (2013) Plos Biology Listeners’ brains track low frequency (theta) oscillations in speech Phase in right Auditory Cortex Amplitude in left

Low-frequency oscillatory entrainment • Paces the Forward Model prediction • q oscillation correspond to ‘beat’ (Schegloff, 2000) • Degree of entrainment modulated by Left Frontal & Motor Areas (Park et al. 2015) • Drives the rate of prediction and the rate of subsequent production – Brain-to-brain coupling for low frequency (delta, theta) oscillations in conversation with EEG (Kawasaki et al. , 2013) • Supports fluent turn-transition

Predicting ‘what’ & ‘when’ of other’s utterance

Other prediction in dialogue

How self- & other-monitoring differ • Self monitoring checks implementation of communicative intentions • Other monitoring allows listeners to check their predictions of other’s intentions • Leads to updating of predictions • Indicates the degree to which the utterance conforms to predictions – Small discrepancy indicates good understanding – Large discrepancy indicates possible misinterpretation

Internal and external modification • When other monitoring discrepancy is large one of two outcomes: • Internal modification using general conflict resolution (including ‘mentalizing’) – Interpreting indirect responses leads to activation of ‘mentalizing’ regions (Bašnáková et al. , 2013) • External modification – giving feedback to speaker

External modification (Drew, 1997) Hal: …an’ Leslie ‘t was marv’lous (. ) D’you know he had (. ) forty-nine g’rillas. . hh ththere. (. ) br[eeding in ( ) Leslie: [pf- f- Forty-nine wha: t? Hal: G’rillas Leslie: . hh Oh ye-s?

Joint monitoring in dialogue • Dialogue interweaves production & comprehension • As a consequence self- and othermonitoring become interwoven • Interlocutors predict each other’s utterances (Pickering & Garrod, 2013) • Joint prediction underlies joint monitoring in dialogue

J Joint prediction in dialogue (Pickering & Garrod, 2013)

Rubenstein’s hands again • 15 ---A: [just completely soft and [limp • 16 ---B: [mush • • Here, B interrupts A, and it is clear that B must be as ready to contribute as A. B could not contribute by comprehending A’s utterance and then preparing a response “from scratch” as traditional “serial monologue” accounts would assume. Instead, we propose that B covertly imitates A’s utterance, derives the act of communication, “runs” it ahead, and produces an overt completion Presuming what A would say, but in fact what B believes A would say

Monitoring and concurrent feedback A: and um it- you know it’s rea- it’s it was really good and of course she teaches theology that was another thing B: mm A: I- m- I- Isabelle B: Oh that’s great. • Presumably B has detected a discrepancy probably at ‘theology’ and reverts to external modification with ‘mm’

Other- and self-monitoring interact in dialogue • In both the Hal-Leslie and theology examples the concurrent feedback ‘forty-nine what? ’ and ‘mm’ lead to the interlocutor to stop immediately and correct their own utterance. • Concurrent feedback is incorporated as an input into self-monitoring during dialogue

Summary • Speakers & listeners monitor themselves and others using forward models • Self monitoring checks correct implementation (monologue) and takes into account concurrent feedback (dialogue) • Other monitoring checks for alignment in both monologue & dialogue • Joint monitoring in dialogue underlies fluent turn -taking

Monitoring and interactive alignment • In dialogue interlocutors align their linguistic representations and understanding (Pickering & Garrod, 2004) • Alignment supports other monitoring because it simplifies prediction by simulation • Joint monitoring helps to detect misalignment and instigate general conflict resolution and repair

Overall Conclusions • Mutual understanding achieved through two mechanisms • Backward looking mechanism of interactive alignment • Forward looking mechanism of predictionbased monitoring & repair • Prediction-by-simulation operates both in terms of content and timing • Content prediction digital/discrete • Timing prediction analogue

Two Dynamic mechanisms of Mutual Understanding Alignment Prediction Mutual Understanding

~ The End ~ Thank you

Intentional act of communication i(t) Production system Utterance [psem, psyn, pphon](t) Comprehension system Utterance percept [csem, csyn, cphon](t) Efference copy Predicted utterance percept [c^sem, c^syn, c^phon](t) Forward comprehension model Forward production model Predicted utterance [p^sem, p^syn, p^phon](t) monitor

Van Berkum et al. (2005)

Utterance [psem, psyn, pphon] Person B Comprehension system Covert imitation Person A Comprehension system Perceived utterance [csem, csyn, cphon]B(t) comparator Inverse model + context Derived intentional act of communication i. B(t) Production system Potential overt imitation [psem, psyn, pphon]B(t) Utterance percept [csem, csyn, cphon]B(t+1) Predicted utterance percept [c^sem, c^syn, c^phon]B(t+1) , Derived intentional act of communication i. B(t+1) Production system Potential overt completion [psem, psyn, pphon]B(t+1) Forward comprehension model Efference copy Forward production model Predicted utterance [p^sem, p^syn, p^phon]B(t+1)

Forward modeling of action perception (Wolpert et al. 2005) Person B B’s act a. B(t) B’s act a. B(t+1) Perceptual system Percept s. B(t) Covert imitation Person A Percept s. B(t+1) comparator Inverse model + context Derived Action command u. B(t) Derived Action command u. B(t+1) Action system Potential overt imitation a. B(t) Potential overt completion a. B(t+1) Predicted percept s. B^(t+1) Forward perceptual model Efference copy Forward action model Predicted act a. B^(t+1)

Covert imitation and forward modeling in language comprehension • How do people make predictions in comprehension? – Could use a comprehension-based route (based on what they have heard others say) – We propose they also use a production-based route • Roughly, “what they would say under the circumstances”, but modified by context – Predicting what the producer would say, not what they would say • Using structured linguistic representations – Making predictions about semantics, syntax, phonology

Evidence for prediction: semantics • Participants view edible and inedible objects (Altmann & Kamide, 1999) – They hear The man ate the… – And start looking at edible objects (predictive eye movements) – But not when ate is replaced by moved • Eye movements depend on the prior context, not just ate • Participants predict upcoming events as well as upcoming referent (Knoeferle et al. , 2005)