Language memory and brain A cognitive neuroscience perspective

Language, memory, and brain: A cognitive neuroscience perspective on first and second language Michael T. Ullman Departments of Neuroscience, Linguistics, Psychology and Neurology Georgetown University michael@georgetown. edu 1

Collaborators and Funders Georgetown Laura Babcock (now at Trieste, Italy) Harriet Bowden (now at Univ. of Tennessee) Claudia Brovetto (now at Montevideo, Uruguay) John Drury (now at SUNY Stony Brook) Cristina Dye (now at Newcastle, UK) Matthew Gelfand Sarah Grey (now at Penn State) Joshua Hartshorne (now at MIT) Darlene Howard Kaitlyn Litcofksy (now at Penn State) Jarrett Lovelett Christopher Maloof Robbin Miranda (now at Shafer Corp) Kara Morgan-Short (now at UIC) Elizabeth Pierpont (now at Wisconsin) Elizabeth Prado (now at UC Davis) Mariel Pullman Cristina Sanz Karsten Steinhauer (now at Mc. Gill) Kaitlyn Tagarelli Peter Turkeltaub Joao Verissimo (now at Potsdam) Matthew Walenski (now at Northwestern) Main collaborators Stefano Cappa (Milan, Italy) Gina Conti-Ramsden (Manchester, UK) Laurie Cutting (Vanderbilt) Ingrid Finger (Porto Alegre, Brazil) Myrna Gopnik (Mc. Gill, emeritus) Martina Hedenius (Uppsala, Sweden) Jarrad Lum (Melbourne, Australia) Stewart Mostofksy (KKI, Johns Hopkins) Helen Neville (Univ of Oregon) Dezso Nemeth (Hungary) Aaron Newman (Dalhousie) Steven Pinker (Harvard) Anu Shankar (Harvard) Bruce Tomblin (U. of Iowa) Heather van der Lely (Harvard; deceased) Funding NIH: R 01 MH 58189; R 01 HD 049347 (& -09 S 3); R 03 HD 050671; R 21 HD 087088 NSF: SBR-9905273; BCS-0519133; BCS-0001961; BCS-1124144; BCS-1439290 Defense: DAMD-17 -93 -V-3018/3019/3020, DAMD-17 -99 -2 -9007 Foundations: Mc. Donnell Foundation; National Alliance for Autism Research; Simons Foundation; Mabel Flory Trust Corporations: Pfizer, Inc. International: US-Israel Binational Science Foundation; 2 Grand Challenges Canada

Motivation and Approach Evolution and biology re-use existing systems for new functions • Air bladder in fish evolved into the lung; limbs became wings, hands, … • The wing may be used to swim, shade prey, … Therefore: Language should depend in part on pre-existing brain systems • We focus on two systems: Declarative memory and procedural memory Approach: • Test if and how various aspects of language depend on these systems Advantages of this approach: • From animal and human studies we know a lot about these systems (their computational, anatomical, molecular, genetic, etc. substrates) • Thus we can make specific novel testable predictions that might be 3 unwarranted in the more circumscribed study of language

Bottom Line Converging evidence suggests: • Language depends on both memory systems • Lexical and grammatical knowledge depend differentially on the two systems The learning and processing of Idiosyncratic (lexical) knowledge Rule-governed (grammatical) knowledge Declarative Memory Yes, to a fair extent, as a function of subject, item, and other factors. . . (eg, females, L 2, SLI, higher frequency items) Procedural Memory ? Yes The two memory systems play at least partially redundant roles 4

Declarative Memory System Characteristics: • • Knowledge for “what”: remembering an event (episodic); knowing a fact (semantic) Specialized for learning arbitrary bits of information and associating them Learning rapid in this system (single presentation of information) Knowledge not just explicit, but also implicit Functional Neuroanatomy: • Hippocampus, other medial temporal lobe structures: Learning and consolidation • Neocortex, especially in temporal lobes: Long-term storage • Frontal regions (BA 45/47): Recall Biological substrates: • The neurotransmitter acetylcholine; the hormone estrogen (females better than males) • Various genes play a role, including for the proteins BDNF, APOE, others 5

Procedural Memory System Characteristics: • Knowledge for “how”: Learning and processing of motor and cognitive skills • Examples: typing, riding a bike, sequences, categories, rules, route-learning (navigation) • May be specialized for aspects of sequences and rules (possibly: learning to predict) • Learning gradual; once learned, skills applied rapidly and reliably; excellent retention • Knowledge apparently only implicit Functional Neuroanatomy: • Frontal/basal-ganglia circuits; other structures (cerebellum) • Basal ganglia (caudate nucleus): important for learning & consolidation • Frontal (BA 44, (pre)motor cortex): more important for processing Biological substrates: • The neurotransmitter dopamine • Possible genes include those for the proteins FOXP 2, DARPP-32

The Two Memory Systems Interact Cooperatively: redundant mechanisms • Some functions are learnable only by one or the other system • Declarative memory: arbitrary bits of information and their relations • Procedural memory: possibly motor skills • Others can be learned by both systems, but generally in different ways • Route-learning (navigation), sequences, categories, rules • Various factors affect which system is relied on more (eg, functionality) Competitively: see-saw effect • Dysfunction of one system may enhance the functionality of the other • Estrogen enhances declarative memory and inhibits procedural memory 7

Declarative/Procedural Theory of Language Lexical Memory Grammar Rule-governed hierarchical least) Relative reliance of Memory store: (at memory system grammar on each & sequential (de-)composition of information: all L 2 vs. L 1, procedural forms: Subject factors: sex, word-specificcomplex system dysfunction -syntax of complex forms -simple cat; NP VP) Item factors: the frequency (thewords (cat) -morphology input, immersion vs. -irregulars: (dig-dug) Input factors: how much L 2 (regulars: walk -ed) instruction --phonology (novel forms: blick) complements (devour [direct obj]) Declarative Memory (Ullman, 2001, 2004, 2008, 2013, in press) Procedural Memory 8

Converging Evidence Behavioral/Psycholinguistic • Correlational evidence • Evidence for storage: frequency, imageability, similarity effects • Evidence for composition: working memory, priming effects Neurological • Developmental disorders: SLI, dyslexia, autism, TS, ADHD • Neurodegenerative disease: AD, PD, HD, semantic dementia • Aphasia, amnesia Neuroimaging • f. MRI • PET Electrophysiological • Event-Related Potentials 9

The rest of the talk: Evidence First language: Behavioral, neurological, neuroimaging, electrophysiological Second language: Behavioral, neurological, neuroimaging, electrophysiological Neurodevelopmental disorders (SLI): Behavioral, neuroimaging, electrophysiological 10

First Language Behavioral Evidence: Correlational Meta-analysis of correlational studies Children (ages 4 -13): 9 studies, 27 correlations, 554 children Correlations Declarative Memory Procedural Memory Lexicon r =. 41, p <. 001 r =. 04, p =. 727 Grammar (syntax, morphology) r =. 13, p =. 096 r =. 22, p =. 005 (Hamrick, Lum and Ullman, in preparation) 11

First Language Neurological Evidence Medial Temporal Lobe Amnesia Specific Language Impairment Medial Temporal Abnormal Normal Declarative Memory Abnormal Relatively Normal Word (Learning) Abnormal Relatively Normal Basal Ganglia Normal Abnormal Procedural Memory Normal Abnormal Grammar (Learning) Normal Abnormal • Similar dissociations between Alzheimer’s & Parkinson’s diseases (Ullman & Pierpont, 2005; Ullman, Pullman, Lovelett, Pierpont & Turkeltaub, Under Rev. ; Ullman, 2004; Lum, Conti-Ramsden, Page, and Ullman, 2012; Kensinger, Ullman & Corkin, 2001; Postle & Corkin, 12 1998; Meulemansa & Linden, 2003; Reber et al, 2003; Ferreira et al 2008)

First Language Neuroimaging Evidence Medial Temporal (hippocampus) Basal Ganglia (caudate nucleus) Activation Word Learning Artificial Grammar Learning ? Activation during chunk-learning Activation during rule-learning (Brietenstein et al, 2005; Lieberman et al, 2004; Forkstam et al, 2006; Moro et al, 2001; Wong et al. 2013; Folia et al, 2011; Karuza et al, 2013) 13

First Language Electrophysiological Evidence: Event-Related Potentials (ERPs) N 400 LAN Lexical anomalies Consistently No Grammatical anomalies Rarely Reasonably often • Also elicited by nonverbal semantic stimuli • Linked to temporal lobe structures • Declarative Memory • Also elicited by nonverbal sequences • Automatic, early • Linked to frontal regions • Procedural Memory 14

Second Language Hypotheses - Declarative memory improves during childhood, and plateaus in adolescence and early adulthood - The developmental trajectory of procedural memory is less clear, but it seems to be established early, and may attenuate during childhood - Therefore: earlier language learners (L 1, early L 2 learners) should rely more on procedural memory, and later learners more on declarative memory, for those tasks and functions (including grammar) than can rely on either system - However, procedural memory should only be somewhat attenuated in adults. With enough exposure (perhaps of the right type: immersion? ) at least some proceduralization of the grammar should occur (contra critical period) (Ullman, 2001, 2005, 2012, 2015, in press) 15

Second vs. First Language (L 2 vs. L 1): Predictions Declarative Memory Procedural Memory Lexical Memory: L 1 and L 2 Grammar L 2 > L 1 L 2: lower > higher exposure Grammar L 1 > L 2: higher > lower exposure 16

Second Language Behavioral Evidence: Correlational Meta-analysis of correlational studies Adult learned L 2: 6 studies, 17 correlations, 181 learners Correlations Grammar at low exposure/proficiency Grammar at high exposure/proficiency Declarative Memory Procedural Memory r =. 48, p <. 001 r = -. 17, p =. 155 r =. 19, p =. 181 (Hamrick, Lum and Ullman, in preparation) r =. 43, p <. 001 17

Second Language Neurological Evidence Lesion Temporal Lobe (Alzheimer’s disease) Frontal or Basal Ganglia (Parkinson’s disease, stroke) L 1 = L 2 Lexical Not Available L 2 < L 1 Grammatical L 2 higher exposure = L 2 lower exposure L 1 < L 2 higher exposure < L 2 lower exposure (Fabbro and Paradis, 1995; Fabbro, 1999; Hyltenstam and Stroud, 1989; Zanini et al 2004; Johari et al, 2013; Ullman, 2001) 18

Second Language Neuroimaging Evidence Activation Likelihood Estimation (ALE) meta-analysis: • 22 f. MRI/PET studies, with a total of 432 subjects • Preliminary analyses: 1) Lexical/semantics: L 1 and L 2 similar (e. g. , BA 45/47, temporal pole, superior temporal, fusiform) but more bilateral in L 2 2) Grammar: L 1 and L 2 similar (e. g. , BA 44, BA 6) but more bilateral in L 2, and different: basal ganglia, especially caudate (head), only in L 2 (Ullman, Tagarelli, Grey, & Turkeltaub, in prep. ) 19

Second Language Electrophysiological Evidence: ERPs Lexical/semantic anomalies L 1 N 400 Syntactic anomalies LAN Low exposure L 2 N 400 No LAN High exposure L 2 N 400 Sometimes N 400 LAN 20 (Ullman, 2001; Ullman, 2005; Ullman, 2012; Morgan-Short and Ullman, 2012; Bowden et al 2013; Mc. Laughlin et al 2010; Osterhout et al 2008; Tanner et al 2013; Weber and Lavric 2008; Friederici et al 2002; Morgan-Short, Steinhauer, Sanz & Ullman 2012; Morgan-Short, Finger, Grey & Ullman 2012)

Neurodevelopmental Disorders: A Neurocognitive Hypothesis: Various developmental disorders result (at least in part) • from abnormalities of brain structures underlying procedural memory • whereas declarative memory remains largely normal, and compensates These disorders may include (at least): 1) Specific Language Impairment (SLI) 2) Dyslexia 3) Autism 4) Tourette syndrome 5) ADHD This hypothesis may explain a wide range of behavioral and brain patterns in these disorders, and their comorbidities with one another (Ullman & Gopnik, 1999; Ullman 2004, 2008; Ullman & Pierpont, 2005; Walenski, Tager-Flusberg & Ullman, 2006; 21 Walenski, Mostofsky & Ullman, 2007; Ullman, Pullman, Lovelett, Pierpont & Turkeltaub, under rev; Ullman, Pullman, Tagarelli & Curchack, in prep; Ullman & Pullman, 2015)

Specific Language Impairment (SLI): What is it? Developmental language disorder in the absence of hearing impairments, environmental deprivation, emotional problems, deficient IQ, and gross neurological damage. Grammatical aspects of language, in particular syntax, morphology, and phonology, are especially affected. 22

SLI Explanatory Accounts Previous Explanatory Hypotheses: • • SLI is due to grammar-specific deficits SLI is due to a processing deficit (e. g. , of working memory) Procedural Deficit Hypothesis (PDH): 1) SLI can be largely explained by abnormalities of brain structures underlying procedural memory, in particular of frontal/basal ganglia, especially Broca’s region and the caudate nucleus. 2) These abnormalities lead to impairments of procedural memory, including grammar, and will also likely result in problems with other functions dependent on these structures, including working memory 3) These grammatical and other deficits will be at least partly compensated for by declarative memory 23 (Ullman 2004; Ullman & Pierpont, 2005; Ullman, Pullman, Lovelett, Pierpont & Turkeltaub, under rev. ; Ullman & Pullman, 2015)

SLI SLI: Overview of Behavioral and Neuroanatomical Evidence Relatively Normal Neuroanatomy Learning/Memory Language Other Cognition Abnormal Medial Temporal Basal Ganglia (Caudate) Declarative Memory Procedural Memory Lexical Memory Grammar Often Impaired: Motor Skills Working Memory Rapid Temporal Processing (Ullman 2004; Ullman and Pierpont, 2005; Ullman, Pullman, Lovelett, Pierpont & Turkeltaub, under rev) 24

SLI Behavioral: Procedural Learning A Meta-Analysis • • 8 SRT studies: 186 individuals with SLI, 203 typical controls Findings: Significant SRT learning deficits in SLI 25 (Lum, Morgan, Conti-Ramsden, and Ullman, 2014)

SLI Frontal and Caudate Abnormalities Neuroanatomical Meta-analysis of SLI: • New meta-analysis technique: Co-localization Likelihood Estimation • 25 studies: 270 individuals with SLI, 265 typically-developing controls 26 (Ullman, Pullman, Lovelett, Pierpont and Turkeltaub, under rev)

SLI Frontal and Caudate Abnormalities 27 (Ullman, Pullman, Lovelett, Pierpont & Turkeltaub, under rev. )

SLI Declarative Memory Remains Relatively Normal Learning new information • Normal for non-verbal material • Relatively normal for verbal material, especially once language and working memory impairments are controlled for Long-term knowledge • Lexical knowledge relatively spared (any lexical impairments appear primarily due to working memory, phonological, and recall deficits) (for a review, see Ullman and Pullman, 2015. Also: see Ullman and Pierpont, 2005; Lum, Conti-Ramsden, Page, and Ullman, 2012; Riccio et al. , 2007; Dewey and Wall, 1997) 28

SLI Declarative Memory Compensation: Behavioral Evidence 1) Individuals with SLI memorize complex forms as chunks (“walked”, “the cat”), rather than implicitly combining their parts (“the” + “cat”) 2) Individuals with SLI learn explicit grammar rules 3) Grammatical abilities correlate with procedural memory in typicallydeveloping children, but with declarative memory in children with SLI (for reviews, see Ullman and Pierpont, 2005; Ullman and Pullman, 2015; Also see: Oetting and Horohov, 1997; Ullman and Gopnik, 1999; Lum, Conti-Ramsden, Page, and Ullman, 29 2012)

SLI Declarative Memory Compensation: Behavioral Evidence: Correlational Typically developing children: Correlations Declarative Memory Vocabulary Grammar Procedural Memory r =. 480** r =. 233 r =. 235 r =. 305* Children with SLI: Correlations Declarative Memory Procedural Memory Vocabulary r =. 394* r = -. 008 Grammar r =. 305* r =. 112 30 (Lum, Conti-Ramsden, Page, and Ullman, 2012)

SLI Declarative Memory Compensation: Electrophysiological Evidence: Event-Related Potentials Children Lexical/semantic anomalies Syntactic anomalies N 400 LAN N 400 Typically Developing SLI (Fonteneau & Van der Lely, 2008)

Overall: Summary Converging behavioral, neurological, neuroimaging and electrophysiological evidence suggests: • Language depends on both memory systems • Lexical and grammatical knowledge depend differentially on the two systems The learning and processing of Idiosyncratic (lexical) knowledge Rule-governed (grammatical) knowledge Declarative Memory Yes, to a fair extent, as a function of subject, item, and other factors. . . Subject: good declarative memory (eg, ♀, L 2) bad procedural memory (eg, SLI, L 2) Item: (eg, higher frequency) Procedural Memory ? Yes The two memory systems play at least partially redundant roles 32

Some Implications And Future Directions Second Language: Enhancing L 2 learning/retention may be achieved via interventions (e. g. , behavioral, pharmacological) that enhance learning/retention in the memory systems Specific Language Impairment (and other neurodevelopmental disorders): Translational implications - diagnostic: • Possible early diagnosis via neural markers of caudate dysfunction (but with caution) • Better diagnosis via tests that do not allow for declarative memory compensation Translational implications - therapeutic: • Predicted therapeutic improvements with interventions (behavioral, pharmacological) that enhance learning/retention in the memory systems 33

34