Self-Assemblying Hypernetworks for Cognitive Learning of Linguistic Memory

Self-Assemblying Hypernetworks for Cognitive Learning of Linguistic Memory Int. Conf. on Cognitive Science, CESSE-2008, Feb. 6 -8, 2008, Sheraton Hotel, Cairo, Egypt Byoung-Tak Zhang and Chan-Hoon Park Biointelligence Laboratory School of Computer Science and Engineering Cognitive Science, Brain Science, and Bioinformatics Programs Seoul National University Seoul 151 -744, Korea btzhang@bi. snu. ac. kr http: //bi. snu. ac. kr/

Talk Outline A Language Game l Learning the Linguistic Memory l ¨ The Hypernetwork Model of Language Sentence Recall Experiments l Extension to Multimodal Memory Game (Language + Vision) l Conclusion l

The Language Game Platform 3 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac.

A Language Game We ? ? a lot ? gifts. We don't have a

Text Corpus: TV Drama Series Friends, 24, House, Grey Anatomy, Gilmore Girls, Sex and the City I don't know what happened. What ? ? ? here. ? have ? visit the ? room. Take a look at this. … … 289, 468 Sentences (Training Data) 700 Sentences with Blanks (Test Data) © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 5

Step 1: Learning a Linguistic Memory k=2 k=3 k=4 … Hypernetwork Memory 6 ©

Step 2: Recalling from the Memory He is my best He is a strong boy Strong friend likes pretty Storage friend girl X 1 X 2 X 3 my He is is a Strong friend likes X 8 best a strong boy likes strong pretty girl X 4 X 6 friend X 7 X 5 Recall He is a friend Strong a ? friend Self-assembly He is a strong friend strong best friend © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 7

1 x 1 =1 x 2 =0 x 3 =0 x 4 =1 x 5 =0 x 6 =0 x 7 =0 x 8 =0 x 9 =0 x 10 =1 x 11 =0 x 12 =1 x 13 =0 x 14 =0 x 15 =0 y =1 2 x 1 =0 x 2 =1 x 3 =1 x 4 =0 x 5 =0 x 6 =0 x 7 =0 x 8 =0 x 9 =1 x 10 =0 x 11 =0 x 12 =0 x 13 =0 x 14 =1 x 15 =0 y =0 3 x 1 =0 x 2 =0 x 3 =1 x 4 =0 x 5 =0 x 6 =1 x 7 =0 x 8 =1 x 9 =0 x 10 =0 x 11 =0 x 12 =0 x 13 =1 x 14 =0 x 15 =0 y =1 4 x 1 =0 x 2 =0 x 3 =0 x 4 =0 x 5 =0 x 6 =0 x 7 =0 x 8 =1 x 9 =0 x 10 =0 x 11 =1 x 12 =0 x 13 =0 x 14 =0 x 15 =1 y =1 4 sentences (with labels) x 1 x 2 x 10 y=1 x 4 x 12 x 10 x 12 Round 1 3 2 y=1 x 4 x 15 y=1 x 3 x 14 x 2 x 3 x 14 y=0 x 9 x 14 y=0 x 6 x 8 y=1 x 3 x 6 x 13 y=1 x 6 4 y=0 x 3 3 x 9 x 3 2 x 3 x 8 x 13 y=1 x 8 x 11 x 15 y=0 x 13 x 12 x 5 x 6 x 11 x 7 x 10 x 8 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ x 9 8

The Hypernetwork Memory x 1 x 2 [Zhang, DNA 12 -2006] x 15 x

Molecular Self-Assembly of Hypernetworks xi xj y Molecular Encoding Hypernetwork Representation X 1 X 2 X 8 X 3 X 7 X 4 X 6 X 5 DNA Computing © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 10

Experimental Setup l The order (k) of an hyperedge ¨ Range: 2~4 ¨ Fixed order for each experiment l The method of creating hyperedges from training data ¨ Sliding window method ¨ Sequential sampling from the first word l The number of blanks (question marks) in test data ¨ Range: 1~4 ¨ Maximum: k - 1 11 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Learning Behavior Analysis (1/3) Sentences with One Missing Words Completion 800 700 600 500 400 300 200 Order 2 Order 3 Order 4 100 0 40 K 80 K 120 K 160 K 200 K 240 K 280 K 290 K The performance monotonically increases as the learning corpus grows. l The low-order memory performs best for the one-missing-word problem. l 12 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Learning Behavior Analysis (2/3) Sentences with Two Missing Words Completion 800 700 600 500 400 300 200 Order 2 Order 3 Order 4 100 0 40 K l 80 K 120 K 160 K 200 K 240 K 280 K 290 K The medium-order (k=3) memory performs best for the two-missing-words problem. 13 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Learning Behavior Analysis (3/3) Sentences with Three Missing Words Completion 800 700 600 500 400 300 200 Order 2 Order 3 Order 4 100 0 40 K l 80 K 120 K 160 K 200 K 240 K 280 K 290 K The high-order (k=4) memory performs best for the three-missing-words problem. 14 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

The Language Game: Results ? gonna ? upstairs ? ? a shower I'm gonna go upstairs and take a shower We ? ? a lot ? gifts We don't have a lot of gifts ? have ? visit the ? room I have to visit the ladies' room ? ? don't need your ? If I don't need your help ? still ? believe ? did this I still can't believe you did this ? ? a dream about ? In ? I had a dream about you in Copenhagen ? ? ? decision to make a decision What ? ? ? here What are you doing here ? appreciate it if ? call her by ? ? I appreciate it if you call her by the way ? you ? first ? of medical school Are you go first day of medical school Would you ? to meet ? ? Tuesday ? Would you nice to meet you in Tuesday and I'm standing ? the ? ? ? cafeteria I'm standing in the one of the cafeteria Why ? you ? come ? down ? Why are you go come on down here ? think ? I ? met ? somewhere before I think but I am met him somewhere before 15 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Extension to Multimodal Memory Game

Image. Generation Game Text Generation Game Scene 1 He is Scene 2 She is Scene 3 Map friend a 1 likes b 2 best He strong He pretty a 3 friend is boy is girl a 4 Scene 1 a 1 friend Scene 2 a 1 b 1 Scene 1 a 4 a 3 Scene 3 b 1 c 1 Scene 2 friend can it How a 1 Scene 3 be c 1 b 3 samples done? b 2 best a 1 best Text: is sequential N best But, I'm getting. Image: N tomorrow married random samples He Map a 2 a 3 a 4 b 2 a 2 b 3 a 3 b 4 a 4 c 2 b 2 a 3 c 2 b 1 c 3 b 3 a 4 c 3 c 4 b 4 c 4 Well, maybe I am. . . Hamming Distance I keep thinking about you. Generating an Image But, I'm getting married made a mistake giving up so fast. And I'm wondering if we tomorrow Well, thinking about is b 1 b 3 b 5 a 4 is Area 1 maybe I am. . . me? She you a 3 I keep thinking about tonight. But if youa 1 calla 3 you. are, me best friend is And I'm wondering if we a 2 a 1 b 2 a 3 a 4 made a mistake giving up so fast. a 3 b 3 Scene 1 a 1 strong a 2 b 2 Are you thinking about me? are, a 1=2 But if you a 3=2 call me tonight. voting Map b 2=1 a 4=2 a 1 b 2 a 3 b 1=1 b 3=1 He Image Hint He is best friend is a 1 a 3 Image Memorizing Images to Retrieve Texts a 4 b 1 best Sound is best Machine Learner Hint Texta 4 friend b 2 b 3 Text a 3 a 1 b 1 a 4 Memorizing Texts to Retrieve Images © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ a 4

Image-to-Text Crossmodal Recall Image Text Learning by Viewing Question: - Where am I giving birth - You guys really don't know anything - So when you guys get in there - I know it's been really hard for you -… Where ? I giving ? Answer: User Where am I giving birth © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ Text Corpus 18

Text-to-Image Crossmodal Recall Text Image Learning by Viewing - Where am I giving birth - You guys really don't know anything - So when you guys get in there - I know it's been really hard for you -… Question: You've been there Answer: Image Corpus © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ User 19

The Multimedia (Movie) Corpus l Dataset: 2 dramas ¨ Images and the corresponding scripts ¨ Titles < Friends, Prison Break ¨ Training data: 2, 808 images and scripts ¨ Image size: 80 x 60 = 4800 pixels ¨ Vocabulary: 2, 579 words Where am I giving birth I know it's been really hard for you So when you guys get in there 20 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Experimental Setup l The order (k) of memory units ¨ Text: k = 2, 3, 4 ¨ Image: k = 10, …, 340 l Constructing hyperedges from training data ¨ Text: Sequential sampling from a random position ¨ Image: Random sampling from 4, 800 pixel positions l The number of repetitive samples from an image-text pair ¨ N = 150, …, 300 21 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Image-to-Text Recall Examples Query Matching & Completion Answer I don't know what happened I don't know what happened There's a a kitty in … in my guitar case There's a kitty in my guitar case Maybe there's something I … I get pregnant Maybe there's something I can do to make sure I get pregnant © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Text-to-Image Recall Examples Matching & Completion Query I don't know what happened Take a look at this There's a kitty in my guitar case Maybe there's something I can do to make sure I get pregnant © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ Answer

Image to Text (Recall Rate) Recall Rate 1 0. 9 0. 8 0. 7 Rate 0. 6 0. 5 0. 4 0. 3 0. 2 Perfect Recall 0. 1 Tolerant Recall 0 10 40 70 100 130 160 190 220 250 Image Order generated sentence is 280 310 340 Note: In the tolerant recall, the evaluated correct if the number of mismatches is within the specified tolerance level (here two words). 24 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Text to Image (Recall Rate) 1 Recall Rate 0. 9 0. 8 0. 7 Rate 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Text Order Note: The retrieved image is evaluated correct if its hamming distance to the target image is the smallest. © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 25

Conclusion l l l Hypernetworks are a random graph model employing higher-order edges and allowing for a more natural representation for learning higher-order interactions. We introduce a linguistic memory model based on a self-organizing hypernetwork inspired by mental chemistry. The hypernetwork stores the sentences in random fragments and recalls a sentence by self-assemblying them given a partial, query sentence. Applied to a sentence corpus of 290 K sentences, we obtain a recall performance of 90 -100%, depending on the difficulty of the task. Cognitive plausibility: ¨ “Multiple representations of partially overlapping micromodules which are partially active simultaneously” [Fuster, 2003] ¨ Neural microcircuits [Grillner et al, 2006] ¨ Cognitive schema or cognitive code [Tse et al. , 2007] 26 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Acknowledgements Data Acquisition and Experimentation Text: Ha-Young Jang Image: Min-Oh Heo Sun Kim Joo-Kyoung Kim Ho-Sik Seok Kwonil Kim Sang-Yoon Lee Supported by - National Research Lab Program of Min. of Sci. & Tech. (2002 -2007) - Next Generation Tech. Program of Min. of Ind. & Comm. (2000 -2010) - BK 21 -IT Program of Min. of Education (2006 -2009) - SK Telecom (2007 -2008) More Information at - http: //bi. snu. ac. kr/ Research MMG (to be open soon)

The Hypernetwork Model of Learning [Zhang, 2006] 29 © 2008, SNU Biointelligence Lab, http:

Deriving the Learning Rule 30 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac.

Derivation of the Learning Rule 31 © 2008, SNU Biointelligence Lab, http: //bi. snu.

Molecular Self-Assembly

Encoding a Hypernetwork with DNA a) z 1 : (x 1=0, x 2=1, x 3=0, y=1) z 2 : (x 1=0, x 2=0, x 3=1, x 4=0, x 5=0, y=0) z 3 : (x 2=1, x 4=1, y=1) Collection of (labeled) hyperedges z 4 : (x 2=1, x 3=0, x 4=1, y=0) b) z 1 : AAAACCAATTGGAAGGCCATGCGG z 2 : AAAACCAATTCCAAGGGGCCTTCCCCAACCATGCCC z 3 : AATTGGCCTTGGATGCGG Library of DNA molecules z 4 : AATTGGAAGGCCCCTTGGATGCCC corresponding to (a) where x 1 AATT x 2 AAGG x 3 AAAA x 4 CCAA x 5 CCTT ATGC 0 GG 1 CC y 33 © 2008, SNU Biointelligence Lab, http: //bi. snu. ac. kr/