Natural Language Inference Bill Mac. Cartney NLP Group Stanford University 8 May 2009
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Natural language inference (NLI) • Aka recognizing textual entailment (RTE) • Does premise P justify an inference to hypothesis H? • An informal, intuitive notion of inference: not strict logic • Emphasis on variability of linguistic expression P H Several airlines polled saw costs grow more than expected, even after adjusting for inflation. Some of the companies in the poll reported cost increases. yes • Necessary to goal of natural language understanding (NLU) • Many more immediate applications … 2
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Applications of NLI semantic search [King et al. 07] question answering [Harabagiu & Hickl 06] Q: How much did Georgia’s gas price increase? A: In 2006, Gazprom doubled Georgia’s gas bill. A: Georgia’s main imports are natural gas, machinery, . . . A: Tbilisi is the capital and largest city of Georgia’s gas bill doubled Search summarization [Tatar et al. 08] A: Natural gas is a gas consisting primarily of methane. MT evaluation[Pado et al. 09] input: Gazprom va doubler le prix du gaz pour la Géorgie. …double Georgia’s gas bill… …two-fold increase in gas price… Economist. com …price of gas will be doubled… machine translation X X output: Gazprom will double the price of gas for Georgia. evaluation: does output paraphrase target? target: Gazprom will double Georgia’s gas Bill. 3
4 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion NLI problem sets • RTE (Recognizing Textual Entailment) • 4 years, each with dev & test sets, each 800 NLI problems • Longish premises taken from (e. g. ) newswire; short hypotheses • Balanced gather in Argentina ahead of this weekends regional talks, Hugo Chávez, Venezuela’s populist As leaders 2 -way classification: entailment vs. non-entailment P president is using an energy windfall to win friends and promote his vision of 21 st-century socialism. H Hugo Chávez acts as Venezuela’s president. yes • Fra. Ca. S test suite • 346 NLI problems, constructed by semanticists in mid-90 s • 55% have single premise; remainder have 2 or more premises • 3 -way classification: entailment, contradiction, compatibility Smith wrote a report in two hours. P H Smith spend more than two hours writing the report. no
5 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion NLI: a spectrum of approaches Solution? natural logic (this work) robust, but shallow lexical/ semantic overlap deep, but brittle Jijkoun & de Rijke 2005 Romano et al. 2006 patterned relation extraction semantic graph matching Mac. Cartney et al. 2006 Hickl et al. 2006 Problem: imprecise easily confounded by negation, quantifiers, conditionals, factive & implicative verbs, etc. FOL & theorem proving Bos & Markert 2006 Problem: hard to translate NL to FOL idioms, anaphora, ellipsis, intensionality, tense, aspect, vagueness, modals, indexicals, reciprocals, propositional attitudes, scope ambiguities, anaphoric adjectives, nonintersective adjectives, temporal & causal relations, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, comparatives, phrasal
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Shallow approaches to NLI • Example: the bag-of-words approach [Glickman et al. 2005] • Measures approximate lexical similarity of H to (part of) P P H Several airlines polled saw costs grow No 0. 9 0. 6 0. 4 0. 9 0. 8 more than expected, even after 0. 9 adjusting for inflation. Some of the companies in the poll reported cost increases. None • Robust, and surprisingly effective for many NLI problems • But imprecise, and hence easily confounded • Ignores predicate-argument structure — this can be remedied • Struggles with antonymy, negation, verb-frame alternation • Non-upward-monotone constructions are rife! [Danescu et al. 2009] • not, all, most, depends ontallest, without, doubt, avoid, regardless, unable, … Crucially, few, rarely, if, assumption of upward monotonicity 6
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The formal approach to NLI Relies on full semantic interpretation of P & H • Translate to formal representation & apply automated reasoner • Can succeed in restricted domains, but not in open-domain P Several airlines polled saw costs grow more than expected, NLI! even after adjusting for inflation. (exists p (and (poll-event p) (several x (and (airline x) (obj p x) (exists c (and (cost c) (has x c) (exists g (and (grow-event g) (subj g c) (greater-than (magnitude g) . . . ? • Need background axioms to complete proofs — but from where? • Besides, NLI task based on informal definition of inferability • Bos & Markert 06 found FOL proof for just 4% of RTE problems 7
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Solution? Natural logic! ( natural deduction) • Characterizes valid patterns of inference via surface forms • precise, yet sidesteps difficulties of translating to FOL • A long history • traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, … • modern natural logic begins with Lakoff (1970) • van Benthem & Sánchez Valencia (1986 -91): monotonicity calculus • Nairn et al. (2006): an account of implicatives & factives • We introduce a new theory of natural logic… • extends monotonicity calculus to account for negation & exclusion • incorporates elements of Nairn et al. ’s model of implicatives 8
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Outline • Introduction • Alignment for NLI • A theory of entailment relations • A theory of compositional entailment • The Nat. Log system • Conclusions [Not covered today: the bag-of-words model, the Stanford RTE system] 9
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Alignment for NLI • Most approaches to NLI depends on a facility for alignment P H Gazprom today confirmed a two-fold increase in its gas price for Georgia, beginning next Monday. Gazprom will double Georgia’s gas bill. yes • Linking corresponding words & phrases in two sentences • Alignment problem is familiar in machine translation (MT) 10
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Alignment example H (hypothesis) P (premise) unaligned content: “deletions” from P approximate match: price ~ bill phrase alignment: two-fold increase ~ double 11
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Approaches to NLI alignment • Alignment addressed variously by current NLI systems • In some approaches to NLI, alignments are implicit: • NLI via lexical overlap [Glickman et al. 05, Jijkoun & de Rijke 05] • NLI as proof search [Tatu & Moldovan 07, Bar-Haim et al. 07] • Other NLI systems make alignment step explicit: • Align first, then determine inferential validity [Marsi & Kramer 05, Mac. Cartney et al. 06] • What about using an MT aligner? • Alignment is familiar in MT, with extensive literature [Brown et al. 93, Vogel et al. 96, Och & Ney 03, Marcu & Wong 02, De. Nero et al. 06, Birch et al. 06, De. Nero & Klein 08] • Can tools & techniques of MT alignment transfer to NLI? • Dissertation argues: not very well 12
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The MANLI aligner A model of alignment for NLI consisting of four components: 1. Phrase-based representation 2. Feature-based scoring function 3. Decoding using simulated annealing 4. Perceptron learning 13
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Phrase-based alignment representation Represent alignments by sequence of phrase edits: EQ, SUB, DEL, INS EQ(Gazprom 1, Gazprom 1) INS(will 2) DEL(today 2) DEL(confirmed 3) DEL(a 4) SUB(two-fold 5 increase 6, double 3) DEL(in 7) DEL(its 8) … • One-to-one at phrase level (but many-to-many at token level) • Avoids arbitrary alignment choices; can use phrase-based resources 14
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion A feature-based scoring function • Score edits as linear combination of features, then sum: • Edit type features: EQ, SUB, DEL, INS • Phrase features: phrase sizes, non-constituents • Lexical similarity feature: max over similarity scores • • Word. Net: synonymy, hyponymy, antonymy, Jiang-Conrath Distributional similarity à la Dekang Lin Various measures of string/lemma similarity Contextual features: distortion, matching neighbors 15
16 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Decoding using simulated annealing 1. Start … 2. Generate successors 3. Score 4. Smooth/sharpen = P(A) 5. Sample 6. Lower temp T = 0. 9 T 7. Repeat … 100 times 1/T
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Perceptron learning of feature weights We use a variant of averaged perceptron [Collins 2002] Initialize weight vector w = 0, learning rate R 0 = 1 For training epoch i = 1 to 50: For each problem Pj, Hj with gold alignment Ej: Set Êj = ALIGN(Pj, Hj, w) Set w = w + Ri ( (Ej) – (Êj)) Set w = w / ‖w‖ 2 (L 2 normalization) Set w[i] = w (store weight vector for this epoch) Set Ri = 0. 8 Ri– 1 (reduce learning rate) Throw away weight vectors from first 20% of epochs Return average weight vector Training runs require about 20 hours (on 800 RTE problems) 17
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The MSR RTE 2 alignment data • Previously, little supervised data • Now, MSR gold alignments for RTE 2 • • • Token-based, but many-to-many • • [Brockett 2007] dev & test sets, 800 problems each allows implicit alignment of phrases 3 independent annotators • • • 3 of 3 agreed on 70% of proposed links 2 of 3 agreed on 99. 7% of proposed links merged using majority rule 18
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Evaluation on MSR data • We evaluate several alignment models on MSR data • Baseline: a simple bag-of-words aligner • • Two well-known MT aligners: GIZA++ & Cross-EM • • • Supplemented with lexicon; tried various symmetrization heuristics A representative NLI aligner: the Stanford RTE aligner • • Matches each token in H to most string-similar token in P Can’t do phrase alignments, but can exploit syntactic features The MANLI aligner just presented How well do they recover gold-standard alignments? • Assess per-link precision, recall, and F 1; and exact match rate 19
20 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Aligner evaluation results System Bag-of-words GIZA++ Cross-EM Stanford RTE MANLI • • P% RTE 2 dev R % F 1 % 57. 8 83. 0 67. 6 81. 1 83. 4 81. 2 66. 4 80. 1 75. 8 85. 5 67. 5 72. 1 78. 4 84. 4 E% 3. 5 9. 4 1. 3 0. 5 21. 7 P% RTE 2 test R % F 1 % E% 62. 1 85. 1 70. 3 82. 7 85. 4 82. 6 69. 1 81. 0 75. 8 85. 3 11. 3 0. 8 0. 3 21. 3 70. 9 74. 8 74. 1 79. 1 85. 3 Bag-of-words aligner: good recall, but poor precision MT aligners fail to learn word-word correspondences Stanford RTE aligner struggles with function words MANLI outperforms all others on every measure • F 1: 10. 5% higher than GIZA++, 6. 2% higher than Stanford • Good balance of precision & recall; matched >20% exactly
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion MANLI results: discussion • Three factors contribute to success: 1. Lexical resources: jail ~ prison, prevent ~ stop , injured ~ wounded 2. Contextual features enable matching function words 3. Phrases: death penalty ~ capital punishment, abdicate ~ give up 1. But phrases help less than expected! • If we set max phrase size = 1, we lose just 0. 2% in F 1 2. Recall errors: room to improve • 40%: need better lexical resources: conservation ~ protecting, organization ~ agencies, bone fragility ~ osteoporosis 3. Precision errors harder to reduce • equal function words (49%), forms of be (21%), punctuation (7%) 21
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Alignment for NLI: conclusions • MT aligners not directly applicable to NLI • • • MANLI succeeds by: • • They rely on unsupervised learning from massive amounts of bitext They assume semantic equivalence of P & H Exploiting (manually & automatically constructed) lexical resources Accommodating frequent unaligned phrases Using contextual features to align function words Phrase-based representation shows potential • But not yet proven: need better phrase-based lexical resources 22
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Outline • Introduction • Alignment for NLI • A theory of entailment relations • A theory of compositional entailment • The Nat. Log system • Conclusion 23
24 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Entailment relations in past work X is a couch RTE 1, 2, 3 3 -way Fra. Ca. S, PARC, RTE 4 containment Sánchez-Valencia X is a fish X is a hippo X is a man X is a sofa 2 -way X is a crow X is a bird X is a carp X is hungry X is a woman Yes No entailment non-entailment Yes Unknown No entailment compatibility contradiction P=Q P<Q P>Q P#Q equivalence forward entailment reverse entailment non-entailment
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion 16 elementary set relations Assign sets x, y to one of 16 relations, depending on emptiness or nonemptiness of each of four partitions y y x ? ? empty non-empty 25
26 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion 16 elementary set relations But 9 of 16 are degenerate: either x or y is either empty or universal. I. e. , they correspond to semantically vacuous expressions, which are rare outside logic textbooks. We therefore focus on the remaining seven relations. x^y x y x⊏y x ‿y x⊐y x|y x#y
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The set of basic entailment relations symbo l name example x y equivalence couch sofa x⊏y forward entailment crow ⊏ bird x⊐y diagram reverse entailment European ⊐ French negation human ^ nonhuman alternation cat | dog cover animal ‿nonhuman independence hungry # hippo x^y x|y x‿ y x#y (strict) (exhaustive exclusion) (non-exhaustive exclusion) (exhaustive non-exclusion) Relations are defined for all semantic types: tiny ⊏ small, hover ⊏ fly, kick ⊏ strike, this morning ⊏ today, in Beijing ⊏ in China, everyone ⊏ someone, all ⊏ most ⊏ some 27
28 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Joining entailment relations x R ? y S z fish | human ^ nonhuman ⊏ ⋈ ⊏ ? ⊏ ⋈ ⊏ ⊐ ⋈ ⊐ ⊐ ^ ⋈ ^ R ⋈ R ⋈ R R
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Some joins yield unions of relations! What is | ⋈ | ? x | y y | z couch | table | sofa couch sofa pistol | knife | gun pistol dog | cat rose | orchid woman | frog x ? z ⊏ gun dog ⊐ terrier cat | terrier orchid | daisy frog | Eskimo rose | daisy woman # Eskimo | ⋈ | { , ⊏, ⊐, |, #} 29
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The complete join table Of 49 join pairs, 32 yield relations in ; 17 yield unions Larger unions convey less information — limits power of inference In practice, any union which contains # can be approximated by # — so, in practice, we can avoid the complexity of unions 30
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Outline • Introduction • Alignment for NLI • A theory of entailment relations • A theory of compositional entailment • The Nat. Log system • Conclusion 31
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Lexical entailment relations ) will depend on: ( x, e(x) 1. the lexical entailment relation generated by e: (e) atomic edit: DEL, INS, SUB 2. other properties of the context x in which e is applied compound expression entailment relation Example: suppose x is red car If e is SUB(car, convertible), then (e) is ⊐ If e is DEL(red), then (e) is ⊏ Crucially, (e) depends solely on lexical items in e, independent of context x But how are lexical entailment relations determined? 32
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Lexical entailment relations: SUBs (SUB(x, y)) = (x, y) For open-class terms, use lexical resource (e. g. Word. Net) for synonyms: sofa couch, forbid prohibit ⊏ for hypo-/hypernyms: crow ⊏ bird, frigid ⊏ cold, soar ⊏ rise | for antonyms and coordinate terms: hot | cold, cat | dog or | for proper nouns: USA United States, JFK | FDR # for most other pairs: hungry # hippo Closed-class terms may require special handling Quantifiers: all ⊏ some, some ^ no, no | all, at least 4 ‿at most 6 See dissertation for discussion of pronouns, prepositions, … 33
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Lexical entailment relations: DEL & INS Generic (default) case: (DEL( • )) = ⊏, (INS( • )) = ⊐ • Examples: red car ⊏ car, sing ⊐ sing off-key • Even quite long phrases: car parked outside since last week ⊏ car • Applies to intersective modifiers, conjuncts, independent clauses, … • This heuristic underlies most approaches to RTE! • Does P subsume H? Deletions OK; insertions penalized. Special cases • Negation: didn’t sleep ^ did sleep • Implicatives & factives (e. g. refuse to, admit that): discussed later • Non-intersective adjectives: former spy | spy, alleged spy # spy • Auxiliaries etc. : is sleeping sleeps, did sleep slept 34
35 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The impact of semantic composition How are entailment relations affected by semantic How is (x, y) projected by f? composition? ? @ f @ x y f @ means fn application The monotonicity calculus provides a partial answer If f has monotonicity… ⊏ ⊐ # UP ⊏ ⊐ # DOWN ⊏ ⊐ # ⊐ ⊏ # But how are other relations (|, ^, ‿ projected? ) ⊏ ⊐ # NON # # #
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion A typology of projectivity Projectivity signatures: a generalization of monotonicity classes Each projectivity signature is a map↦ In principle, 77 possible signatures, but few actually realized negatio n ⊏ ⊐ ⊐ ⊏ ^ ^ | ‿ ‿ | # # not happy didn’t kiss not ill not human not French not more than 4 isn’t swimming ⊐ ⊏ ^ ‿ | # not glad didn’t touch not seasick not nonhuman not German not less than 6 isn’t hungry 36
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion A typology of projectivity Projectivity signatures: a generalization of monotonicity classes ↦ Each projectivity signature is a map In principle, 77 possible signatures, but few actually intersective negatio modification realized n ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ ^ | ‿ # ⊏ ⊐ | | # # live human | live nonhuman French wine | Spanish wine metallic pipe # nonferrous pipe See dissertation for projectivity of connectives, quantifiers, verbs 37
38 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projecting through multiple levels Propagate entailment relation between atoms upward, according to projectivity class of each node on path to root nobody can enter with a shirt ⊏ nobody can enter with clothes ⊏ ⊐ @ @ @ nobody can without a shirt enter @ ⊐ @ ⊏ @ nobody can without clothes enter
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Implicatives & factives 39 [Nairn et al. 06] 9 signatures, per implications (+, –, or o) in positive and negative contexts signatur e implicatives example he was forced to sell o/– he was permitted to live –/+ he forgot to pay –/o he refused to fight o/+ factives he managed to escape +/o implicatives +/– he hesitated to ask +/+ he admitted that he knew –/– he pretended he was sick o/o he wanted to fly
40 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Implicatives & factives We can specify relation generated by DEL or INS of each signature signatur e implicatives +/– (DEL) example he managed to escape he escaped (INS) +/o ⊏ ⊐ o/– implicatives he was forced to sell ⊏ he sold he was permitted to live ⊐ he lived ⊐ ⊏ –/+ he forgot to pay ^ he paid ^ ^ | | –/o o/+ nonfactives he refused to fight | he hesitated to ask ‿ he asked ‿ ‿ o/o he wanted to fly # he flew # # he fought Room for variation w. r. t. infinitives, complementizers, passivation, etc. Some more intuitive when negated: he didn’t hesitate to ask | he didn’t ask Doesn’t cover factives, which involve presuppositions — see dissertation
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Putting it all together 1. Find a sequence of edits e 1, …, en which transforms p into h. Define x 0 = p, xn = h, and xi = ei(xi– 1) for i [1, n]. 2. For each atomic edit ei: 1. Determine the lexical entailment relation (ei). 2. Project (ei) upward through the semantic composition tree of expression xi– 1 to find the atomic entailment relation (xi– 1, xi) 3. Join atomic entailment relations across the sequence of edits: (p, h) = (x 0, xn) = (x 0, x 1) ⋈ … ⋈ (xi– 1, xi) ⋈ … ⋈ (xn– 1, xn) Limitations: need to find appropriate edit sequence connecting p and h; tendency of ⋈ operation toward less-informative entailment relations; lack of general mechanism for combining multiple premises Less deductive power than FOL. Can’t handle e. g. de Morgan’s 41
42 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion An example P H i The doctor didn’t hesitate to recommend Prozac. The doctor recommended medication. yes ei xi lex atom join The doctor didn’t hesitate to recommend Prozac. 1 DEL(hesitate to) ‿ | | ^ ^ ⊏ ⊏ yes The doctor didn’t recommend Prozac. 2 DEL(didn’t) The doctor recommended Prozac. 3 SUB(Prozac, medication) The doctor recommended medication.
43 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Different edit orders? i ei lex atom join i ei 1 DEL(hesitate to) 2 ‿ | | 1 DEL(hesitate to) ‿ | | DEL(didn’t) ^ ^ ⊏ 2 SUB(Prozac, medication) ⊏ ⊐ | 3 SUB(Prozac, medication) ⊏ ⊏ ⊏ 3 DEL(didn’t) ^ ^ ⊏ i ei 1 DEL(didn’t) ^ ^ ^ 2 DEL(hesitate to) ‿ ‿ ⊏ 2 SUB(Prozac, medication) ⊏ ⊐ | 3 SUB(Prozac, medication) ⊏ ⊏ ⊏ 3 DEL(hesitate to) ‿ ‿ ⊏ i ei 1 SUB(Prozac, medication) ⊏ ⊏ ⊏ 2 DEL(hesitate to) ‿ | | 2 DEL(didn’t) ^ ^ | 3 DEL(didn’t) ^ ^ ⊏ 3 DEL(hesitate to) ‿ ‿ ⊏ lex atom join lex atom join Intermediate steps may vary; final result is typically (though not necessarily) the sam
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Outline • Introduction • Alignment for NLI • A theory of entailment relations • A theory of compositional entailment • The Nat. Log system • Conclusion 44
45 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The Nat. Log system NLI problem 1 next slide linguistic analysis from outside sources 2 alignment 3 lexical entailment classification 4 entailment projection 5 entailment joining prediction core of system covered shortly straightforward not covered further
46 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Stage 1: Linguistic analysis • Tokenize & parse input sentences (future: & NER & coref & …) • Identify items w/ special projectivity & determine scope S category: –/o implicatives • Problem: PTB-style parse tree semantic structure! VP refuse examples: refuse, forbid, prohibit, … S VP scope: S complement pattern: __ > (/VB. */ > without. S=arg) VP $. Jimmy Dean projectivity: { : , ⊏: ⊐, ⊐: ⊏, ^: |, |: #, _: #, #: #} VP move PP NP NP NNP VBD TO VB IN JJ NNS Jimmy Dean refused to move without blue jeans + + + – – – + blue jeans + • Solution: specify scope in PTB trees using Tregex [Levy & Andrew 06]
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Stage 3: Lexical entailment classification • Goal: predict entailment relation for each edit, based solely on lexical features, independent of context • Approach: use lexical resources & machine learning • Feature representation: • Word. Net features: synonymy ( ), hyponymy (⊏/⊐), antonymy (|) • Other relatedness features: Jiang-Conrath (WN-based), Nom. Bank • Fallback: string similarity (based on Levenshtein edit distance) • Also lexical category, quantifier category, implication signature • Decision tree classifier 47
48 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The Fra. Ca. S test suite • Fra. Ca. S: a project in computational semantics [Cooper et al. 96] • 346 “textbook” examples of NLI problems • 3 possible answers: yes, no, unknown (not balanced!) • 55% single-premise, 45% multi-premise (excluded) P H At most ten commissioners spend time at home. At most ten commissioners spend a lot of time at home. yes P H Dumbo is a large animal. Dumbo is a small animal. no P H Smith believed that ITEL had won the contract in 1992. ITEL won the contract in 1992. unk
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Results on Fra. Ca. S System most common class Mac. Cartney & Manning 07 this work 183 prec % 55. 7 183 68. 9 60. 8 59. 6 183 89. 3 65. 7 70. 5 # rec % acc % 100. 0 55. 7 27% error reduction 49
50 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Results on Fra. Ca. S System most common class Mac. Cartney & Manning 07 this work § 183 prec % 55. 7 183 68. 9 60. 8 59. 6 183 89. 3 65. 7 70. 5 # Category # 1 Quantifiers 2 Plurals 3 Anaphora 4 Ellipsis 5 Adjectives 6 Comparatives 7 Temporal 8 Verbs 9 Attitudes 1, 2, 5, 6, 9 44 24 6 25 15 16 36 8 9 108 prec % 95. 2 90. 0 100. 0 71. 4 88. 9 85. 7 80. 0 100. 0 90. 4 rec % acc % 100. 0 55. 7 rec % acc % 100. 0 64. 3 60. 0 5. 3 83. 3 88. 9 70. 6 66. 7 83. 3 85. 5 97. 7 75. 0 50. 0 24. 0 80. 0 81. 3 58. 3 62. 5 88. 9 87. 0 27% error reduction in largest category, all but one correct high accuracy in sections most amenable to natural logic high precision even outside areas of expertise
51 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The RTE 3 test suite • Somewhat more “natural”, but not ideal for Nat. Log • Many kinds of inference not addressed by Nat. Log: paraphrase, temporal reasoning, relation extraction, … • Big edit distance propagation of errors from atomic model P As leaders gather in Argentina ahead of this weekends regional talks, Hugo Chávez, Venezuela’s populist president is using an energy windfall to win friends and promote his vision of 21 st-century socialism. H Hugo Chávez acts as Venezuela’s president. P Democrat members of the Ways and Means Committee, where tax bills are written and advanced, do not have strong small business voting records. H Democrat members had strong small business voting records. yes no
52 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Results on RTE 3: Nat. Log System Data % Yes Prec % Rec % Acc % Stanford RTE dev 50. 2 68. 7 67. 0 67. 2 test 50. 0 61. 8 60. 2 60. 5 dev 22. 5 73. 9 32. 4 59. 2 test 26. 4 70. 1 36. 1 59. 4 Nat. Log (each data set contains 800 problems) • Accuracy is unimpressive, but precision is relatively high • Strategy: hybridize with Stanford RTE system • As in Bos & Markert 2006 • But Nat. Log makes positive prediction far more often (~25% vs. 4%)
53 Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Results on RTE 3: hybrid system System Data % Yes Prec % Rec % Acc % Stanford RTE dev 50. 2 68. 7 67. 0 67. 2 test 50. 0 61. 8 60. 2 60. 5 dev 22. 5 73. 9 32. 4 59. 2 test 26. 4 70. 1 36. 1 59. 4 dev 56. 0 69. 2 75. 2 70. 0 test 54. 5 64. 4 68. 5 64. 5 Nat. Log Hybrid (each data set contains 800 problems) 4% gain (significant, p < 0. 05)
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Outline • Introduction • Alignment for NLI • A theory of entailment relations • A theory of compositional entailment • The Nat. Log system • Conclusion 54
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion What natural logic can’t do • Not a universal solution for NLI • Many types of inference not amenable to natural logic Paraphrase: Eve was let go Eve lost her job Verb/frame alternation: he drained the oil ⊏ the oil drained Relation extraction: Aho, a trader at UBS… ⊏ Aho works for UBS Common-sense reasoning: the sink overflowed ⊏ the floor got wet • etc. • • • Also, has a weaker proof theory than FOL • Can’t explain, e. g. , de Morgan’s laws for quantifiers: Not all birds fly Some birds don’t fly 55
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion What natural logic can do • Enables precise reasoning about semantic containment … • • hypernymy & hyponymy in nouns, verbs, adjectives, adverbs containment between temporal & locative expressions quantifier containment adding & dropping of intersective modifiers, adjuncts • … and semantic exclusion … • antonyms & coordinate terms: mutually exclusive nouns, adjectives • mutually exclusive temporal & locative expressions • negation, negative & restrictive quantifiers, verbs, adverbs, nouns • … and implicatives and nonfactives • Sidesteps myriad difficulties of full semantic interpretation 56
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Contributions of this dissertation • Undertook first systematic study of alignment for NLI • Examined the relation between alignment in NLI and MT • Evaluated bag-of-words, MT, and NLI aligners for NLI alignment • Proposed a new model of alignment for NLI: MANLI • Extended natural logic to incorporate semantic exclusion • Defined expressive set of entailment relations (& join algebra) • Introduced projectivity signatures: a generalization of monotonicity • Unified account of implicativity under same framework • Implemented a robust system for natural logic inference 57
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion The future of NLI • No silver bullet for NLI — problems are too diverse • A full solution will need to combine disparate reasoners • • • simple lexical similarity (e. g. , bag-of-words) relation extraction natural logic & related forms of “semantic” reasoning temporal, spatial, & simple mathematical reasoning commonsense reasoning • Key question: how can they best be combined? • Apply in parallel, then combine predictions? How? • Fine-grained “interleaving”? Collaborative proof search? 58
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Thanks! My heartfelt appreciation to… • My committee: Profs. Genesereth, Jurafsky, Manning, Peters, and van Benthem • My collaborators: Marie-Catherine de Marneffe, Michel Galley, Teg Grenager, and many others • My advisor: Prof. Chris Manning • My girlfriend: Destiny Man Li Zhao : -) Thanks! Questions? 59
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Backup slides follow 60
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion NLI alignment vs. MT alignment Doubtful — NLI alignment differs in several respects: 1. Monolingual: can exploit resources like Word. Net 2. Asymmetric: P often longer & has content unrelated to H 3. Cannot assume semantic equivalence • NLI aligner must accommodate frequent unaligned content 4. Little training data available • • MT aligners use unsupervised training on huge amounts of bitext NLI aligners must rely on supervised training & much less data 61
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projectivity of connectives negation (not) conjunction (and) / intersective modification ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ | | # # 62
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projectivity of connectives negation (not) conjunction (and) / intersective modification disjunction (or) ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ | | # # ⊏ ⊐ ‿ # waltzed or sang ⊏ danced or sang human or equine ‿ nonhuman or equine red or yellow # blue or yellow 63
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projectivity of connectives negation (not) conjunction (and) / intersective modification disjunction (or) conditional (if) (antecedent) ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ | | # # ⊏ ⊐ ‿ # ⊐ ⊏ # # If he drinks tequila, he feels nauseous ⊐ If he drinks liquor, he feels nauseous If it’s sunny, we surf # # If it’s not sunny, we surf If it’s rainy, we surf 64
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projectivity of connectives negation (not) conjunction (and) / intersective modification disjunction (or) conditional (if) (antecedent) conditional (if) (consequent) ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ | | # # ⊏ ⊐ ‿ # ⊐ ⊏ # # ⊏ ⊐ | | # # If he drinks tequila, he feels nauseous ⊏ If he drinks tequila, he feels sick If it’s sunny, we surf | | If it’s sunny, we don’t surf If it’s sunny, we ski 65
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projectivity of connectives negation (not) conjunction (and) / intersective modification disjunction (or) conditional (if) (antecedent) conditional (if) (consequent) biconditional (if and only if) ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ | | # # ⊏ ⊐ ‿ # ⊐ ⊏ # # ⊏ ⊐ | | # # ^ # # # 66
Introduction • Alignment for NLI • Entailment relations • Compositional entailment • The Nat. Log system • Conclusion Projectivity of quantifiers 67
Скачать презентацию Natural Language Inference Bill Mac Cartney NLP Group
6865cb71e4b98397ecfa452459279c9c.ppt