Скачать презентацию Overview of Machine Learning for NLP Tasks part Скачать презентацию Overview of Machine Learning for NLP Tasks part

8245bff547e83baf8dd4cf0a1e8fae64.ppt

  • Количество слайдов: 46

Overview of Machine Learning for NLP Tasks: part I (based partly on slides by Overview of Machine Learning for NLP Tasks: part I (based partly on slides by Kevin Small and Scott Yih) 1

Goals of Introduction n n Frame specific natural language processing (NLP) tasks as machine Goals of Introduction n n Frame specific natural language processing (NLP) tasks as machine learning problems Provide an overview of a general machine learning system architecture n n n Describe some specific aspects of our tool suite in regards to the general architecture n n Introduce a common terminology Identify typical needs of ML system Build some intuition for using the tools Focus here is on Supervised learning 2

Overview 1. 2. 3. 4. 5. 6. Some Sample NLP Problems Solving Problems with Overview 1. 2. 3. 4. 5. 6. Some Sample NLP Problems Solving Problems with Supervised Learning Framing NLP Problems as Supervised Learning Tasks Preprocessing: cleaning up and enriching text Machine Learning System Architecture Feature Extraction using FEX 3

Context Sensitive Spelling[2] n A word level tagging task: I would like a peace Context Sensitive Spelling[2] n A word level tagging task: I would like a peace of cake for desert. I would like a piece of cake for dessert. In principal, we can use the solution to the duel problem. In principle, we can use the solution to the dual problem. 4

Part of Speech (POS) Tagging n Another word-level task: Allen Iverson is an inconsistent Part of Speech (POS) Tagging n Another word-level task: Allen Iverson is an inconsistent player. While he can shoot very well, some nights he will score only a few points. (NNP Allen) (NNP Iverson) (VBZ is) (DT an) (JJ inconsistent) (NN player) (. . ) (IN While) (PRP he) (MD can) (VB shoot) (RB very) (RB well) (, , ) (DT some) (NNS nights) (PRP he) (MD will) (VB score) (RB only) (DT a) (JJ few) (NNS points) (. . ) 5

Phrase Tagging n Named Entity Recognition – a phrase-level task: After receiving his M. Phrase Tagging n Named Entity Recognition – a phrase-level task: After receiving his M. B. A. from Harvard Business School, Richard F. America accepted a faculty position at the Mc. Donough School of Business (Georgetown University) in Washington. After receiving his [MISC M. B. A. ] from [ORG Harvard Business School], [PER Richard F. America] accepted a faculty position at the [ORG Mc. Donough School of Business] ([ORG Georgetown University]) in [LOC Washington]. 6

Some Other Tasks n n n n Text Categorization Word Sense Disambiguation Shallow Parsing Some Other Tasks n n n n Text Categorization Word Sense Disambiguation Shallow Parsing Semantic Role Labeling Preposition Identification Question Classification Spam Filtering : : 7

Supervised Learning/SNo. W 8 Supervised Learning/SNo. W 8

Learning Mapping Functions n Binary Classification n {Feature, Instance, Input} Space – space used Learning Mapping Functions n Binary Classification n {Feature, Instance, Input} Space – space used to describe each instance; often n Multi-class Classification n labels; very dependent on problem n n Ranking n Output Space – space of possible output Hypothesis Space – space of functions that can be selected by the machine learning algorithm; algorithm dependent (obviously) Regression 9

Multi-class Classification[3, 4] One Versus All (Ov. A) Constraint Classification 10 Multi-class Classification[3, 4] One Versus All (Ov. A) Constraint Classification 10

Online Learning[5] n n n SNo. W algorithms include Winnow, Perceptron Learning algorithms are Online Learning[5] n n n SNo. W algorithms include Winnow, Perceptron Learning algorithms are mistake driven Search for linear discriminant along function gradient (unconstrained optimization) Provides best hypothesis using data presented up to to the present example Learning rate determines convergence n n Too small and it will take forever Too large and it will not converge 11

Framing NLP Problems as Supervised Learning Tasks 12 Framing NLP Problems as Supervised Learning Tasks 12

Defining Learning Problems[6] n n n ML algorithms are mathematical formalisms and problems must Defining Learning Problems[6] n n n ML algorithms are mathematical formalisms and problems must be modeled accordingly Feature Space – space used to describe each instance; often Rd, {0, 1}d, Nd Output Space – space of possible output labels, e. g. n n n Set of Part-of-Speech tags Correctly spelled word (possibly from confusion set) Hypothesis Space – space of functions that can be selected by the machine learning algorithm, e. g. n n Boolean functions (e. g. decision trees) Linear separators in Rd 13

Context Sensitive Spelling Did anybody (else) want too sleep for to more hours this Context Sensitive Spelling Did anybody (else) want too sleep for to more hours this morning? n Output Space n n n Could use the entire vocabulary; Y={a, aback, . . . , zucchini} Could also use a confusion set; Y={to, too, two} Model as (single label) multi-classification Hypothesis space is provided by SNo. W Need to define the feature space 14

What are ‘feature’, ‘feature type’, anyway? n A feature type is any characteristic (relation) What are ‘feature’, ‘feature type’, anyway? n A feature type is any characteristic (relation) you can define over the input representation. n Example: feature TYPE = word bigrams Sentence: The man in the moon eats green cheese. Features: [The_man], [man_in], [in_the], [the_moon]…. n In Natural Language Text, sparseness is often a problem n n n How many times are we likely to see “the_moon”? How often will it provide useful information? How can we avoid this problem? 15

Preprocessing: cleaning up and enriching text n Assuming we start with plain text: The Preprocessing: cleaning up and enriching text n Assuming we start with plain text: The quick brown fox jumped over the lazy dog. It landed on Mr. Tibbles, the slow blue cat. n Problems: n n Often, want to work at the level of sentences, words Where are sentence boundaries – ‘Mr. ’ vs. ‘Cat. ’? Where are word boundaries -- ‘dog. ’ Vs. ‘dog’? Enriching the text: e. g. POS-tagging: (DT The) (JJ quick) (NN brown) (NN fox) (VBD jumped) (IN over) (DT the) (JJ lazy) (NN dog) (. . ) 16

Download Some Tools n http: : /l 2 r. cs. uiuc. edu/~cogcomp/ n n Download Some Tools n http: : /l 2 r. cs. uiuc. edu/~cogcomp/ n n n Software: : tools, Software: : packages Sentence segmenter Word segmenter POS-tagger FEX NB: RIGHT-CLICK on “download” link n select “save link as. . . ” 17

Preprocessing scripts n n http: //l 2 r. cs. uiuc. edu/~cogcomp/ sentence-boundary. pl. /sentence-splitter. Preprocessing scripts n n http: //l 2 r. cs. uiuc. edu/~cogcomp/ sentence-boundary. pl. /sentence-splitter. pl –d HONORIFICS –i nyttext. txt -o nytsentence. txt n word-splitter. pl. /word-splitter. pl nytsentence. txt > nytword. txt n Invoking the tagger: . /tagger –i nytword. txt –o nytpos. txt n Check output 18

Problems running. pl scripts? n Check the first line: #!/usr/bin/perl n Find perl library Problems running. pl scripts? n Check the first line: #!/usr/bin/perl n Find perl library on own machine n E. g. might need. . . #!/local/bin/perl n Check file permissions. . . > ls –l sentence-boundary. pl > chmod 744 sentence-boundary. pl 19

Minor Problems with install n Possible (system-dependent) compilation errors: n n doesn’t recognize ‘optarg’ Minor Problems with install n Possible (system-dependent) compilation errors: n n doesn’t recognize ‘optarg’ POS-tagger: change Makefile in subdirectory snow/ where indicated sentence-boundary. pl: try ‘perl sentence-boundary. pl’ Link error (POS tagger): linker can’t find –lxnet n n remove ‘-lxnet’ entry from Makefile generally, check README, makefile for hints 20

The System View 21 The System View 21

A Machine Learning System Raw Text Preprocessing Formatted Text Feature Extraction Feature Vectors Machine A Machine Learning System Raw Text Preprocessing Formatted Text Feature Extraction Feature Vectors Machine Learner Training Examples Function Parameters Labels Inference Labels Classifier(s) 22 Testing Examples

Preprocessing Text They recently recovered a small piece of a live Elvis concert recording. Preprocessing Text They recently recovered a small piece of a live Elvis concert recording. He was singing gospel songs, including “Peace in the Valley. ” n n Sentence splitting, Word Splitting, etc. Put data in a form usable for feature extraction 0 0 0 0 0 piece 0 0 0 : 0 1 peace 1 0 1 0 1 0 1 2 3 4 5 6 They recently recovered a small piece of 6 7 8 9 10 11 12 13 including QUOTE Peace in the Valley. QUOTE 23

A Machine Learning System Raw Text Preprocessing Formatted Text Feature Extraction Feature Vectors 24 A Machine Learning System Raw Text Preprocessing Formatted Text Feature Extraction Feature Vectors 24

Feature Extraction with FEX 25 Feature Extraction with FEX 25

Feature Extraction with FEX n FEX (Feature Extraction tool) generates abstract representations of text Feature Extraction with FEX n FEX (Feature Extraction tool) generates abstract representations of text input n n n Has a number of specialized modes suited to different types of problem Can generate very expressive features Works best when text enriched with other knowledge sources – i. e. , need to preprocess text S = I would like a piece of cake too! n FEX converts input text into a list of active features… 1: 1003, 1005, 1101, 1330… Where each numerical feature corresponds to a specific textual feature: 1: 1003: label[piece] word[like] BEFORE word[a] 26

Feature Extraction 0 0 0 0 0 piece 0 0 0 : 0 1 Feature Extraction 0 0 0 0 0 piece 0 0 0 : 0 1 peace 1 0 1 0 1 0 1 2 3 4 5 6 They recently recovered a small piece of 6 7 8 9 10 11 12 13 including QUOTE Peace in the Valley. QUOTE 0, 1001, 1013, 1134, 1175, 1206 1, 1021, 1055, 1085, 1182, 1252 Lexicon File n n Converts formatted text into feature vectors Lexicon file contains feature descriptions 27

Role of FEX Ø Why won't you accept the facts? Ø No one saw Role of FEX Ø Why won't you accept the facts? Ø No one saw her except the postman. Feature Extraction FEX lab[accept], w[you], w[the], w[you*], 1006: 1, 1003, 1004, w[*the] lab[except], w[her], 1003, w[her*], 1006: 2, 1002, w[the], 1005, w[*the] 28

Four Important Files Corpus A new representation of for Feature vectors the raw. SNo. Four Important Files Corpus A new representation of for Feature vectors the raw. SNo. W text data Example FEX Script Mapping of feature and feature id 1. 2. Lexicon Control FEX’s behavior Define the “types” of features 29

Corpus – General Linear Format n n n The corpus file contains the preprocessed Corpus – General Linear Format n n n The corpus file contains the preprocessed input with a single sentence per line. When generating examples, Fex never crosses line boundaries. The input can be any combination of: n n n 1 st form: words separated by white spaces 2 nd form: tag/word pairs in parentheses There is a more complicated 3 rd form, but deprecated in view of alternative, more general format (later) 30

Corpus – Context Sensitive Spelling Ø Why won't you accept the facts? (WRB Why) Corpus – Context Sensitive Spelling Ø Why won't you accept the facts? (WRB Why) (VBD wo) (NN n't) (PRP you) (VBP accept) (DT the) (NNS facts) (. ? ) Ø No one saw her except the postman. (DT No) (CD one) (VBD saw) (PRP her) (IN except) (DT the) (NN postman) (. . ) 31

Script – n n n Means of Feature Engineering Fex does not decide or Script – n n n Means of Feature Engineering Fex does not decide or find good features. Instead, Fex provides you an easy method to define the feature types and extracts the corresponding features from data. Feature Engineering is in fact very important in practical learning tasks. 32

Script – n What can be good features? n n Description of Feature Types Script – n What can be good features? n n Description of Feature Types Let’s try some combinations of words and tags. Feature types in mind n n n Words around the target word (accept, except) POS tags around the target Conjunctions of words and POS tags? Bigrams or trigrams? Include relative locations? 33

Graphical Representation Window [-2, 2] 0 -4 -3 -2 -1 Target 1 2 3 Graphical Representation Window [-2, 2] 0 -4 -3 -2 -1 Target 1 2 3 0 1 2 3 4 5 6 7 WRB VBD NN PRP VBP DT NNS . Why won 't you accept the facts ? 34

Script – Syntax n Syntax: targ [inc] [loc]: RGF [[left-offset, right-offset]] n n n Script – Syntax n Syntax: targ [inc] [loc]: RGF [[left-offset, right-offset]] n n n targ – target index n If targ is ‘-1’… n target file entries are used to identify the targets n If no target file is specified, then EVERY word is treated as a target inc – use the actual target instead of the generic place-holder (‘*’) loc – include the location of feature relative to the target RGF – define “types” of features like words, tags, conjunctions, bigrams, trigrams, …, etc left-offset and right-offset: specify the window range 35

Basic RGF’s – Sensors (1/2) n n Sensor is the fundamental method of defining Basic RGF’s – Sensors (1/2) n n Sensor is the fundamental method of defining “feature types. ” It is applied on the element, and generates active features. Type Mnemonic Interpretation Example Word w the word (spelling) w[you] Tag t part-of-speech tag t[NNP] Vowel v active if the word starts with a vowel v[eager] Length length of the word len[5] 36

Basic RGF’s – Sensors (2/2) n Sensors are also an elegant way to incorporate Basic RGF’s – Sensors (2/2) n Sensors are also an elegant way to incorporate our background knowledge. Type Mnemonic Interpretation Example City List is. City active is the phrase is the name of a city is. City[Chicago] Verb Class v. Cls return Levin’s verb class v. Cls[51. 2] More sensors can be found by looking at FEX source (Sensors. h) lab: a special RGF that generates labels Ø lab(w), lab(t), … 37

Complex RGF’s n Existential Usage n n Conjunction and Disjunction n n len(x=3), v(X) Complex RGF’s n Existential Usage n n Conjunction and Disjunction n n len(x=3), v(X) w&t; w|t Collocation and Sparse Collocation n n coloc(w, w); coloc(w, t, w); coloc(w|t, w|t) scoloc(t, t); scoloc(t, w, t); scoloc(w|t, w|t) 38

(Sparse) Collocation 0 -4 -3 -2 -1 Target 1 2 3 0 1 2 (Sparse) Collocation 0 -4 -3 -2 -1 Target 1 2 3 0 1 2 3 4 5 6 7 WRB VBD NN PRP VBP DT NNS . Why won 't you accept the facts ? -1 inc: coloc(w, t)[-2, 2] -1 inc: scoloc(w, t)[-2, 2] w[‘t]-t[PRP], w[you]-t[VBP] w[accept]-t[DT], w[the]-t[NNS] w[‘t]-t[PRP], w[‘t]-t[VBP], w[‘t]-t[DT], w[‘t]-t[NNS], w[you]-t[VBP], w[you]-t[DT], w[you]-t[NNS], w[accept]-t[DT], w[accept]-t[NNS], w[the]-t[NNS] 39

Examples – 2 Scripts n Download examples from tutorial page: ‘context sensitive spelling materials’ Examples – 2 Scripts n Download examples from tutorial page: ‘context sensitive spelling materials’ link n accept-except-simple. scr -1: lab(w) -1: w[-1, 1] n accept-except. scr -1: lab(w) -1: w|t [-2, 2] -1 loc: coloc(w|t, w|t) [-3, -3] 40

Lexicon & Example (1/3) n Corpus: … (NNS prices) (CC or) (VB accept) (JJR Lexicon & Example (1/3) n Corpus: … (NNS prices) (CC or) (VB accept) (JJR slimmer) (NNS profits) … n Script: ae-simple. scr -1 lab(w); -1: w[-1, 1] n Lexicon: 1 label[w[except]] 2 label[w[accept]] Generated by lab(w) 1001 w[or] 1002 w[slimmer] Generated by w[-1, 1] n Feature indices of lab start from 1. Example: 2, 1001, 1002; Feature indices of regular features start from 1001. 41

Lexicon & Example (2/3) n Target file: fex -t ae. targ … accept except Lexicon & Example (2/3) n Target file: fex -t ae. targ … accept except n We treat only these two words as targets. Lexicon file n n n If the file does not exist, fex will create it. If the file already exists, fex will first read it, and then append the new entries to this file. This is important because we don’t want two different feature indices representing the same feature. 42

Lexicon & Example (3/3) n Example file n n If the file does not Lexicon & Example (3/3) n Example file n n If the file does not exist, fex will create it. If the file already exists, fex will append new examples to it. Only active features and their corresponding lexicon items are generated. If the read-only lexicon option is set, only those features from the lexicon that are present (active) in the current instance are listed. 43

Now practice – change script, run FEX, look at the resulting lexicon/examples >. /fex Now practice – change script, run FEX, look at the resulting lexicon/examples >. /fex –t ae. targ ae-simple. scr ae-simple. lex short-ae. pos short-ae. ex 44

Citations 1) 2) 3) 4) 5) F. Sebastiani. Machine Learning in Automated Text Categorization. Citations 1) 2) 3) 4) 5) F. Sebastiani. Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34(1): 1 -47, 2002. A. R. Golding and D. Roth. A Winnow-Based Approach to Spelling Correction. Machine Learning, 34: 107 -130, 1999. E. Allewin, R. Schapire, and Y. Singer. Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers. Journal of Machine Learning Research, 1: 113 -141, 2000. S. Har-Peled, D. Roth, and D. Zimak. Constraint Classification: A New Approach to Multiclass Classification. In Proc. 13 th Annual Intl. Conf. of Algorithmic Learning Theory, pp. 365 -379, 2002. A. Blum. On-Line Algorithms in Machine Learning. 1996. 45

Citations 6) 7) 8) 9) T. Mitchell. Machine Learning, Mc. Graw Hill, 1997. A. Citations 6) 7) 8) 9) T. Mitchell. Machine Learning, Mc. Graw Hill, 1997. A. Blum. Learning Boolean Functions in an Infinite Attribute Space. Machine Learning, 9(4): 373 -386, 1992. J. Kivinen and M. Warmuth. The Perceptron Algorithm vs. Winnow: Linear vs. Logarithmic Mistake Bounds when few Input Variables are Relevant. UCSC-CRL-95 -44, 1995. T. Dietterich. Approximate Statistical Tests for Comparing Supervised Classfication Learning Algorithms. Neural Computation, 10(7): 18951923, 1998 46