Скачать презентацию Text Categorization actually methods apply for categorizing anything Скачать презентацию Text Categorization actually methods apply for categorizing anything

d5cba5e2caf07fbebe8df0eb8a3bcbd6.ppt

  • Количество слайдов: 70

Text Categorization (actually, methods apply for categorizing anything into fixed categories – tagging, WSD, Text Categorization (actually, methods apply for categorizing anything into fixed categories – tagging, WSD, PP attachment. . . ) 600. 465 - Intro to NLP - J. Eisner 1

Why Text Categorization? § Is it spam? § Is it Spanish? § Is it Why Text Categorization? § Is it spam? § Is it Spanish? § Is it interesting to this user? § News filtering § Helpdesk routing § Is it interesting to this NLP program? § e. g. , should my calendar system try to interpret this email as an appointment (using info. extraction)? § Where should it go in the directory? § Yahoo! / Open Directory / digital libraries § Which mail folder? (work, friends, junk, urgent. . . ) 600. 465 - Intro to NLP - J. Eisner 2

Measuring Performance § Classification accuracy: What % of messages were classified correctly? § Is Measuring Performance § Classification accuracy: What % of messages were classified correctly? § Is this what we care about? System 1 Overall accuracy 95% Accuracy on spam 99. 99% Accuracy on gen 90% System 2 95% 90% 99. 99% § Which system do you prefer? 600. 465 - Intro to NLP - J. Eisner 3

Measuring Performance § Precision = good messages kept all messages kept § Recall = Measuring Performance § Precision = good messages kept all messages kept § Recall = good messages kept all good messages Trade off precision vs. recall by setting threshold Measure the curve on annotated dev data (or test data) Choose a threshold where user is comfortable 600. 465 - Intro to NLP - J. Eisner 4

F-measure = 1 / (average(1/precision, 1/recall)) Measuring Performance OK for search engines (maybe) high F-measure = 1 / (average(1/precision, 1/recall)) Measuring Performance OK for search engines (maybe) high threshold: all we keep is good, but we don’t keep much point where precision=recall (sometimes reported) 600. 465 - Intro to NLP - J. Eisner low threshold: keep all the good stuff, but a lot of the bad too would prefer to be here! OK for spam filtering and legal search 5

More Complicated Cases of Measuring Performance § For multi-way classifiers: § Average accuracy (or More Complicated Cases of Measuring Performance § For multi-way classifiers: § Average accuracy (or precision or recall) of 2 -way distinctions: Sports or not, News or not, etc. § Better, estimate the cost of different kinds of errors § e. g. , how bad is each of the following? § putting Sports articles in the News section § putting Fashion articles in the News section § putting News articles in the Fashion section § § Now tune system to minimize total cost Which articles are most Sports-like? For ranking systems: Which articles / webpages most relevant? § Correlate with human rankings? § Get active feedback from user? § Measure user’s wasted time by tracking clicks? 600. 465 - Intro to NLP - J. Eisner 6

How to Categorize? Subject: would you like to. . drive a new vehicle for How to Categorize? Subject: would you like to. . drive a new vehicle for free ? ? ? this is not hype or a hoax , there are hundreds of people driving brand new cars , suvs , minivans , trucks , or rvs. it does not matter to us what type of vehicle you choose. if you qualify for our program , it is your choice of vehicle , color , and options. we don ' t care. just by driving the vehicle , you are promoting our program. if you would like to find out more about this exciting opportunity to drive a brand new vehicle for free , please go to this site : http : / / 209. 134. 131 / ntr to watch a short 4 minute audio / video presentation which gives you more information about our exciting new car program. if you do n't want to see the short video , but want us to send you our information package that explains our exciting opportunity for you to drive a new vehicle for free , please go here : http : / / 209. 134. 131 / ntr / form. htm we would like to add you the group of happy people driving a new vehicle for free. happy motoring. 600. 465 - Intro to NLP - J. Eisner 7

How to Categorize? (supervised) We’ve seen lots of options in this course! 1. Build How to Categorize? (supervised) We’ve seen lots of options in this course! 1. Build n-gram model of each category § § Question: How to classify test message? Answer: Bayes’ Theorem 600. 465 - Intro to NLP - J. Eisner 8

How to Categorize? (supervised) We’ve seen lots of options in this course! 2. Represent How to Categorize? (supervised) We’ve seen lots of options in this course! 2. Represent each document as a vector (must choose representation and distance measure; use SVD? ) § § Question: How to classify test message? Answer 1: Category whose centroid is most similar (may not work well if category is diverse) § Answer 2: Cluster each category into subcategories (then use answer 1 to pick a subcategory) (return the category that the subcategory is in) (this can also be useful for n-gram models) § Answer 3: Just look at labels of nearby training docs (e. g. , let the k nearest neighbors vote – flexible!) (maybe the closer ones get a bigger vote) 600. 465 - Intro to NLP - J. Eisner 9

How to Categorize? (supervised) We’ve seen lots of options in this course! 3. Treat How to Categorize? (supervised) We’ve seen lots of options in this course! 3. Treat it like word-sense disambiguation a) Vector model – use all the features (we just saw this) b) Decision list – use single most indicative feature c) Naive Bayes – use all the features, weighted by how well they discriminate among the categories d) Decision tree – use some of the features in sequence e) Other options from machine learning, like perceptron, Support Vector Machine (SVM), logistic regression, … Features matter more than which machine learning method 600. 465 - Intro to NLP - J. Eisner 10

Review: Vector Model These two documents are similar: After normalizing vector length to 1, Review: Vector Model These two documents are similar: After normalizing vector length to 1, Close in Euclidean space (similar endpoint) High dot product (similar direction) ed k ar cus don t n bo duct ove dv ba a ar a a ab ab (0, 0, 3, 1, 0, 7, . . . e urgy ot m g zy zy 1, 0) (0, 0, 1, 0, 3, . . . 1) Can play lots of encoding games when creating vector: Remove function words or reduce their weight Use features other than unigrams 600. 465 - Intro to NLP - J. Eisner 11

slide courtesy of D. Yarowsky (modified) Review: Decision Lists To disambiguate a token of slide courtesy of D. Yarowsky (modified) Review: Decision Lists To disambiguate a token of lead : § Scan down the sorted list § The first cue that is found gets to make the decision all by itself § Not as subtle as combining cues, but works well for WSD Cue’s score is its log-likelihood ratio: log [ p(cue | sense A) [smoothed] / p(cue | sense B) ] 600. 465 - Intro to NLP - J. Eisner 12

slide courtesy of D. Yarowsky (modified) Review: Combining Cues via Naive Bayes these stats slide courtesy of D. Yarowsky (modified) Review: Combining Cues via Naive Bayes these stats come from term papers of known authorship (i. e. , supervised training) 600. 465 - Intro to NLP - J. Eisner 13

slide courtesy of D. Yarowsky (modified) Review: Combining Cues via Naive Bayes 1 2 slide courtesy of D. Yarowsky (modified) Review: Combining Cues via Naive Bayes 1 2 “Naive Bayes” model for classifying text (Note the naive independence assumptions!) 600. 465 - Intro to NLP - J. Eisner Would this kind of sentence be more typical of a student A paper or a student B paper? 14

example from Manning & Schütze Decision Trees Is this Reuters article an Earnings Announcement? example from Manning & Schütze Decision Trees Is this Reuters article an Earnings Announcement? 2301/7681 = 0. 3 of all docs split on feature that reduces our uncertainty most contains “cents” 2 times contains “cents” < 2 times 1607/1704 = 0. 943 694/5977 = 0. 116 contains “versus” 2 times 1398/1403 = 0. 996 “yes” contains “versus” < 2 times 209/301 = 0. 694 600. 465 - Intro to NLP - J. Eisner contains “net” 1 time 422/541 = 0. 780 contains “net” < 1 time 272/5436 = 0. 050 “no” 15

Features Besides Unigrams § All these approaches (except n-gram model) can use “interesting” features, Features Besides Unigrams § All these approaches (except n-gram model) can use “interesting” features, not just unigrams. § There’s generally a heuristic feature selection problem § Use some very large set of features defined by a template § Maybe restrict to features that look useful in isolation? § Add features greedily, one at a time § Measure or guess expected improvement of each feature § Make sure to smooth when doing this – why? § At the end, remove features that hurt performance on held-out data § What does Spam. Assassin use? 600. 465 - Intro to NLP - J. Eisner 16

Spam. Assassin Features 100 4. 0 3. 994 3. 970 3. 910 3. 801 Spam. Assassin Features 100 4. 0 3. 994 3. 970 3. 910 3. 801 3. 472 3. 437 3. 371 3. 350 3. 284 3. 283 3. 261 3. 250 3. 200 From: address is in the user's black-list Sender is on www. habeas. com Habeas Infringer List Invalid Date: header (timezone does not exist) Written in an undesired language Listed in Razor 2, see http: //razor. sf. net/ Subject is full of 8 -bit characters Claims compliance with Senate Bill 1618 exists: X-Precedence-Ref Reverses Aging Claims you can be removed from the list 'Hidden' assets Claims to honor removal requests Contains "Stop Snoring" Received: contains a name with a faked IP-address Received via a relay in list. dsbl. org Character set indicates a foreign language 600. 465 - Intro to NLP - J. Eisner 17

Spam. Assassin Features 3. 198 3. 193 3. 180 3. 140 3. 123 3. Spam. Assassin Features 3. 198 3. 193 3. 180 3. 140 3. 123 3. 090 3. 072 3. 044 3. 009 3. 005 2. 991 2. 975 2. 968 2. 932 2. 900 2. 879 Forged eudoramail. com 'Received: ' header found Free Investment Received via SBLed relay, seehttp: //www. spamhaus. org/sbl/ Character set doesn't exist Dig up Dirt on Friends No MX records for the From: domain X-Mailer contains malformed Outlook Expressversion Stock Disclaimer Statement Apparently, NOT Multi Level Marketing Bulk email software fingerprint (jpfree) found inheaders exists: Complain-To Bulk email software fingerprint (VC_IPA) found inheaders Invalid Date: year begins with zero Mentions Spam law "H. R. 3113" Received forged, contains fake AOL relays Asks for credit card details 600. 465 - Intro to NLP - J. Eisner 18

Spam. Assassin Features 2. 858 2. 851 2. 842 2. 826 2. 800 2. Spam. Assassin Features 2. 858 2. 851 2. 842 2. 826 2. 800 2. 796 2. 795 2. 786 2. 784 2. 783 2. 782 2. 748 2. 744 2. 737 To: username at front of subject Claims you actually asked for this spam To header contains 'recipient' marker Compare Rates Received: says mail bounced all around the world Mentions Spam Law "UCE-Mail Act" Received via buggy SMTP server (MDaemon 2. 7. 4 SP 4 R) Bulk email software fingerprint (Storm. Post) foundin headers Broken CGI script message Message-Id generated by a spam tool Urges you to call now Tells you it's an ad RAND found, spammer forgot to run the random-IDgenerator Cable Converter No Age Restrictions Possible porn - Celebrity Porn 600. 465 - Intro to NLP - J. Eisner 19

Spam. Assassin Features 2. 782 2. 748 2. 744 2. 737 2. 735 2. Spam. Assassin Features 2. 782 2. 748 2. 744 2. 737 2. 735 2. 730 2. 726 2. 720 2. 702 2. 695 2. 693 2. 668 2. 660 2. 658 Tells you it's an ad RAND found, spammer forgot to run the random-IDgenerator Cable Converter No Age Restrictions Possible porn - Celebrity Porn Bulk email software fingerprint (Ji. Xing) found inheaders DNSBL: sender is Confirmed Spam Source Bulk email software fingerprint (MMailer) found inheaders exists: X-Encoding DNSBL: sender is Confirmed Open Relay SEC-mandated penny-stock warning -- thanks SEC Claims you can be removed from the list Removes Wrinkles Offers a stock alert Listed in DCC, seehttp: //rhyolite. com/anti-spam/dcc/ Common pyramid scheme phrase (1) 600. 465 - Intro to NLP - J. Eisner 20

Spam. Assassin Features 2. 654 2. 645 2. 642 2. 640 2. 639 2. Spam. Assassin Features 2. 654 2. 645 2. 642 2. 640 2. 639 2. 622 2. 620 2. 611 2. 566 2. 565 2. 541 2. 516 2. 513 2. 510 2. 502 2. 500 Offers a free consultation Bulk email software fingerprint (EVAMAIL) foundin headers Possible porn - Amateur Porn Listed in Razor 1, see http: //razor. sf. net/ Subject contains lots of white space exists: X-x Received via a relay in relays. visi. com Bulk email software fingerprint (IMktg) found inheaders Compete for your business Possible porn - Pay Site Contains "CBYI" Spam phrases score is 34 to 55 (high) Possible porn - Lesbian Site Contains 'free installation' with capitals Free Grant Money Listed in Pyzor, see http: //pyzor. sf. net/ 600. 465 - Intro to NLP - J. Eisner 21

Spam. Assassin Features 2. 500 2. 496 2. 492 2. 488 2. 456 2. Spam. Assassin Features 2. 500 2. 496 2. 492 2. 488 2. 456 2. 450 2. 445 2. 443 2. 425 2. 421 2. 398 Tre¶æ zawiera 'odes³anie z dopiskiem NIE' Tre¶æ zawiera 'Artykul 25 ust 2 punkt 2' Tresc zawiera 'przepraszamy za zajêty czas' Tresc zawiera 'Zamów teraz!!!' Tresc zawiera 'Je¿eli (Pañstwo) nie ¿yczycie(sz)sobie' Tresc zawiera 'Aby usun±æ adres e-mail. . . ' Spam tool pattern in MIME boundary 'Message-Id' was added by a relay Bulk email software fingerprint (screwup 1) found inheaders University Diplomas Character set indicates foreign language body Claims you can be removed from the list Headers include 3 consecutive 8 -bit characters Date: is 24 to 48 hours after Received: date 'From' juno. com does not match 'Received' headers Meet Singles 600. 465 - Intro to NLP - J. Eisner 22

Spam. Assassin Features 2. 362 2. 361 2. 357 2. 351 2. 334 2. Spam. Assassin Features 2. 362 2. 361 2. 357 2. 351 2. 334 2. 331 2. 314 2. 292 2. 290 2. 280 2. 276 2. 261 2. 250 2. 242 2. 240 Serious Enquiries Only. Claims auto-email removal Mi. ME-Version header (oddly capitalized) A "microsoft" header was found X-Mailer contains "Out. Look Express 3. 14159" Possible porn - Rape "Collect Child Support" Scam Claims spam helps the environment Free Leads Fake name used in SMTP HELO command Received via a relay in ipwhois. rfc-ignorant. org Possible porn - Cum Shot Amazing Stuff Received via a relay in orbs. dorkslayers. com Possible porn - Mega Porn Offers pure profit 600. 465 - Intro to NLP - J. Eisner 23

Spam. Assassin Features 2. 216 2. 210 2. 209 2. 206 2. 203 2. Spam. Assassin Features 2. 216 2. 210 2. 209 2. 206 2. 203 2. 202 2. 180 2. 176 2. 170 2. 145 2. 114 2. 109 2. 100 2. 088 2. 083 Received contains a faked HELO hostname Tells you it's an ad Uses control sequences inside a URL's hostname Claims spam helps the environment Tells you to 'take action now!' Cash Bonus From an address @btamail. net. cn exists: X-Library Contains "My wife, Jody" testimonial Possible porn - Nasty Girls Promise you. . . ! Claims to be in accordance with some Spam law Uses a numeric IP address in URL Possible porn - Live Porn Discusses search engine listings HTML comments which obfuscate text 600. 465 - Intro to NLP - J. Eisner 24

Spam. Assassin Features 2. 066 2. 060 2. 052 2. 044 2. 030 2. Spam. Assassin Features 2. 066 2. 060 2. 052 2. 044 2. 030 2. 022 2. 011 2 2 2 2 Information on getting a larger penis or breasts (2) Contains 'free preview' with capitals A foreign language charset used in headers Says "We strongly oppose the use of spam email" trail of Received: headers seems to be forged Credit Bureaus Claims compliance with House Bill 4176 No Investment Tre¶æ zawiera 'adres e-mail zostalznaleziony/pozyskany' Tre¶æ zawiera 'adres (e-mail) pochodzi zogólnodostêpnych. . ' Tre¶æ zawiera 'Ustawy o ochronie danychosobowych' Tresc zawiera 'temat USUN' Tresc zawiera 'na podstawie adresow e-mailpublicznie. . . ' Tresc zawiera 'kliknij w poni¿szy link' Tresc zawiera 'do nabycia u nas' Tresc zawiera 'Wys³aæ pusty mail' 600. 465 - Intro to NLP - J. Eisner 25

Spam. Assassin Features 2 2 2 2 2 1. 995 1. 984 1. 977 Spam. Assassin Features 2 2 2 2 2 1. 995 1. 984 1. 977 1. 952 1. 910 1. 904 Tresc zawiera 'Wiadomo¶æ nadano na podstawie. . . ' Tresc zawiera 'Wiadomo¶æ nadano jednorazowo. . . ' Tresc zawiera 'USUN Z BAZY' Tresc zawiera 'Prosimy o przes³anie pustego maila' Tresc zawiera 'Je¿eli nie interesuj±. . . ' Tresc zawiera 'Je¿eli nie chcesz (otrzymywac). . . ' Tresc zawiera '. . . prosimy o zwrotny e-mail. . . ' Tresc zawiera '. . . adres z bazy. . . ' Dice cumplir con la ley Clama cumplir con la normativa SPAM Serious cash Viagra and other drugs If only it were that easy Nigerian scam key phrase (million dollars) Drastically Reduced Contains "Temple Kiff" 600. 465 - Intro to NLP - J. Eisner 26

Spam. Assassin Features 1. 889 1. 880 1. 858 1. 856 1. 844 1. Spam. Assassin Features 1. 889 1. 880 1. 858 1. 856 1. 844 1. 842 1. 839 1. 836 1. 831 1. 824 1. 813 1. 778 1. 772 1. 754 1. 744 Forged 'by gw 05' 'Received: ' header found Credit Card Offers Find out Anything Contains "Gentle Ferocity" Spam phrases score is 21 to 34 (high) Possible Porn - Porn membership Potential Earnings Bulk email software fingerprint (Group Mail) foundin headers Once in a lifetime, apparently Offers Free (often stolen) Passwords Contains 'Dear (something)' Possible porn - Porn Password Message is 90 -100% HTML tags Sent using a trial version of Communi. Gate Date: is 48 to 96 hours after Received: date To: has no local-part before @ sign 600. 465 - Intro to NLP - J. Eisner 27

Spam. Assassin Features 1. 739 1. 721 1. 697 1. 690 1. 687 1. Spam. Assassin Features 1. 739 1. 721 1. 697 1. 690 1. 687 1. 686 1. 682 1. 681 1. 663 1. 640 1. 639 1. 631 1. 625 1. 598 1. 591 Talk about a check or money order Contains 'for only pennies a day' Spam tool pattern in MIME boundary Form for checking email address Subject: contains advertising tag Talks about bulk email Claims you registered with some kind of partner Long Distance Phone Offer Additional Income Spam phrases score is 05 to 08 (medium) Contains 'subject to credit approval' Talks about tracing by SSN Possible Porn - XXX Photos Contains 'earn (dollar) something per week' Message-Id has characters often found in spam 'X-Mailer' line contains gibberish 600. 465 - Intro to NLP - J. Eisner 28

Spam. Assassin Features 1. 591 1. 578 1. 552 1. 548 1. 546 1. Spam. Assassin Features 1. 591 1. 578 1. 552 1. 548 1. 546 1. 544 1. 539 1. 526 1. 523 1. 518 1. 506 1. 505 1. 503 1. 500 Cures Baldness Subject starts with "Hello" "Refinance your home" Doing something with my income Date: is 96 hours or more before Received: date To: address contains spaces Cents on the Dollar Uses a username in a URL Secretly Recorded Invalid Date: header (not RFC 2822) From and To are same (3) Valid-looking To "undisclosed-recipients" exists: Date-warning Temat zawiera 'oferta' Tre¶æ zawiera 'Zaprosiæ pañstwo' Tre¶æ zawiera 'Szanowni Pañstwo' 600. 465 - Intro to NLP - J. Eisner 29

Spam. Assassin Features 1. 500 1. 495 1. 490 1. 486 1. 479 1. Spam. Assassin Features 1. 500 1. 495 1. 490 1. 486 1. 479 1. 470 1. 466 1. 459 1. 435 1. 410 1. 404 1. 400 1. 394 Tresc zawiera 'publicznie dostêpny (email)' Tresc zawiera 'Upowaznienie do wystawiania faktur. VAT. . . ' Tresc zawiera '. . . mail z tematem. . . ' Possible registry spammer Possible porn - Adult Web Sites 'one time mailing' doesn't mean it isn't spam Forged hotmail. com 'Received: ' header found Talks about opting in Possible porn - Barely Legal Claims compliance with Senate Bill 1618 Direct Marketing Money back guarantee. Date: is 48 to 96 hours before Received: date Instructions on how to increase something NOS CHILLAN PARA DECIR QUE ES GRATIS Plugs Viagra 600. 465 - Intro to NLP - J. Eisner 30

Spam. Assassin Features 1. 385 1. 382 1. 373 1. 370 1. 368 1. Spam. Assassin Features 1. 385 1. 382 1. 373 1. 370 1. 368 1. 363 1. 361 1. 352 1. 337 1. 332 1. 319 1. 314 1. 306 1. 302 1. 301 1. 293 Spam phrases score is 08 to 13 (medium) URL uses words and phrases which indicate porn (4) As seen on national TV! Message text disguised using base-64 encoding Date: is 3 to 6 hours after Received: date Score with babes! From and To are same (6) 'From' yahoo. com does not match 'Received' headers Spam phrases score is 13 to 21 (high) Not intended for residents of XYZ. Faked To "Undisclosed-Recipients" From and To are same (5) Only thing addresses on CD are useful for is spam Contains "Vjestika Aphrodisia" Lower Monthly Payment HTML comment has 3 consecutive 8 -bit characters 600. 465 - Intro to NLP - J. Eisner 31

Spam. Assassin Features 1. 285 1. 283 1. 275 1. 274 1. 273 1. Spam. Assassin Features 1. 285 1. 283 1. 275 1. 274 1. 273 1. 270 1. 269 1. 253 1. 247 1. 246 1. 231 1. 226 1. 224 1. 218 1. 201 From: does not include a real name Uses a dotted-decimal IP address in URL Contains link without http: // prefix 'Subject' contains G. a. p. p. y-T. e. x. t Marketing Solutions Spam tool pattern in MIME boundary 'Prestigious Non-Accredited Universities' Spam tool pattern in MIME boundary Incorporates a tracking ID number From and To are same (2) Contains 'free sample' with capitals Claims compliance with spam regulations Online Pharmacy Received via SMTPD 32 server (SMTPD 32 -n. n) Includes a form which will send an email While you Sleep 600. 465 - Intro to NLP - J. Eisner 32

Spam. Assassin Features 1. 187 1. 175 1. 148 1. 146 1. 138 1. Spam. Assassin Features 1. 187 1. 175 1. 148 1. 146 1. 138 1. 131 1. 119 1. 118 1. 112 1. 110 1. 099 1. 098 1. 092 1. 084 1. 078 Uses non-standard port number for HTTP Possible porn - in ALL CAPS Subject contains a unique ID Bulk email software fingerprint (hash 2) found inheaders Get Paid Contains 'URGENT BUSINESS' Why Pay More? Requires Initial Investment Javascript to open a new window exists: X-List-Unsubscribe Date: is 6 to 12 hours after Received: date Subject starts with dollar amount Increase your ejaculation! Subject: contains Korean unsolicited email tag Spam phrases score is 03 to 05 (medium) Plugs "Herbal Viagra" 600. 465 - Intro to NLP - J. Eisner 33

Spam. Assassin Features 1. 187 1. 175 1. 148 1. 146 1. 138 1. Spam. Assassin Features 1. 187 1. 175 1. 148 1. 146 1. 138 1. 131 1. 119 1. 118 1. 112 1. 110 1. 099 1. 098 1. 092 1. 084 1. 078 Uses non-standard port number for HTTP Possible porn - in ALL CAPS Subject contains a unique ID Bulk email software fingerprint (hash 2) found inheaders Get Paid Contains 'URGENT BUSINESS' Why Pay More? Requires Initial Investment Javascript to open a new window exists: X-List-Unsubscribe Date: is 6 to 12 hours after Received: date Subject starts with dollar amount Increase your ejaculation! Subject: contains Korean unsolicited email tag Spam phrases score is 03 to 05 (medium) Plugs "Herbal Viagra" 600. 465 - Intro to NLP - J. Eisner 34

Spam. Assassin Features 1. 077 1. 057 1. 045 1. 042 1. 039 1. Spam. Assassin Features 1. 077 1. 057 1. 045 1. 042 1. 039 1. 038 1. 023 1. 021 1. 009 1 1 1 1 Apparently, you'll be amazed People just leave money laying around Bulk email software fingerprint (e. Groups) found inheaders Date: is 24 to 48 hours before Received: date Talks about direct email Unneeded encoding of HTML tags Javascript to move windows around No such thing as a free lunch (3) Save big money Frequent SPAM content Frequent SPAM content 600. 465 - Intro to NLP - J. Eisner 35

Spam. Assassin Features 1 1 1 1 Filename is just a '#'; probably a Spam. Assassin Features 1 1 1 1 Filename is just a '#'; probably a JS trick Old Murkowski disclaimer Obfuscated action attribute in HTML form Mentions monsterhut. com Form for verifying email address Contains signature of unregistered spam tool Publicidad por e-mail Contiene la palabra gratis en las cabeceras exists: X-Fix To: non-existent 'Investors' address Subject contains 'Your Membership Exchange' Spam tool pattern in MIME boundary Reply-To: is empty Received via a relay in bl. spamcop. net Received via RSSed relay, seehttp: //www. mail-abuse. org/rss/ Received via RBLed relay, seehttp: //www. mail-abuse. org/rbl/ 600. 465 - Intro to NLP - J. Eisner 36

Spam. Assassin Features 1 1 1 1 Received from first hop dialup, seehttp: //www. Spam. Assassin Features 1 1 1 1 Received from first hop dialup, seehttp: //www. mail-abuse. org/dul/ Received from dialup, seehttp: //www. mail-abuse. org/dul/ Received contains fake 'Post. cz' hostname From an address @email-publisher. com Bulk email software fingerprint (xmailer tag) foundin headers Bulk email software fingerprint (pascual) found inheaders Bulk email software fingerprint (e. Bizmailer) foundin headers Bulk email software fingerprint (charset) found inheaders Bulk email software fingerprint (Yam) found inheaders Bulk email software fingerprint (V 3161) found inheaders Bulk email software fingerprint (Uproar) found inheaders Bulk email software fingerprint (Seednet) found inheaders Bulk email software fingerprint (Power. Campaign)found in headers Bulk email software fingerprint (Opt-In Lightning)found in headers Bulk email software fingerprint (Matchmaker) foundin headers Bulk email software fingerprint (Mail Bomber)found in headers 600. 465 - Intro to NLP - J. Eisner 37

Spam. Assassin Features 1 1 1 1 Bulk email software fingerprint (Henry Su) found Spam. Assassin Features 1 1 1 1 Bulk email software fingerprint (Henry Su) found inheaders Bulk email software fingerprint (GRMessage. Queue)found in header Bulk email software fingerprint (EPaper) found inheaders Bulk email software fingerprint (Diffondi. Cool)found in headers Bulk email software fingerprint (Current. Mailer)found in headers Bulk email software fingerprint (Caretop) found inheaders Bulk email software fingerprint (Campaign Blaster)found in header Bulk email software fingerprint ("outlook") found inheaders 'Received: ' contains huge hostname 'From' contains more than one address Tre¶æ jest od wydawnictwa Verlag Dashofer(spamerzy) Tresc zawiera 'Za zaliczeniem pocztowym. . . ' /zam. wieni/i /zainteresowan. {0, 50}wsp. . prac/ /www. adresy. org/i /specjaln. {0, 50}ofert/i 600. 465 - Intro to NLP - J. Eisner 38

Spam. Assassin Features 1 1 1 1 Presentación de un nuevo producto. Porno gratis. Spam. Assassin Features 1 1 1 1 Presentación de un nuevo producto. Porno gratis. Para dejar de fumar Pago contra reembolso. Nos animan a contestar si estamos interesados No se puede considerar spam Mensaje enviado por error Mas informacion. Los regalos no existen, salvo de nuestros amigos. Inmigración legal (? ) a los Estados Unidos Informacion y reserva If you want to subscribe. . . If you send an email you will be Opt. Out IMPERATIVOS EN MAYUSCULAS. Haga click aqui. Ha sido ganador. 600. 465 - Intro to NLP - J. Eisner 39

Spam. Assassin Features 1 1 1 1 Ha sido ganador. El correo como alternativa Spam. Assassin Features 1 1 1 1 Ha sido ganador. El correo como alternativa comercial Conviertete en Spammer. Claims you can opt-out Claims you can be removed in Spanish Claims not to be spam in Spanish Alta en buscadores hispanos. spam software: Pop. Launch mentions Cyber Fire. Power!, a spam-tool Will not Belive your Eyes! Well known spam senders Wants you to do business online Things incredible They keep your money -- No Refund! Terms and conditions Suspect you might have received the message bymistake 600. 465 - Intro to NLP - J. Eisner 40

Spam. Assassin Features 1 1 1 1 Slashed Price SSPL found, spammer forgot to Spam. Assassin Features 1 1 1 1 Slashed Price SSPL found, spammer forgot to run the random-IDgenerator Psychics Scam Prices won't Last Possible porn - Galleries of Pictures Plugs "Natural Viagra" Outstanding Values Orders shipped by priority mail No Middleman No Medical Exams No Gimmick Nigerian scam, cfhttp: //www. snopes 2. com/inboxer/scams/nigeria. htm New Customers Only More Internet Traffic Luxury Car List removal information 600. 465 - Intro to NLP - J. Eisner 41

Spam. Assassin Features 1 Get Started Now 1 Cyber Fire. Power! rant about losing Spam. Assassin Features 1 Get Started Now 1 Cyber Fire. Power! rant about losing dropboxes 1 Confidentially on all orders 1 Claims you were on a list 1 Claims to listen to some removal request list 1 Claims not to be spam 1 Claims not to be selling anything 1 Claims compliance with spam regulations 1 Claims "This is not junk email" 1 Cell Phone Cancer Scam 1 Buying judgements 1 Achieve Wealth 0. 982 Talks about future mailings 0. 977 Excessive quoted-printable encoding in body 0. 975 Multi Level Marketing mentioned 600. 465 - Intro to NLP - J. Eisner 42

Spam. Assassin Features 0. 968 0. 959 0. 954 0. 952 0. 948 0. Spam. Assassin Features 0. 968 0. 959 0. 954 0. 952 0. 948 0. 947 0. 935 0. 931 0. 910 0. 908 0. 906 0. 904 0. 900 0. 893 0. 885 0. 882 Possible porn - Hardcore Porn Missing To: header From: has no local-part before @ sign Targeted Traffic / Email Addresses Information on getting a larger penis or breasts Message is 70 -90% HTML tags Free Membership To: and Cc: contain similar domains at least 8 times Received contains a (dollar) variable reference Claims compliance with spam regulations 'From' ebay. com does not match 'Received' headers Unlimited in caps Accept Credit Cards From: ends in numbers 'Message-Id' was added by a relay (3) Gives information about an opportunity 600. 465 - Intro to NLP - J. Eisner 43

Spam. Assassin Features 0. 874 0. 863 0. 853 0. 849 0. 838 0. Spam. Assassin Features 0. 874 0. 863 0. 853 0. 849 0. 838 0. 820 0. 817 0. 810 0. 796 0. 795 0. 781 0. 779 Don't delete me! Nooooo!!!! Fast Viagra Delivery Frequent SPAM content exists: X-Stormpost-To Missing Date: header List removal information Consolidate Debt and Credit Financial Freedom Lots and lots of Cc: headers Received via a relay in multihop. dsbl. org Contains word 'guarantee' in all-caps Claims you can be removed from the list Spam phrases score is 00 to 01 (low) HTML message is a saved web page Claims compliance with Senate Bill 1618 exists: X-PMFLAGS 600. 465 - Intro to NLP - J. Eisner 44

Spam. Assassin Features 0. 676 0. 673 0. 670 0. 666 0. 665 0. Spam. Assassin Features 0. 676 0. 673 0. 670 0. 666 0. 665 0. 658 0. 653 0. 646 0. 643 0. 630 0. 628 0. 622 0. 620 0. 614 0. 612 See for yourself You'd better read all of this spam! Easy Terms Contains "Toner Cartridge" Human Growth Hormone Trying to sell insurance online No experience needed! Claims to be legitimate email Subject: starts with advertising tag Frequent SPAM content illegal Nigerian transactions (2) Subject GUARANTEED DNSBL: sender ip address in in a dialup block Possible porn - Must be 18 Tells you to click on a URL (in caps) Free Quote 600. 465 - Intro to NLP - J. Eisner 45

Spam. Assassin Features 0. 611 0. 610 0. 608 0. 606 0. 605 0. Spam. Assassin Features 0. 611 0. 610 0. 608 0. 606 0. 605 0. 601 0. 600 0. 594 0. 573 0. 560 0. 556 0. 553 0. 552 0. 549 Refinance Home Received via a relay in relays. ordb. org Contains 'free access' with capitals Uses a long numeric IP address in URL Have you been turned down? Includes a URL link to send an email with the subject'remove' No Credit Check No Inventory To: has a malformed address Be your own boss Information on how to work at home (2) Contains mail-in order form One hundred percent guaranteed Guaranteed Stuff Information on mortgage rates Frequent SPAM content 600. 465 - Intro to NLP - J. Eisner 46

Spam. Assassin Features 0. 544 0. 542 0. 541 0. 539 0. 536 0. Spam. Assassin Features 0. 544 0. 542 0. 541 0. 539 0. 536 0. 531 0. 525 0. 521 0. 518 0. 514 0. 513 0. 511 0. 506 0. 505 From and To the same (1) Bulk email software fingerprint (screwup 2) found inheaders Gives an excuse for why message was sent Avoid Bankruptcy Includes a link for AOL users to click Form for changing email address Apply online (with capital O) List removal information Date: is 12 to 24 hours after Received: date Asks you for your signature on a form Subject talks about losing pounds Lower Interest Rates Do it Today Unsecured Credit/Debt The best Rates From: starts with nums 600. 465 - Intro to NLP - J. Eisner 47

Spam. Assassin Features 0. 505 0. 503 0. 501 0. 500 0. 496 0. Spam. Assassin Features 0. 505 0. 503 0. 501 0. 500 0. 496 0. 489 0. 488 0. 483 0. 466 0. 459 0. 448 Spam phrases score 55 or higher (high) Impotence cure Vacation Offers Spam is 100% natural? ! Possible porn - Free Porn Possible porn - Best, Largest Porn Collections Spam phrases score is 01 to 02 (low) Can not be combined with any other offer Message contains disclaimer Claims to be Legal Subject is all capitals MS-Outlook-style To "" Date: is 96 hours or more after Received: date Spam tool pattern in MIME boundary Date: is 6 to 12 hours before Received: date Says: "to be removed, reply via email" or similar 600. 465 - Intro to NLP - J. Eisner 48

Spam. Assassin Features 0. 448 0. 446 0. 443 0. 435 0. 434 0. Spam. Assassin Features 0. 448 0. 446 0. 443 0. 435 0. 434 0. 431 0. 429 0. 428 0. 426 0. 424 0. 422 0. 421 Possible porn - Porn Fest Sent with 'X-Priority' set to high Local part containing a "4 u" variant HTML font color is magenta Join Millions of Americans Asks for a billing address Nigerian scam key phrase ((dollar) NNN. Nm/USDNNN. N m/US(doll Claims "This is not spam" Sent with 'X-Msmail-Priority' set to high Subject contains "FREE" in CAPS exists: X-Mailing. ID MIME section missing boundary Asks you to fill out a form HTML font color is unknown to us Domain name containing a "4 u" variant HTML font color is yellow 600. 465 - Intro to NLP - J. Eisner 49

Spam. Assassin Features 0. 419 0. 418 0. 417 0. 416 0. 415 0. Spam. Assassin Features 0. 419 0. 418 0. 417 0. 416 0. 415 0. 414 0. 413 0. 412 0. 411 0. 410 0. 408 0. 407 Includes a link to send a mail with a subject Standard investment opportunity spam Javascript to hide URLs in browser Offers Extra Cash Eliminate Bad Credit Lose Weight Spam Subject talks about savings Subject ends with lots of white space Offers a full refund Gives instructions for removal from list Free Cell Phone Frontpage used to create the message Offers a limited time offer Claims you can be removed from the list Attempt at obfuscating the word "mortgage" Opportunity - What a deal! 600. 465 - Intro to NLP - J. Eisner 50

Spam. Assassin Features 0. 407 0. 406 0. 405 0. 404 0. 403 0. Spam. Assassin Features 0. 407 0. 406 0. 405 0. 404 0. 403 0. 402 Nobody's perfect Tells you about a strong buy HTML table has thick border Buy Direct Instant Access button HTML font color is green HTML font color is cyan Discusses money making Asks you to click below (in caps) Uses open redirection service exists: X-Server. Host Claims you can be removed from the list List removal information Message with extraneous Content-type: . . . type=header There is no obligation. Talks about lots of money 600. 465 - Intro to NLP - J. Eisner 51

Spam. Assassin Features 0. 402 0. 401 0. 400 0. 386 0. 382 0. Spam. Assassin Features 0. 402 0. 401 0. 400 0. 386 0. 382 0. 380 0. 369 0. 365 0. 364 0. 362 Contains 'Get it now' with capitals Supplies are Limited No such thing as a free lunch (2) You won't be dissapointed. Possible porn - Offers Instant Access Nigerian scam key phrase ((dollar)NN, NNN. NN) How dear can you be if you don't know my name? No Strings Attached HTML with embedded plugin object Received via a relay in relays. osirusoft. com Off Shore Scams Information on how to work at home (1) Possible porn - Hot, Nasty, Wild, Young Contains word 'amazing' in all-caps exists: X-SMTPExp-Version There is no catch. 600. 465 - Intro to NLP - J. Eisner 52

Spam. Assassin Features 0. 361 0. 360 0. 344 0. 336 0. 335 0. Spam. Assassin Features 0. 361 0. 360 0. 344 0. 336 0. 335 0. 334 0. 333 0. 330 0. 329 0. 327 0. 326 0. 325 0. 324 sent to [email protected] com or similar Received from first hop dialup listed inrelays. osirusoft. com HTML font color is same as background Subject: is empty or missing FONT Size +2 and up or 3 and up Lowest Price HTML font color has unusual name Contains word 'profits' in all-caps HTML font color is gray What are you waiting for One Time Rip Off Talks about prizes Free Website To: and Cc: contain similar usernames at least 5 times HTML font face is not a commonly used face Quoted-printable line longer than 76 characters 600. 465 - Intro to NLP - J. Eisner 53

Spam. Assassin Features 0. 324 0. 323 0. 321 0. 320 0. 319 0. Spam. Assassin Features 0. 324 0. 323 0. 321 0. 320 0. 319 0. 318 0. 317 0. 315 From: has a malformed address exists: X-SMTPExp-Registration Message-Id has no @ sign No such thing as a free lunch (1) URL of CGI script called "unsubscribe" or "remove" Satisfaction Guaranteed "if you do not wish to receive any more" Message contains a lot of ^M characters exists: x-esmtp Claims you are a winner From: contains numbers mixed in with letters Can't live without? HTML mail with non-white background Talks about email marketing Save big money HTML font color is red 600. 465 - Intro to NLP - J. Eisner 54

Spam. Assassin Features 0. 315 0. 313 0. 312 0. 308 0. 307 0. Spam. Assassin Features 0. 315 0. 313 0. 312 0. 308 0. 307 0. 306 0. 305 0. 304 0. 302 0. 301 3 WHOLE LINES OF YELLING DETECTED Save Up To Domain registration spam body Tells you to click on a URL Subject: domain registration spam subject URL contains spamhaus signature: numbered servers Name Brand Asks you to click below Act Now! Don't Hesitate! Talks about Hidden Charges Message is 50 -70% HTML tags While Supplies Last Easily-executed Java. Script code Subject starts with "Free" HTML font color not within safe 6 x 6 x 6 palette No Purchase Necessary 600. 465 - Intro to NLP - J. Eisner 55

Spam. Assassin Features 0. 301 0. 300 0. 299 0. 296 0. 294 0. Spam. Assassin Features 0. 301 0. 300 0. 299 0. 296 0. 294 0. 281 0. 279 0. 245 0. 242 0. 239 0. 224 0. 222 Auto-executing Java. Script code DNSBL: sender is a Spamware site or vendor Significant Savings No Fees Click-to-remove with PHP/ASP action found X-Mailer header indicates a non-spam MUA (The. Bat!) 'remove' URL contains an email address Being a Member Investment Decision Date: is 3 to 6 hours before Received: date Contains a Privacy Statement Tells you how to stop further spam Month Trial Offer Save (dollar) Sign up Free Today To: repeats address as real name 600. 465 - Intro to NLP - J. Eisner 56

Spam. Assassin Features 0. 218 0. 217 0. 216 0. 214 0. 212 0. Spam. Assassin Features 0. 218 0. 217 0. 216 0. 214 0. 212 0. 211 0. 210 Congratulations - you've been scammed? 2 WHOLE LINES OF YELLING DETECTED Weekend Getaway Trying to offer you something Member Stuff HTML font color is missing hash ( Doesn't ask any questions Contains 'Special Promotion' A WHOLE LINE OF YELLING DETECTED To: is empty Winning in Caps Stuff on Sale Only (dollar) Encourages you to waste no time in ordering Who really wins? HTML font face has excess capital characters 600. 465 - Intro to NLP - J. Eisner 57

Spam. Assassin Features 0. 209 0. 207 0. 206 0. 205 0. 204 0. Spam. Assassin Features 0. 209 0. 207 0. 206 0. 205 0. 204 0. 203 0. 202 0. 201 Free DVD Date: is 12 to 24 hours before Received: date Java. Script code Header with all capitals found HTML font color is blue Winner in Caps HTML font face is not a word Fantastic Deal Includes a 'remove' email address Includes a URL link to send an email Possible porn - Large Number of movies, pics Free Offer Contains a tollfree number illegal Nigerian transactions (1) Image tag with an ID code to identify you Frame wanted to load outside URL 600. 465 - Intro to NLP - J. Eisner 58

Spam. Assassin Features 0. 201 0. 181 0. 150 0. 146 0. 144 0. Spam. Assassin Features 0. 201 0. 181 0. 150 0. 146 0. 144 0. 137 0. 134 0. 127 0. 123 0. 117 0. 114 0. 111 0. 108 0. 107 0. 106 Contains 'for only' some amount of cash X-Mailer header indicates a non-spam MUA(Outlook Express) Spam tool pattern in MIME boundary Cancel at any time! Talks about social security numbers Click to perform an action on an account Gives an excuse about why you were sent this spam Nigerian scam key phrase ((dollar) NNN. Nm/USDNNN. N m/US(doll Contains a comment with nothing but unique ID No Claim Forms 'Message-Id' was added by a relay (2) Free Trial They're just giving it away! Message-Id has characters indicating spam Dear [email protected] com? Free Hosting 600. 465 - Intro to NLP - J. Eisner 59

Spam. Assassin Features 0. 105 0. 104 0. 103 0. 102 0. 101 0. Spam. Assassin Features 0. 105 0. 104 0. 103 0. 102 0. 101 0. 100 0. 038 0. 032 0. 031 0. 028 0. 014 0. 009 Contains an ASCII-formatted form I wonder how many emails they sent in error. . . URL of page called "unsubscribe" Subject has exclamation mark and question mark Offer Expires Contains 'Dear Somebody' Javascript protocol in a URI Message includes Microsoft executable program MIME filename does not match content Spam tool pattern in MIME boundary 'Received: ' has 'may be forged' warning Message-Id is not valid, according to RFC 2822 Offers Coupon Please read this! Please oh please! Shopping Spree Contains a line >=199 characters long 600. 465 - Intro to NLP - J. Eisner 60

Spam. Assassin Features 0. 009 0. 008 0. 005 0. 004 0. 003 -0. Spam. Assassin Features 0. 009 0. 008 0. 005 0. 004 0. 003 -0. 006 -0. 019 -0. 026 -0. 069 -0. 075 -0. 102 -0. 118 -0. 123 -0. 133 Spam tool pattern in MIME boundary Risk free. Suuurreeee. . Reserves the right Expect to earn Contains 'G. a. p. p. y-T. e. x. t' Gift Certificate Big Bucks X-Mailer header indicates a non-spam MUA(Outlook) From Majordomo Missing From: header Free money! Forwarded email (Outlook style) Email came from some known mailing list software Mailer daemon failure notice (1) Message text is over 40 K in size Came via Internet Mail Service plugin 600. 465 - Intro to NLP - J. Eisner 61

Spam. Assassin Features -0. 137 -0. 143 -0. 196 -0. 200 -0. 207 -0. Spam. Assassin Features -0. 137 -0. 143 -0. 196 -0. 200 -0. 207 -0. 211 -0. 215 -0. 217 -0. 231 -0. 233 -0. 240 -0. 298 -0. 300 -0. 301 -0. 302 -0. 304 Correct for MIME 'null block' X-Mailer header indicates a non-spam MUA(Netscape) Mailing list headers are suspicious exists: Resent-To exists: X-Authentication-Warning Where are you working at? exists: X-Accept-Language Subject contains newsletter header (list) 'Message-Id' was added by yahoo. com, that's OK exists: X-Loop X-Mailer header indicates a non-spam MUA (AOL) To: repeats local-part as real name User-Agent header indicates a non-spam MUA(Entourage) Short signature present (no empty lines) exists: X-Mailing-List Long signature present (empty lines) 600. 465 - Intro to NLP - J. Eisner 62

Spam. Assassin Features -0. 484 -0. 489 -0. 506 -0. 518 -0. 522 -0. Spam. Assassin Features -0. 484 -0. 489 -0. 506 -0. 518 -0. 522 -0. 558 -0. 601 -0. 605 -0. 616 -0. 641 -0. 695 -0. 708 Subject contains a month name - probable newsletter(2) Subject contains a month name - probable newsletter Common footer for Hotmail Contains a PGP-signed message Appears to be from yahoo groups Yahoo! Groups message exists: User-Agent Has a valid-looking References header Forwarded email User-Agent header indicates a non-spam MUA(Mozilla) User-Agent header indicates a non-spam MUA(Outlook Express) Subject contains newsletter header (news) Message-Id indicates a non-spam MUA (Pine) Contains what looks like an 'E-Mail Disclaimer' Contains a PGP-signed message (signature attached) Message text is over 20 K in size 600. 465 - Intro to NLP - J. Eisner 63

Spam. Assassin Features -0. 725 -0. 754 -0. 832 -0. 847 -0. 864 -0. Spam. Assassin Features -0. 725 -0. 754 -0. 832 -0. 847 -0. 864 -0. 897 -0. 949 -0. 986 -1 -1 Subject contains a frequency - probable newsletter X-Mailer header indicates a non-spam MUA(T-Offline) Contains what looks like a quoted email text exists: In-Reply-To Has an Approved-By moderated list header User-Agent header indicates a non-spam MUA(IMP) Contains what looks like a patch from diff -u Mailer daemon failure notice (2) X-Mailer header indicates a non-spam MUA (Gnus) User-Agent header indicates a non-spam MUA(Gnus) Subject contains newsletter header (in review) From: looks like US Telephone Number recommended page from Mail. Bits. com Talks about tracking numbers Common footer for MSN A Mail. Man confirm-your-address message 600. 465 - Intro to NLP - J. Eisner 64

Spam. Assassin Features -1. 118 -1. 128 -1. 152 -1. 176 -1. 301 -1. Spam. Assassin Features -1. 118 -1. 128 -1. 152 -1. 176 -1. 301 -1. 334 -1. 433 -1. 451 -1. 596 -1. 628 -1. 696 -1. 780 -1. 801 -1. 898 -2. 092 -2. 170 Common footer for MSN Contains a password retrieval system Something about registration User-Agent header indicates a non-spam MUA(Mutt) Came from MSN Communities exists: X-Cron-Env Subject looks like order info From the Mailer-Daemon Subject contains a date Contains what looks like an email attribution Common footer for Hotmail X-Mailer header indicates a non-spam MUA (Apple. Mail) Common footer for Hotmail Sent through Microsoft's List. Builder service Short signature present (empty lines) Common footer for Hotmail 600. 465 - Intro to NLP - J. Eisner 65

Spam. Assassin Features -2. 174 -2. 442 -2. 473 -2. 475 -2. 550 -2. Spam. Assassin Features -2. 174 -2. 442 -2. 473 -2. 475 -2. 550 -2. 699 -2. 863 -3. 052 -3. 127 -4. 0 -6 -10 -20 -100 Message from e. Bay Contains what looks like a patch from diff -c Looks like a Debian BTS bug Common footer for Hotmail Subject is an e. Bay question Looks like a Bugzilla bug User-Agent header indicates a non-spam MUA(KMail) non-spam Yahoo! Groups banner found Long signature present (no empty lines) Uses the Habeas warrant mark(http: //www. habeas. com/) User is listed in 'whitelist_to' Not Matt's Scripts formmail. pl Bonded sender, seehttp: //www. bondedsender. org/referred. html User is listed in 'more_spam_to' User is listed in 'all_spam_to' From: address is in the user's white-list 600. 465 - Intro to NLP - J. Eisner 66

How to Categorize? (unsupervised) What if we don’t have supervised training data? Might try How to Categorize? (unsupervised) What if we don’t have supervised training data? Might try an iterative approach as usual: 1. Cluster the messages 2. Train n-gram, Naive Bayes, or decision list model to discriminate among the clusters 3. Use the model to reassign messages to clusters (most will stay put but some will move) 4. Return to step 2 until convergence 600. 465 - Intro to NLP - J. Eisner 67

How to Categorize? (semisupervised) What if we have only a little supervised data? Could How to Categorize? (semisupervised) What if we have only a little supervised data? Could try bootstrapping like Yarowsky’s WSD: 1. Start with very small, rather accurate classes 2. Train n-gram, Naive Bayes, or decision list model to discriminate among the classes 3. Augment each class with new messages that the model confidently classifies there (maybe also move or remove some existing messages) 4. Return to step 2 until convergence 600. 465 - Intro to NLP - J. Eisner 68

How to Categorize? (adaptive) What if we gradually get more new data over time? How to Categorize? (adaptive) What if we gradually get more new data over time? § User feedback (active or passive) on our classifications § News / email systems that categorize, or judge relevance § Add new articles / messages to training data § If they’re unlabeled (no supervision), label them automatically § Add them only if we’re confident? Add them fractionally, like EM? So model adjusts over time: § E. g. , change the cluster centroids or n-gram parameters § May want to weight the more recent data more heavily, since the future is more like the present than the past § E. g. , message from k days ago has weight 0. 9 k (k=0, 1, 2, . . . ) § So today’s model = today’s data + 0. 9 * yesterday’s model 600. 465 - Intro to NLP - J. Eisner 69

How to Categorize? (hierarchical) What if we are putting document in a Yahoo! category? How to Categorize? (hierarchical) What if we are putting document in a Yahoo! category? § There are thousands of categories (at least) – too hard! § § Choose one of the 14 top-level categories, e. g. , Science Then use a Science-specific classifier to choose one of the 54 second-level categories within Science (14 are symlinks) Continue working your way down the tree. . . When you can’t classify with high confidence, ask a human (then use the human’s answer as more training data) 600. 465 - Intro to NLP - J. Eisner 70