Statistical Methods Traditional grammars may be brittle Statistical

Statistical Methods Traditional grammars may be “brittle” Statistical methods are built on formal theories Vary in complexity from simple trigrams to conditional random fields Can be used for language identification, text classification, information retrieval, and information extraction

N-Grams Text is comprised of characters (or words or phonemes) An N-gram is a sequence of n consecutive characters (or words. . . ) Technically, it is a Markov chain of order n-1 unigram, bigram, trigram P(ci | c 1: i-1) = P(ci|ci-n+1: ci-1) Calculate N-grams by looking at large corpus

Example – Language Identification Use P(ci|ci-2: ci-1, l), where l ranges over languages About 100, 000 characters of each language are needed l* = argmaxl P(l|c 1: N) = argmaxl P(l) P(ci|ci-2: ci-1, l) Learn the model from a corpus P(l), the probability of a given language can be estimated Other examples: spelling correction, genre classification, and named-entity recognition

Smoothing Problem: What if a particular n-gram does not appear in the training corpus? Probability would be 0 – should be a small, but positive number Smoothing – adjusting the probability of lowfrequency counts Laplace: use 1/(n+2) instead of 0 (n observations) Backoff model: back off to n-1 grams

Model Evaluation Use cross-validation (split corpus into training and evaluation sets) Need a metric for evaluation Can use perplexity to describe the probability of a sequence Perplexity(c 1: N) = P(c 1: n)-1/N Can be thought of as the reciprocal of probability normalized by the sequence length

N-gram Word Models Can be used for text classification Example: spam vs. ham Problem: out-of-vocabulary word Trick: During training, use <UNK> first time word is seen, then after that use word regularly. Then when an unknown word is seen, treat it as <UNK> Calculate probabilities from a corpus, then randomly generate phrases

Example – Spam Detection Text classification problem Train for P(Message|spam) and P(Message|ham) using n-grams Calculate P(message|spam) P(spam) and P(message|ham) P(ham) and take whichever is greater

Spam Detection – Other Methods Represent message as a set of feature/value pairs Apply a classification algorithm for the feature vector Strongly depends on the features chosen Data compression algorithms such a LZW look for commonly re-occurring sequences and replace later copies with pointers to earlier ones. Append new message to list of spam messages and compress, do the same for ham, and whichever compresses smaller. . .

Information Retrieval Think WWW and search engines Characterized by Corpus of documents Queries in some query language Result set Presentation of result sort (some ordering) Methods: Simple Boolean keyword models, IR scoring functions, Page. Rank algorithm, HITS algorithm

IR Scoring Function - BM 25 Okapi Project (Robertson, et. al. ) Three factors: Frequency word appears in the document (TF) The inverse document frequency (IDF) – inverse of times word appears in all documents Length of document |dj| is the length of the document, L is the average document length, k and b are tuned parameters

BM 25 cont'd.

Precision and Recall Precision measures the proportion of the documents in the result set that are actually relevant, e. g. , if the result set contains 30 relevant documents and 10 non-relevant documents, precision is. 75 Recall is the proportion of relevant documents that are in the result set, e. g. , if 30 relevant documents are in the result set out of a possible 50, recall is. 60

IR Refinement Pivoted document length normalization Longer documents tend to be favored Instead of document length, use a different normalization function that can be tuned Use word stems Use synonyms Look at metadata

Page. Rank Algorithm (Google) Count the links that point to the page Weight links from “high-quality sites” higher Minimizes the effect of creating lots of pages that point to the chosen page where PR(p) is the Page. Rank of p, N is the total number of pages in the corpus, xi is a page that link to p, and C(xi) is the count of the total number of out-links on the page xi

Information Extraction Ability to answer questions Possibilities range from simple template matching to full-blown language understanding systems May be domain specific or general Used as DB front-end, or WWW searching Examples: Ask. MSR, IBM's Watson, Wolfram Alpha, Siri

Template Matching Simple template matching (Weizenbaum's Eliza) Regular Expression matching – finite state automata Relational extraction methods – FASTUS: Processing done in stages: Tokenization, Complex -word handling, Basic-group handling, Complexphrase handling, Structure merging Each stages uses a FSA

Stochastic Methods for NLP Probabilistic Context-Free Parsers Probabilistic Lexicalized Context-Free Parsers Hidden Markov Models – Viterbi Algorithm Statistical Decision-Tree Models

Markov Chain Discrete random process: The system is in various states and we move from state to state. The probability of moving to a particular next state (a transition) depends solely on the current state and not previous states (the Markov property). May be modeled by a finite state machine with probabilities on the edges.

Hidden Markov Model Each state (or transition) may produce an output. The outputs are visible to the viewer, but the underlying Markov model is not. The problem is often to infer the path through the model given a sequence of outputs. The probabilities associated with the transitions are known a priori. There may be more than one start state. The probability of each start state may also be known.

Uses of HMM Parts of speech (POS) tagging Speech recognition Handwriting recognition Machine Translation Cryptanalysis Many other non-NLP applications

Viterbi Algorithm Used to find the mostly likely sequence of states (the Viterbi path) in a HMM that leads to a given sequence of observed events. Runs in time proportional to (number of observations) * (number of states)2. Can be modified if the state depends on the last n states (instead of just the last state). Take time (number of observations) * (number of states)n

Viterbi Algorithm - Assumptions The system at any given time is in one particular state. There a finite number of states. Transitions have an associated incremental metric. Events are cumulative over a path, i. e. , additive in some sense.

Viterbi Algorithm - Code See the http: //en. wikipedia. org/wiki/Viterbi_algorithm.

Example - Using HMMs to parse seminar announcements Look for different features: Speaker, date, etc. Could use one big HMM for all features or separate HMMs for each feature Advantages: resistant to noise, can be trained from data, easily updated Can be used to generate output as well as parse

Example: HMM for speaker recog.

Conditional Random Fields HMM models the full joint probability of observations and hidden states – too much work Instead, model the conditional probability of the hidden attributes given the observations Given a text e 1: N, find the hidden state sequence X 1: N that maximizes P(X 1: N|e 1: N) Conditional Random Field (CRF) does this Linear Chain CRF: variables in temporal sequence

Automated Template Construction Start with examples of output, e. g. , author-title pairs Match over large corpus, noting order, and prefix, suffix, and intermediate text Generate templates from the matches Sensitive to noise

Types of Grammars - Chomsky Recursively Enumerable: unrestricted rules Context-Sensitive: right-hand side must contain at least as many symbols as the left-hand side Context-Free: The left-hand side contains a single symbol Regular Expression: left-hand side is a single nonterminal, right-hand side is a terminal symbol optionally followed by a non-terminal symbol

Probabilistic CFG 1. sent <- np, vp. p(sent) = p(r 1) * p(np) * p(vp). 2. np <- noun. p(np) = p(r 2) * p(noun). . . 9. noun <- dog. p(noun) = p(dog). The probabilities are taken from a particular corpus of text.

Probabilistic Lexicalized CFG 1. sent <- np(noun), vp(verb). p(sent) = p(r 1) * p(np) * p(verb|noun). 2. np <- noun. p(np) = p(r 2) * p(noun). . . 9. noun <- dog. p(noun) = p(dog). Note that we've introduced the probability of a particular verb given a particular noun.