Machine translation I MT overview Ling 571 Fei

Machine translation (I) MT overview Ling 571 Fei Xia Week 9: 11/22/05 – 11/29/05

Plan for the rest of the quarter • MT: – Part I: MT overview: 11/22 -- 11/29 – Part II: Word-based SMT: 12/1 -12/6 – Next quarter: seminar on MT • Starting with a real baseline system • Improving the system in various ways • Reading and presenting recent papers • Project presentation: 12/8

Homework and Quizzes • Hw #8: due on 11/23 (tomorrow) • Hw #9: due on 12/6 • Hw #10: presentation due on 12/8, report due on 12/13 • Quiz #3: 11/29 • Quiz #4: 12/6

Outline • • MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

MT in a nutshell

Q 1: what is the ultimate goal of translation? • Translation: source language target language (S T) • Ultimate goal: find the “perfect” translation for text in S, thus allowing people to “appreciate” the text in S without knowing S: – Accuracy: faithful to S, including meaning, connotation, style, … – Fluency: the translation is as natural as an utterance in T.

Q 2: what are the perfect translations? What do “Accuracy” and “Fluency” mean? • Ex 1: Complement / downplayer (1) A: Your daughter was phenomenal. (2) B: No. She was just so-so. • Ex 2: Greeting: how’s everything? – Old days: chi 1 le 5 ma 5? (Have {you} eaten? ) – 1980 s -- now: fa 1 le 5 ma 5? (Have (you) gotten rich? ) – 2000 s -- now: li 2 le 5 ma 5? (Have (you) gotten divorced? ) • The answer: it depends

Q 3: Can human always get the perfect translations? • Novels: Shakespeare, Cao Xueqin, … - hidden messages: c 1 c 0, c 2 c 0, c 3 c 0, c 4 c 0 c 1’ c 2’ c 3’ c 4’ • Word play, jokes, puns: – What do prisoners use to call each other? – Cell phones. • Concept gaps: double jeopardy, go Greek, fen sui, bei fen, …. • Other constraints: lyrics, dubbing, poem.

“Crazy English” by Richard Lederer • Let’s face it: English is a crazy language. There is no egg in eggplant or ham in hamburger, neither apple nor pine in pineapple. • When a house burns up, it burns down. You fill in a form by filling it out and an alarm clock goes off by going on. • When the stars are out, they are visible, but when the lights are out, they are invisible. And why, when I wind up my watch, I start it, but when I wind up this essay, I end it?

How to translate it? • “Compound” words: Let’s face it: English is a crazy language. There is no egg in eggplant or ham in hamburger, neither apple nor pine in pineapple. • Verb+particle: When a house burns up, it burns down. You fill in a form by filling it out and an alarm clock goes off by going on. • Predicate+argument: When the stars are out, they are visible, but when the lights are out, they are invisible. And why, when I wind up my watch, I start it, but when I wind up this essay, I end it?

Q 4: Can machines be as good as humans in translation quality? • We know there are things that even humans cannot translate “perfectly”. • For things that humans can translate, will machines be ever as good as humans in translation quality? – Never say “never”. – Not in the near future.

Q 5: what is MT good for? • • Rough translation: web data Computer-aided human translation Translation for limited domain Cross-lingual IR • Machine is better than human in: – Speed: much faster than humans – Memory: can easily memorize millions of word/phrase translations. – Manpower: machines are much cheaper than humans – Fast learner: it takes minutes or hours to build a new system. Erasable memory – Never complain, never get tired, …

Q 6: what’s the MT history? (Based on work by John Hutchins) • Before the computer: In the mid 1930 s, a French. Armenian Georges Artsrouni and a Russian Petr Troyanskii applied for patents for ‘translating machines’. • The pioneers (1947 -1954): the first public MT demo was given in 1954 (by IBM and Georgetown University). • The decade of optimism (1954 -1966): ALPAC (Automatic Language Processing Advisory Committee) report in 1966: "there is no immediate or predictable prospect of useful machine translation. "

A brief history of MT (cont) • The aftermath of the ALPAC report (19661980): a virtual end to MT research • The 1980 s: Interlingua, example-based MT • The 1990 s: Statistical MT • The 2000 s: Hybrid MT

Q 7: where are we now? • Huge potential/need due to the internet, globalization and international politics. • Quick development time due to SMT, the availability of parallel data and computers. • Translation is reasonable for language pairs with a large amount of resource. • Start to include more “minor” languages.

MT in a nutshell • What is the ultimate goal of translation? • What are the perfect translations? • Can human always get perfect translations? • Can machines be as good as humans? • What are MT good for? • What is the MT history? • Where are we now?

Outline • • MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

Major challenges in MT

Major challenges • Getting the right words: – Choosing the correct root form – Getting the correct inflected form – Inserting “spontaneous” words • Putting the words in the correct order: – Word order: SVO vs. SOV, … – Unique constructions: – Divergence

Lexical choice • Homonymy/Polysemy: bank, run • Concept gap: no corresponding concepts in another language: go Greek, go Dutch, fen sui, lame duck, Chinese idioms, … • Coding (Concept lexeme mapping) differences: – More distinction in one language: e. g. , kinship vocabulary. – Different division of conceptual space:

More distinction: the “cousin” example Translations: male/female, older or younger than the speaker, from mother or father’s side, parent’s brother or sister’s child: • 堂兄 (tang xiong): father’s brother’s son, who is older • 堂弟 (tang di): father’s brother’s son, who is younger • 表兄 (biao xiong): – 姨表兄 (yi biao xiong): mother’s sister’s son, who is older – 舅表兄 (jiu biao xiong): mother’s brother’s son, who is older – 姑表兄 (gu biao xiong): father’s sister’s son, who is older • 表弟 (biao di): mother’s sibling’s son, who is younger – … • … 16+ translations 8 4.

More distinction: the “aunt” example Mother’s or father’s side, is a sister or is a brother’s wife: • 姑妈：father’s sister • 伯母：father’s younger brother’s wife • 婶： father’s elder brother’s wife • 姨妈：mother’s sister • 舅妈：mother’s brother’s wife

A large happy family • 三姨姥爷：mother’s 3 rd sister’s husband. • 3 rd among all the sisters, or among all the siblings, or among all the members in the extended family with the same “bei fen”. • Same “bei fen”：same distance to the root node of a family tree.

Sapir-Whorf hypothesis • Edward Sapir (1884 -1936): American linguist and anthropologist. • Benjamin Lee Whorf (1897 -1941): Sapir’s student. • Linguistic determinism/relativism: the language we use to some extent determines the way in which we view and think about the world around us. • Strong determinism: language determines thought, that language and thought are identical. • Weak determinism: thought is merely affected by or influenced by our language.

Sapir-Whorf hypothesis (cont) • “Snow” – Whorf: the number of words for “snow” the Inuit people have for ‘snow‘ Inuit people treat snow differently than someone who lives in a less snowdependent environment. – Pullum (1991): Other languages transmit the same ideas using phrases instead of single words. • My personal experience: – Preference of one language over the other:

Color coding study • Hypothesis: if one language categorizes color differently than another language, then the different groups should perceive it differently also. • A 1970 study – Task: give people a sample of 160 colors and ask them to sort it. – People: English speakers (blue-green distinction) and Berinmo speaker (nol-wor distinction) – the Berinmo speakers were better at matching colors across their nol, wor categories than across the English blue and green categories and English speakers were better at matching colors across blue and green than across the Berinmo nol and wor (Sawyer, 1999).

Sapir-Whorf hypothesis (cont) • What’s the relation between language, thought, and cultural perception of reality? • Does language affect thought? If so, to what degree? • The hypothesis is still under much debate.

Major challenges • Getting the right words: – Choosing the correct root form – Getting the correct inflected form – Inserting “spontaneous” words • Putting the words in the correct order: – Word order: SVO vs. SOV, … – Unique construction: – Structural divergence

Choosing the appropriate inflection • Inflection: gender, number, case, tense, … • Ex: – Number: Ch-Eng: all the concrete nouns: ch_book book, books – Gender: Eng-Fr: all the adjectives – Case: Eng-Korean: all the arguments – Tense: Ch-Eng: all the verbs: ch_buy buy, bought, will buy

Inserting spontaneous words • Function words: – Determiners: Ch-Eng: ch_book a book, the books, books – Prepositions: Ch-Eng: … ch_November … in November – Relative pronouns: Ch-Eng: … ch_buy ch_book de ch_person the person who bought /book/ – Possessive pronouns: Ch-Eng: ch_he ch_raise ch_hand He raised his hand(s) – Conjunction: Eng-Ch: Although S 1, S 2 ch_although S 1, ch_but S 2 – …

Inserting spontaneous words (cont) • Content words: – Dropped argument: Ch-Eng: ch_buy le ma Has Subj bought Obj? – Chinese First name: Eng-Ch: Jiang … ch_Jiang ch_Zemin … – Abbreviation, Acronyms: Ch-Eng: ch_12 ch_big the 12 th National Congress of the CPC (Communist Party of China) – …

Major challenges • Getting the right words: – Choosing the correct root form – Getting the correct inflected form – Inserting “spontaneous” words • Putting the words in the correct order: – Word order: SVO vs. SOV, … – Unique construction: – Structural divergence

Word order • SVO, SOV, VSO, … • VP + PP VP • VP + Adv. P + VP • Adj + N N + Adj • NP + PP NP • NP + S S NP • P + NP + P

“Unique” Constructions • Overt wh-movement: Eng-Ch: – Eng: Why do you think that he came yesterday? – Ch: you why think he yesterday come ASP? – Ch: you think he yesterday why come? • Ba-construction: Ch-Eng – She ba homework finish ASP She finished her homework. – He ba wall dig ASP CL hole He digged a hole in the wall. – She ba orange peel ASP skin She peeled the orange’s skin.

Translation divergences (based on Bonnie Dorr’s work) • Thematic divergence: I like Mary S: Marta me gusta a mi (‘Mary pleases me’) • Promotional divergence: John usually goes home S: Juan suele ira casa (‘John tends to go home’) • Demotional divergence: I like eating G: Ich esse gern (“I eat likingly) • Structural divergence: John entered the house S: Juan entro en la casa (‘John entered in the house’)

Translation divergences (cont) • Conflational divergence: I stabbed John S: Yo le di punaladas a Juan (‘I gave knifewounds to John’) • Categorial divergence: I am hungry G: Ich habe Hunger (‘I have hunger’) • Lexical divergence: John broke into the room S: Juan forzo la entrada al cuarto (‘John forced the entry to the room’)

Outline • • MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

How humans do translation? • Learn a foreign language: – Memorize word translations – Learn some patterns: – Exercise: • Passive activity: read, listen • Active activity: write, speak • Translation: – Understand the sentence – Clarify or ask for help (optional) – Translate the sentence Training stage Translation lexicon Templates, transfer rules Reinforced learning? Reranking? Decoding stage Parsing, semantics analysis? Interactive MT? Word-level? Phrase-level? Generate from meaning?

What kinds of resources are available to MT? • Translation lexicon: – Bilingual dictionary • Templates, transfer rules: – Grammar books • Parallel data, comparable data • Thesaurus, Word. Net, Frame. Net, … • NLP tools: tokenizer, morph analyzer, parser, … More resources for major languages, less for “minor” languages.

Major approaches • • • Transfer-based Interlingua Example-based (EBMT) Statistical MT (SMT) Hybrid approach

The MT triangle s esi An a Transfer-based nth Sy lys is Meaning (interlingua) Phrase-based SMT, EBMT Word-based SMT, EBMT word Word

Transfer-based MT • Analysis, transfer, generation: 1. 2. 3. 4. • Resources required: – – – • Parse the source sentence Transform the parse tree with transfer rules Translate source words Get the target sentence from the tree Source parser A translation lexicon A set of transfer rules An example: Mary bought a book yesterday.

Transfer-based MT (cont) • Parsing: linguistically motivated grammar or formal grammar? • Transfer: context-free rules? Additional constraints on the rules? Apply at most one rule at each level? How are rules created? • Translating words: word-to-word translation? • Generation: using LM or other additional knowledge? • How to create the needed resources automatically?

Interlingua • For n languages, we need n(n-1) MT systems. • Interlingua uses a language-independent representation. • Conceptually, Interlingua is elegant: we only need n analyzers, and n generators. • Resource needed: – A language-independent representation – Sophisticated analyzers – Sophisticated generators

Interlingua (cont) • Questions: – Does language-independent meaning representation really exist? If so, what does it look like? – It requires deep analysis: how to get such an analyzer: e. g. , semantic analysis – It requires non-trivial generation: How is that done? – It forces disambiguation at various levels: lexical, syntactic, semantic, discourse levels. – It cannot take advantage of similarities between a particular language pair.

Example-based MT • Basic idea: translate a sentence by using the closest match in parallel data. • First proposed by Nagao (1981). • Ex: – Training data: • w 1 w 2 w 3 w 4 w 1’ w 2’ w 3’ w 4’ • w 5 w 6 w 7 w 5’ w 6’ w 7’ • w 8 w 9 w 8’ w 9’ – Test sent: • w 1 w 2 w 6 w 7 w 9 w 1’ w 2’ w 6’ w 7’ w 9’

EMBT (cont) • Types of EBMT: – Lexical (shallow) – Morphological / POS analysis – Parse-tree based (deep) • Types of data required by EBMT systems: – – Parallel text Bilingual dictionary Thesaurus for computing semantic similarity Syntactic parser, dependency parser, etc.

EBMT (cont) • Word alignment: using dictionary and heuristics exact match • Generalization: – Clusters: dates, numbers, colors, shapes, etc. – Clusters can be built by hand or learned automatically. • Ex: – Exact match: 12 players met in Paris last Tuesday 12 Spieler trafen sich letzen Dienstag in Paris – Templates: $num players met in $city $time $num Spieler trafen sich $time in $city

Statistical MT • Basic idea: learn all the parameters from parallel data. • Major types: – Word-based – Phrase-based • Strengths: – Easy to build, and it requires no human knowledge – Good performance when a large amount of training data is available. • Weaknesses: – How to express linguistic generalization?

Comparison of resource requirement Transferbased Interlingua EBMT dictionary + + + Transfer rules + parser + + + (? ) semantic analyzer parallel data others SMT + + Universal thesaurus representation +

Hybrid MT • Basic idea: combine strengths of different approaches: – – • Syntax-based: generalization at syntactic level Interlingua: conceptually elegant EBMT: memorizing translation of n-grams; generalization at various level. SMT: fully automatic; using LM; optimizing some objective functions. Types of hybrid HT: – Borrowing concepts/methods: • SMT from EBMT: phrase-based SMT; Alignment templates • EBMT from SMT: automatically learned translation lexicon • Transfer-based from SMT: automatically learned translation lexicon, transfer rules; using LM • … – Using two MTs in a pipeline: • Using transfer-based MT as a preprocessor of SMT – Using multiple MTs in parallel, then adding a re-ranker.

Outline • • MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

Evaluation • Unlike many NLP tasks (e. g. , tagging, chunking, parsing, IE, pronoun resolution), there is no single gold standard for MT. • Human evaluation: accuracy, fluency, … – Problem: expensive, slow, subjective, non-reusable. • Automatic measures: – – Edit distance Word error rate (WER), Position-independent WER (PER) Simple string accuracy (SSA), Generation string accuracy (GSA) BLEU

Edit distance • The Edit distance (a. k. a. Levenshtein distance) is defined as the minimal cost of transforming str 1 into str 2, using three operations (substitution, insertion, deletion). • Let the operation cost be sub. Cost, ins. Cost, and del. Cost, respectively. • Let |Str 1|=m and |Str 2|=n, D(i, j) stores the edit distance of converting str 1[1. . i] to str 2[1. . j]. • D(m, n) is the answer that we are looking for. • Use DP and the complexity is O(m*n).

Calculating edit distance • D(0, 0) = 0 • D(i, 0) = del. Cost * i • D(0, j) = ins. Cost * j • D(i+1, j+1) = min( D(i, j) + sub, D(i+1, j) + ins. Cost, D(i, j+1) + del. Cost) sub = 0 = sub. Cost if str 1[i+1]=str 2[j+1] otherwise

An example • Sys: w 1 w 2 w 3 w 4 • Ref: w 1 w 3 w 2 • All three costs are 1. w 3 w 2 w 1 0 1 1 0 2 1 3 2 w 2 2 1 1 1 w 3 3 2 1 2 w 4 • Edit distance=2 w 1 4 3 2 2

WER, PER, and SSA • WER (word error rate) is edit distance, divided by |Ref|. • PER (position-independent WER): same as WER but disregards word ordering • SSA (Simple string accuracy) = 1 - WER • Previous example: – – – Sys: w 1 w 2 w 3 w 4 Ref: w 1 w 3 w 2 Edit distance = 2 WER=2/3 PER=1/3 SSA=1/3

Generation string accuracy (GSA) Example: Ref: w 1 w 2 w 3 w 4 Sys: w 2 w 3 w 4 w 1 Del=1, Ins=1 SSA=1/2 Move=1, Del=0, Ins=0 GSA=3/4

BLEU • Proposal by Papineni et. al. (2002) • Most widely used in MT community. • BLEU is a weighted average of n-gram precision (pn) between system output and all references, multiplied by a brevity penalty (BP).

N-gram precision • N-gram precision: the percent of n-grams in the system output that are correct. • Clipping: – – Sys: the the the Ref: the cat sat on the mat Unigram precision: Max_Ref_count: the max number of times a ngram occurs in any single reference translation.

N-gram precision i. e. the percent of n-grams in the system output that are correct (after clipping).

Brevity Penalty • For each sent si in system output, find closest matching reference ri (in terms of length). • Longer system output is already penalized by the n-gram precision measure.

An example • Sys: The cat was on the mat • Ref 1: The cat sat on a mat • Ref 2: There was a cat on the mat • Assuming N=3 • p 1=5/6, p 2=3/5, p 3=1/4, BP=1 BLEU=0. 50 • What if N=4?

Summary • MT in a nutshell • Major challenges in MT – Choose the right words (root form, inflection, spontaneous words) – Put them in right positions (word order, unique constructions, divergences)

Summary (cont) • Major approaches – – – Transfer-based MT Interlingua Example-based MT Statistical MT Hybrid MT • Evaluation of MT systems – Edit distance – WER, PER, SSA, GSA – BLEU