Speech and Language Processing: Where have we been and where are we going? Kenneth Ward Church AT&T Labs-Research church@att. com www. research. att. com/~kwc Eurospeech 2003
Where have we been? How To Cook A Demo (After Dinner Talk at TMI-1992 & Invited Talk at TMI-2002) • Great fun! • Effective demos Message for After Dinner Talk – Theater, theater – Production quality matters – Entertainment >> evaluation – Strategic vision >> technical correctness • Success/Catastrophe Message for After Breakfast Talk – Warning: demos can be too effective – Dangerous to raise unrealistic expectations Eurospeech 2003 2
Let’s go to the video tape! (Lesson: manage expectations) • Lots of predictions – – Entertaining in retrospect Nevertheless, many of these people went on to very successful careers: president of MIT, Microsoft exec, etc. Eurospeech 2003 3
Let’s go to the video tape! (Lesson: manage expectations) • Lots of predictions – – 1. Entertaining in retrospect Nevertheless, many of these people went on to very successful careers: president of MIT, Microsoft exec, etc. Machine Translation (1950 s) video – Classic example of a demo embarrassment in retrospect Eurospeech 2003 4
Let’s go to the video tape! (Lesson: manage expectations) • Lots of predictions – – 1. Entertaining in retrospect Nevertheless, many of these people went on to very successful careers: president of MIT, Microsoft exec, etc. Machine Translation (1950 s) video – 2. Classic example of a demo embarrassment in retrospect Translating telephone (late 1980 s) video – – Pierre Isabelle pulled a similar demo because it was so effective The limitations of the technology were hard to explain to public • Though well understood by research community Eurospeech 2003 5
Let’s go to the video tape! (Lesson: manage expectations) • Lots of predictions – – 1. Entertaining in retrospect Nevertheless, many of these people went on to very successful careers: president of MIT, Microsoft exec, etc. Machine Translation (1950 s) video – 2. Classic example of a demo embarrassment in retrospect Translating telephone (late 1980 s) video – – Pierre Isabelle pulled a similar demo because it was so effective The limitations of the technology were hard to explain to public • 3. Though well understood by research community Apple (~1990) video – – Still having trouble setting appropriate expectations Factoid: the day of this demo, speech recognition deployed at scale in AT&T network – with significant lasting impact – but little media Eurospeech 2003 6
Let’s go to the video tape! (Lesson: manage expectations) • Lots of predictions – – 1. Entertaining in retrospect Nevertheless, many of these people went on to very successful careers: president of MIT, Microsoft exec, etc. Machine Translation (1950 s) video – 2. Classic example of a demo embarrassment in retrospect Translating telephone (late 1980 s) video – – Pierre Isabelle pulled a similar demo because it was so effective The limitations of the technology were hard to explain to public • 3. Apple (~1990) video – – 4. Though well understood by research community Still having trouble setting appropriate expectations Factoid: the day of this demo, speech recognition deployed at scale in AT&T network – with significant lasting impact – but little media Andy Rooney (~1990): reset expectations video Eurospeech 2003 7
Outline: Where have we been and where are we going? 1. Consistent progress over decades Ø 2. Moore’s Law, Speech Coding, Error Rate History repeats itself • • 3. Empiricism: 1950 s Rationalism: 1970 s Empiricism: 1990 s Rationalism: 2010 s (? ) Managing Expectations Discontinuities: Fundamental changes that invalidate fundamental assumptions • • Petabytes: $2, 000 Can demand keep up with supply? If not Tech meltdown New priorities: Search >> Compression & Dictation Eurospeech 2003 8
Charles Wayne’s Challenge: Demonstrate Consistent Progress Over Time Managing Expectations • Controversial in 1980 s – – • But not in 1990 s Though, lgrumbling Benefits 1. Agreement on what to do 2. Limits endless discussion 3. Helps sell the field • • • Manage expectations Fund raising Risks (similar to benefits) 1. All our eggs are in one basket (lack of diversity) 2. Not enough discussion • Hard to change course 3. Methodology Burden Eurospeech 2003 9
Hockey Stick Business Case Last Year This Year Eurospeech 2003 Next Year 10
Moore’s Law: Ideal Answer Where have we been and where are we going? Eurospeech 2003 11
Where have we been and where are we going? Moore’s Law: Ideal Answer Why different slopes? Hyper-Inflation 1. Progress limited by physics – Disk seek: 10 years (normal inflation) – Disk capacity: 1 year (hyper-inflation) Normal Inflation Physics & Investment Rate of Progress in Speech & Language (and everything) Eurospeech 2003 12
Where have we been and where are we going? Moore’s Law: Ideal Answer Why different slopes? Hyper-Inflation 1. Progress limited by physics – Disk seek: 10 years (normal inflation) – Disk capacity: 1 year (hyper-inflation) 2. Progress limited by investment – Case history: PCs improved faster than supercomputers (Cray) • PCs: larger market more R&D – Irony: “Dis-economy of Scale” – Danny Hillis (Thinking Machines) Normal Inflation Physics & Investment Rate of Progress in Speech & Language (and everything) • Computing is better (cheaper & faster) on smaller machines – PCs >> big iron – LAN routers >> 5 ESS (big phone switch) – Economies of scale depend on size of market, not size of machine • • Market: PC >> big iron (Economist View) Machine: PC << big iron (CS View) Eurospeech 2003 13
Where have we been and where are we going? Moore’s Law: Ideal Answer Why different slopes? Hyper-Inflation 1. Progress limited by physics – Disk seek: 10 years (normal inflation) – Disk capacity: 1 year (hyper-inflation) 2. Progress limited by investment – Case history: PCs improved faster than supercomputers (Cray) • PCs: larger market more R&D – Irony: “Dis-economy of Scale” – Danny Hillis (Thinking Machines) Normal Inflation Physics & Investment Rate of Progress in Speech & Language (and everything) • Computing is better (cheaper & faster) on smaller machines – PCs >> big iron – LAN routers >> 5 ESS (big phone switch) – Economies of scale depend on size of market, not size of machine • • Market: PC >> big iron (Economist View) Machine: PC << big iron (CS View) Eurospeech 2003 14
Where have we been and where are we going? Moore’s Law: Ideal Answer Why different slopes? Hyper-Inflation 1. Progress limited by physics – Disk seek: 10 years (normal inflation) – Disk capacity: 1 year (hyper-inflation) 2. Progress limited by investment – Case history: PCs improved faster than supercomputers (Cray) • PCs: larger market more R&D – Irony: “Dis-economy of Scale” – Danny Hillis (Thinking Machines) Normal Inflation Physics & Investment Rate of Progress in Speech & Language (and everything) • Computing is better (cheaper & faster) on smaller machines – PCs >> big iron – LAN routers >> 5 ESS (big phone switch) – Economies of scale depend on size of market, not size of machine • • Market: PC >> big iron (Economist View) Machine: PC << big iron (CS View) Eurospeech 2003 15
Borrowed Slide Rich Cox Evolution of Speech Coder Performance Speech Quality Excellent Good North American TDMA 2000 Fair 1990 Poor 1980 ITU Recommendations Cellular Standards Secure Telephony 1980 Profile 1990 Profile 2000 Profile Bad Bit Rate (kb/s) Eurospeech 2003 16
Ceiling Speech Coding (Telephony) • More complicated than Moore’s Law – Many Dimensions: Bit Rate, Quality, Complexity and Delay – Quality ceiling (imposed by telephone standards) • Easy to reach the ceiling at high bit rates (≥ 8 kb/s) • More room for progress at low bit rates (≤ 8 kb/s) Eurospeech 2003 17
Ceiling Speech Coding (Telephony) • More complicated than Moore’s Law – Many Dimensions: Bit Rate, Quality, Complexity and Delay – Quality ceiling (imposed by telephone standards) • Easy to reach the ceiling at high bit rates (≥ 8 kb/s) • More room for progress at low bit rates (≤ 8 kb/s) • Moore’s Law Time Constant – Bit rates half every decade (≤ 8 kb/s) – Relatively slow by Moore’s Law standards (not hyper-inflation) • Performance doubles every decade • Like disk seek or money in the bank (normal inflation) – Limited more by physics than investment Eurospeech 2003 18
Ceiling Speech Coding (Telephony) • More complicated than Moore’s Law – Many Dimensions: Bit Rate, Quality, Complexity and Delay – Quality ceiling (imposed by telephone standards) • Easy to reach the ceiling at high bit rates (≥ 8 kb/s) • More room for progress at low bit rates (≤ 8 kb/s) • Moore’s Law Time Constant – Bit rates half every decade (≤ 8 kb/s) – Relatively slow by Moore’s Law standards (not hyper-inflation) • Performance doubles every decade • Like disk seek or money in the bank (normal inflation) – Limited more by physics than investment • Potential compression opportunity – At most 10 x: 8 kb/s 2 kb/s 1 kb/s (? ) – Entropy: 50 bits per sec (Roger Moore) Eurospeech 2003 19
Ceiling Speech Coding (Telephony) • More complicated than Moore’s Law – Many Dimensions: Bit Rate, Quality, Complexity and Delay – Quality ceiling (imposed by telephone standards) • Easy to reach the ceiling at high bit rates (≥ 8 kb/s) • More room for progress at low bit rates (≤ 8 kb/s) • Moore’s Law Time Constant – Bit rates half every decade (≤ 8 kb/s) – Relatively slow by Moore’s Law standards (not hyper-inflation) • Performance doubles every decade • Like disk seek or money in the bank (normal inflation) – Limited more by physics than investment • Potential compression opportunity – At most 10 x: 8 kb/s 2 kb/s 1 kb/s (? ) 50 bits per sec (? ? ) • Speech (2 kb/s) >> text (2 bits/char): 10 -1000 times more bits – Speech coding will not close this gap foreseeable future Eurospeech 2003 20
Where have we been and where are we going? 1. Consistent progress over decades • • Ø 2. Moore’s Law Speech Coding Reducing Speech Recognition Error Rates History repeats itself • • 3. Empiricism: 1950 s Rationalism: 1970 s Empiricism: 1990 s Rationalism: 2010 s (? ) Discontinuities: Fundamental changes that invalidate fundamental assumptions • • Petabytes: $2, 000 Can demand keep up with supply? If not Tech meltdown New priorities: Search >> Compression & Dictation Eurospeech 2003 21
Error Rate Borrowed Slide Audrey Le (NIST) Moore’s Law Time Constant: • 10 x improvement per decade • Limited by R&D Investment • (Not Physics) Date (15 years) Eurospeech 2003 22
Milestones in Speech and Multimodal Technology Research Borrowed Slide Small Vocabulary, Acoustic Phoneticsbased Isolated Words Filter-bank analysis; Timenormalization ; Dynamic programming 1962 Medium Large Vocabulary, Template-based Statistical-based Isolated Words; Connected Digits; Continuous Speech Pattern recognition; LPC analysis; Clustering algorithms; Level building; 1967 1972 Large Vocabulary; Syntax, Semantics, Connected Words; Continuous Speech; Speech Understanding Hidden Markov models; Stochastic Language modeling; Stochastic language understanding; Finite-state machines; Statistical learning; 1977 1982 1987 1992 Very Large Vocabulary; Semantics, Multimodal Dialog, TTS Spoken dialog; Multiple modalities Concatenative synthesis; Machine learning; Mixedinitiative dialog; 1997 2002 Year Consistent improvement over time, but unlike Moore’s Law, hard to extrapolate (predict future) Eurospeech 2003 23
Speech-Related Technologies Where will the field go in 10 years? Niels Ole Bernsen (ed) 2003 Useful speech recognition-based language tutor 2003 Useful portable spoken sentence translation systems 2003 First pro-active spoken dialogue with situation awareness 2004 Satisfactory spoken car navigation systems 2005 Small-vocabulary (> 1000 words) spoken conversational systems 2006 Multiple-purpose personal assistants (spoken dialog, animated characters) 2006 Task-oriented spoken translation systems for the web 2006 Useful speech summarization systems in top languages 2008 Useful meeting summarization systems 2010 Medium-size vocabulary conversational systems Eurospeech 2003 24
Where have we been and where are we going? Manage Consistent Progress over Time Expectations Extrapolation/Prediction Physics and is Applicable Physics Investment Eurospeech 2003 Extrapolation/Prediction is Not Applicable 25
Where have we been and where are we going? 1. Consistent progress over decades • Ø Moore’s Law, Speech Coding, Error Rate History repeats itself • • 3. Empiricism: 1950 s Rationalism: 1970 s Empiricism: 1990 s Rationalism: 2010 s (? ) Discontinuities: Fundamental changes that invalidate fundamental assumptions • • Petabytes: $2, 000 Can demand keep up with supply? If not Tech meltdown New priorities: Search >> Compression & Dictation Eurospeech 2003 26
It has been claimed that Recent progress made possible by Empiricism Progress (or Oscillating Fads)? • 1950 s: Empiricism was at its peak – Dominating a broad set of fields • Ranging from psychology (Behaviorism) • To electrical engineering (Information Theory) – Psycholinguistics: Word frequency norms (correlated with reaction time, errors) • Word association norms (priming): bread and butter, doctor / nurse – Linguistics/psycholinguistics: focus on distribution (correlate of meaning) • Firth: “You shall know a word by the company it keeps” • Collocations: Strong tea v. powerful computers • 1970 s: Rationalism was at its peak – with Chomsky’s criticism of ngrams in Syntactic Structures (1957) – and Minsky and Papert’s criticism of neural networks in Perceptrons (1969). • 1990 s: Revival of Empiricism – Availability of massive amounts of data (popular arg, even before the web) • “More data is better data” • Quantity >> Quality (balance) – Pragmatic focus: • What can we do with all this data? • Better to do something than nothing at all – Empirical methods (and focus on evaluation): Speech Language • 2010 s: Revival of Rationalism (? ) Eurospeech 2003 27
It has been claimed that Recent progress made possible by Empiricism Progress (or Oscillating Fads)? • 1950 s: Empiricism was at its peak – Dominating a broad set of fields • Ranging from psychology (Behaviorism) • To electrical engineering (Information Theory) – Psycholinguistics: Word frequency norms (correlated with reaction time, errors) • Word association norms (priming): bread and butter, doctor / nurse – Linguistics/psycholinguistics: focus on distribution (correlate of meaning) • Firth: “You shall know a word by the company it keeps” • Collocations: Strong tea v. powerful computers • 1970 s: Rationalism was at its peak – with Chomsky’s criticism of ngrams in Syntactic Structures (1957) – and Minsky and Papert’s criticism of neural networks in Perceptrons (1969). • 1990 s: Revival of Empiricism – Availability of massive amounts of data (popular arg, even before the web) • “More data is better data” • Quantity >> Quality (balance) – Pragmatic focus: • What can we do with all this data? • Better to do something than nothing at all – Empirical methods (and focus on evaluation): Speech Language • 2010 s: Revival of Rationalism (? ) Eurospeech 2003 28
It has been claimed that Recent progress made possible by Empiricism Progress (or Oscillating Fads)? • 1950 s: Empiricism was at its peak – Dominating a broad set of fields • Ranging from psychology (Behaviorism) • To electrical engineering (Information Theory) – Psycholinguistics: Word frequency norms (correlated with reaction time, errors) • Word association norms (priming): bread and butter, doctor / nurse – Linguistics/psycholinguistics: focus on distribution (correlate of meaning) • Firth: “You shall know a word by the company it keeps” • Collocations: Strong tea v. powerful computers • 1970 s: Rationalism was at its peak – with Chomsky’s criticism of ngrams in Syntactic Structures (1957) – and Minsky and Papert’s criticism of neural networks in Perceptrons (1969). • 1990 s: Revival of Empiricism – Availability of massive amounts of data (popular arg, even before the web) • “More data is better data” • Quantity >> Quality (balance) – Pragmatic focus: • What can we do with all this data? • Better to do something than nothing at all – Empirical methods (and focus on evaluation): Speech Language • 2010 s: Revival of Rationalism (? ) Eurospeech 2003 29
It has been claimed that Recent progress made possible by Empiricism Progress (or Oscillating Fads)? • 1950 s: Empiricism was at its peak – Dominating a broad set of fields • Ranging from psychology (Behaviorism) • To electrical engineering (Information Theory) • Periodic signals are continuous • Support extrapolation/prediction • Progress? Consistent progress? – Psycholinguistics: Word frequency norms (correlated with reaction time, errors) • Word association norms (priming): bread and butter, doctor / nurse – Linguistics/psycholinguistics: focus on distribution (correlate of meaning) • Firth: “You shall know a word by the company it keeps” • Collocations: Strong tea v. powerful computers • 1970 s: Rationalism was at its peak – with Chomsky’s criticism of ngrams in Syntactic Structures (1957) – and Minsky and Papert’s criticism of neural networks in Perceptrons (1969). • 1990 s: Revival of Empiricism – Availability of massive amounts of data (popular arg, even before the web) • “More data is better data” • Quantity >> Quality (balance) Consistent progress? – Pragmatic focus: • What can we do with all this data? • Better to do something than nothing at all – Empirical methods (and focus on evaluation): Speech Language • 2010 s: Revival of Rationalism (? ) Extrapolation/Prediction: Applicable? Eurospeech 2003 30
Speech Language Has the pendulum swung too far? • What happened between TMI-1992 and TMI-2002 (if anything)? • Have empirical methods become too popular? – Has too much happened since TMI-1992? • I worry that the pendulum has swung so far that – We are no longer training students for the possibility • that the pendulum might swing the other way • We ought to be preparing students with a broad education including: • – Statistics and Machine Learning – as well as Linguistic Theory History repeats itself: Mark Twain; bad idea then and still a bad idea now – 1950 s: empiricism – 1970 s: rationalism (empiricist methodology became too burdensome) – 1990 s: empiricism – 2010 s: rationalism (empiricist methodology is burdensome, again) Eurospeech 2003 31
Speech Language Has the pendulum swung too far? • What happened between TMI-1992 and TMI-2002 (if anything)? • Have empirical methods become too popular? Plays well at – Has too much happened since TMI-1992? Machine Translation – We are no longer training students for the possibility conferences • that the pendulum might swing the other way • We ought to be preparing students with a broad education including: • I worry that the pendulum has swung so far that • – Statistics and Machine Learning – as well as Linguistic Theory History repeats itself: Mark Twain; bad idea then and still a bad idea now – 1950 s: empiricism – 1970 s: rationalism (empiricist methodology became too burdensome) – 1990 s: empiricism – 2010 s: rationalism (empiricist methodology is burdensome, again) Eurospeech 2003 32
Speech Language Has the pendulum swung too far? • What happened between TMI-1992 and TMI-2002 (if anything)? • Have empirical methods become too popular? Plays well at – Has too much happened since TMI-1992? Machine Translation – We are no longer training students for the possibility conferences • that the pendulum might swing the other way • We ought to be preparing students with a broad education including: • I worry that the pendulum has swung so far that • – Statistics and Machine Learning – as well as Linguistic Theory History repeats itself: Mark Twain; bad idea then and still a bad idea now – 1950 s: empiricism – 1970 s: rationalism (empiricist methodology became too burdensome) – 1990 s: empiricism – 2010 s: rationalism (empiricist methodology is burdensome, again) Eurospeech 2003 33
Speech Language Has the pendulum swung too far? • What happened between TMI-1992 and TMI-2002 (if anything)? • Have empirical methods become too popular? Plays well at – Has too much happened since TMI-1992? Machine Translation – We are no longer training students for the possibility conferences • that the pendulum might swing the other way • We ought to be preparing students with a broad education including: • I worry that the pendulum has swung so far that – Statistics and Machine Learning – as well as Linguistic Theory • History repeats itself: – – 1950 s: empiricism 1970 s: rationalism (empiricist methodology became too burdensome) 1990 s: empiricism 2010 s: rationalism (empiricist methodology is burdensome, again) Mark Twain; bad idea then and still a bad idea now Eurospeech 2003 34
Rationalism Well-known Chomsky, Minsky advocates Model Competence Model Contexts of Interest Phrase-Structure Empiricism Shannon, Skinner, Firth, Harris Noisy Channel Model N-Grams All and Only Explanatory Descriptive Theoretical Goals Minimize Prediction Error (Entropy) Applied Linguistic Agreement & Wh. Generalizations movement Principle-Based, Parsing Strategies CKY (Chart), ATNs, Unification Understanding Applications Who did what to whom Eurospeech 2003 Collocations & Word Associations Forward-Backward (HMMs), Inside-outside (PCFGs) Recognition Noisy Channel Applications 35
Where have we been and where are we going? 1. Consistent progress over decades • 2. Moore’s Law, Speech Coding, Error Rate History repeats itself • • Ø Empiricism: 1950 s Rationalism: 1970 s Empiricism: 1990 s Rationalism: 2010 s (? ) Discontinuities: Fundamental changes that invalidate fundamental assumptions • • Petabytes: $2, 000 Can demand keep up with supply? If not Tech meltdown New priorities: Search >> Compression & Dictation Eurospeech 2003 36
Meeting Demand for Petabytes Bet: Speech >> Text (because we aren’t going to solve all “speech” problems) • Moore’s Law More and More Supply – Disks, Memory, Network Bandwidth, everything… – Petabytes are coming: $2, 000 (today) $2, 000 (in 10 years) • Can demand keep up? – If not, revenues will collapse tech meltdown – Much worse than the Dot-Bomb… Discontinuity • Ans 1: no problem – Demand has always kept up – Pundits have never been able to explain why • Thomas J. Watson (1943): I think there is a world market for maybe five computers – But if you build it, they will come • Ans 2: big problem (prices for PCs & Networks are collapsing) – – Demand is everything Anyone (even a dot-com) can build a network, But the challenge is to sell it Need a kill app (more minutes on the network) Eurospeech 2003 37
Meeting Demand for Petabytes Bet: Speech >> Text (because we aren’t going to solve all “speech” problems) • Moore’s Law More and More Supply – Disks, Memory, Network Bandwidth, everything… – Petabytes are coming: $2, 000 (today) $2, 000 (in 10 years) • Can demand keep up? – If not, revenues will collapse tech meltdown – Much worse than the Dot-Bomb… Eurospeech 2003 Discontinuity 38
Meeting Demand for Petabytes Bet: Speech >> Text (because we aren’t going to solve all “speech” problems) • Moore’s Law More and More Supply – Disks, Memory, Network Bandwidth, everything… – Petabytes are coming: $2, 000 (today) $2, 000 (in 10 years) • Can demand keep up? – If not, revenues will collapse tech meltdown – Much worse than the Dot-Bomb… Discontinuity • Ans 1: no problem – Demand has always kept up – Pundits have never been able to explain why • Thomas J. Watson (1943): I think there is a world market for maybe five computers www. wikipedia. org/wiki/Thomas+J. +Watson – But if you build it, they will come • Ans 2: big problem (prices for PCs & Networks are collapsing) – – Demand is everything Anyone (even a dot-com) can build a network, But the challenge is to sell it Need a killer app (more minutes on the network) Eurospeech 2003 39
How much is a Petabyte? (1015 bytes) • Question from execs: – How do I explain to a lay audience • How much is a petabyte • And why everyone will buy lots of them • Wrong answer: – 106 is a million (a floppy disk/email msg) – 109 is a billion (a billion here, a billion there…) – 1012 is a trillion (the US debt) – 1015 is a zillion (= , an unimaginably large #) Eurospeech 2003 40
How much is a Petabyte? (1015 bytes) • Question from execs: – How do I explain to a lay audience • How much is a petabyte • And why everyone will buy lots of them • Wrong answer: – 106 is a million (a floppy disk/email msg) – 109 is a billion (a billion here, a billion there…) – 1012 is a trillion (the US debt) – 1015 is a zillion (= , an unimaginably large #) Eurospeech 2003 41
How much is a Petabyte? Some more wrong answers • Goal: create demand for a petabyte/lifetime – ≈ 1015 bytes/100 years ≈ 18 megabytes/minute – Text: 18, 000 pages/min – Speech: 317 telephone channels for 100 years per capita • Text won’t do it – Speech probably won’t either, but it is closer – DVD video will (1. 8 gigabytes/hour = 1. 6 petabytes/lifetime), but • Too much opportunity for compression • Not enough demand for Picture Phone (privacy concerns) • Bank on speech recognition not working too well – Can’t afford big improvements in compression: • Speech rates Text rates Eurospeech 2003 Fortunately, that won’t happen 42
New Research Challenges • New Priorities • Old Priorities – Dictation application dates back to days of dictation machines – Speech recognition has not displaced typing – Increase demand for space >> Data entry • New Killer Apps – Search >> Dictation • Speech recognition has improved • But typing skills have improved even more • Speech Google! – Data mining – My son will learn typing in 1 st grade – Sec rarely take dictation – Dictation machines are history • My son may never see one • Museums have slide rulers and steam trains – But dictation machines? Eurospeech 2003 43
Data Mining & Call Centers: An Intelligence Bonanza • Some companies are collecting information with technology designed to monitor incoming calls for service quality. • Last summer, Continental Airlines Inc. installed software from Witness Systems Inc. to monitor the 5, 200 agents in its four reservation centers. • But the Houston airline quickly realized that the system, which records customer phone calls and information on the responding agent's computer screen, also was an intelligence bonanza, says André Harris, reservations training and quality-assurance director. Eurospeech 2003 44
Borrowed Slide In Search of Peta. Byte Databases Jim Gray Tony Hey Eurospeech 2003
Borrowed Slide Personal 100 GB today The Personal Petabyte (someday) • It’s coming (2 M$ today… 2 K$ in 10 years) • Today the pack rats have ~ 10 -100 GB – 1 -10 GB in text (e. Mail, PDF, PPT, OCR…) – 10 GB – 50 GB tiff, mpeg, jpeg, … – Some have 1 TB (voice + video). Text won’t do it; Speech won’t either • Video can drive it to 1 PB. • Online PB affordable in 10 years. • Get ready: tools to capture, manage, organize, search, display will be big app. Eurospeech 2003 46
300 TB (cooked) Hotmail / Yahoo Borrowed Slide Per Capita Demand: Tiny • Clone front ends ~10, 000@hotmail. • Application servers – – ~100 @ hotmail Get mail box Get/put mail Disk bound • ~30, 000 disks • ~ 20 admins Cost of storage: People Eurospeech 2003 47
AOL (msn) (1 PB? ) • • • Borrowed Slide Per Capita Demand: Tiny 10 B transactions per day (10% of that) Huge storage Huge traffic Lots of eye candy DB used for security/accounting. GUESS AOL is a petabyte – (40 M x 10 MB = 400 x 1012) Eurospeech 2003 48
Google 1. 5 PB as of last spring • 8, 000 no-name PCs Borrowed Slide 2001 Per Capita Demand: Tiny – Each 1/3 U, 2 x 80 GB disk, 2 cpu 256 MB ram • • • 1. 4 PB online. 2 TB ram online 8 Tera. Ops Slice-price is 1 K$ so 8 M$. 15 admins (!) (== 1/100 TB). Cost of storage: People Eurospeech 2003 49
Digital Immortality: Gordon Bell & Jim Gray (2000) Estimated Lifetime Storage Requirements Data-types Per day Per Lifetime email, papers, text 0. 5 MB 15 GB photos 2 MB 150 GB speech 40 MB 1. 2 TB music 60 MB 5. 0 TB video-lite (200 Kb/s) 1 GB 100 TB DVD video (4. 3 Mb/s = 1. 8 GB/hour) 20 GB 1 PB Eurospeech 2003 50
Future of Tech Industry Depends On… • Supply running into a (physical) limit – Moore’s Law breaking down – And little progress on compression • Demand keeping up Not Likely Not Optimistic – If we build it, they will come… • Bell & Gray underestimating demand by a lot – Everyone wanting lots and lots of speech – Everyone wanting lots of video – A miracle (the fat lady might sing…) Not Likely – Big progress on searching speech & video Best Bet! Eurospeech 2003 51
Bait and Switch Strategy www. elsnet. org • Bait: public Internet – Large, sexy, available, rich hypertext structure • Switch: as large as the web is – There are larger & more valuable private repositories • Private Intranets & telephone networks – Exclusivity Value • No one cares about data that everyone can have • Just as Groucho Marx doesn’t want to be in a club that… • Strategy: Use the public Intranet to develop, test and socialize new ways to extract value from large linguistic repositories – Value to society: Port solutions to private repositories Eurospeech 2003 52
Bait and Switch Strategy www. elsnet. org • Bait: public Internet – Large, sexy, available, rich hypertext structure • Switch: as large as the web is – There are larger & more valuable private repositories • Private Intranets & telephone networks – Exclusivity Value • No one cares about data that everyone can have • Just as Groucho Marx doesn’t want to be in a club that… • Strategy: Use the public Intranet to develop, test and socialize new ways to extract value from large linguistic repositories – Value to society: Port solutions to private repositories Eurospeech 2003 53
Bait and Switch Strategy www. elsnet. org • Bait: public Internet – Large, sexy, available, rich hypertext structure • Switch: as large as the web is – There are larger & more valuable private repositories • Private Intranets & telephone networks – Exclusivity Value • No one cares about data that everyone can have • Just as Groucho Marx doesn’t want to be in a club that… • Strategy: Use the public Intranet to develop, test and socialize new ways to extract value from large linguistic repositories – Value to society: Port solutions to private repositories Eurospeech 2003 54
Switch: How Large is Large? • Web Renewed Excitement – Large, rich hypertext structure & publicly available – Ngram freqs Google = 1000 * BNC 1 TB (ngram freqs) or • Google: 100 Billion Words • British National Corpus (BNC): 100 Million Words Eurospeech 2003 1 PB (Gray)? 55
Switch: How Large is Large? • Web Renewed Excitement – Large, rich hypertext structure & publicly available – Ngram freqs Google = 1000 * BNC 1 TB (ngram freqs) or • Google: 100 Billion Words • British National Corpus (BNC): 100 Million Words 1 PB (Gray)? • It is often said that the web is the largest repository but… – Changes to copyright laws could unlock vast resources: www. lexisnexis. com • Private Intranets and telephone networks >> Public Web – American Telephone Network (FCC): 1 line/person • Usage: 1 hour/day/line • Assume 1 sec ≈ 1 word 10 Google collections/day – Currently, Intranets (data) ≈ telephones (voice) • But data is growing faster than voice – AT&T networks: 1 PB/day • Worldwide networks: tens of PB/day Eurospeech 2003 56
Switch: How Large is Large? • Web Renewed Excitement – Large, rich hypertext structure & publicly available – Ngram freqs Google = 1000 * BNC 1 TB (ngram freqs) or • Google: 100 Billion Words • British National Corpus (BNC): 100 Million Words 1 PB (Gray)? • It is often said that the web is the largest repository but… – Changes to copyright laws could unlock vast resources: www. lexisnexis. com • Private Intranets and telephone networks >> Public Web – American Telephone Network (FCC): 1 line/person • Usage: 1 hour/day/line • Assume 1 sec ≈ 1 word 10 Google collections/day – Currently, Intranets (data) ≈ telephones (voice) • But data is growing faster than voice – AT&T networks: 1 PB/day • Worldwide networks: tens of PB/day Eurospeech 2003 57
Switch: How Large is Large? • Web Renewed Excitement – Large, rich hypertext structure & publicly available – Ngram freqs Google = 1000 * BNC 1 TB (ngram freqs) or • Google: 100 Billion Words • British National Corpus (BNC): 100 Million Words 1 PB (Gray)? • It is often said that the web is the largest repository but… – Changes to copyright laws could unlock vast resources: www. lexisnexis. com • Private Intranets and telephone networks >> Public Web – American Telephone Network (FCC): 1 line/person • Usage: 1 hour/day/line • Assume 1 sec ≈ 1 word 10 Google collections/day – Currently, Intranets (data) ≈ telephones (voice) • But data is growing faster than voice – AT&T networks: 1 PB/day • Worldwide networks: tens of PB/day Eurospeech 2003 58
Switch: How Large is Large? • Web Renewed Excitement – Large, rich hypertext structure & publicly available – Ngram freqs Google = 1000 * BNC 1 TB (ngram freqs) or • Google: 100 Billion Words • British National Corpus (BNC): 100 Million Words 1 PB (Gray)? • It is often said that the web is the largest repository but… – Changes to copyright laws could unlock vast resources: www. lexisnexis. com • Private Intranets and telephone networks >> Public Web – American Telephone Network (FCC): 1 line/person • Usage: 1 hour/day/line • Assume 1 sec ≈ 1 word 10 Google collections/day – Currently, Intranets (data) ≈ telephones (voice) • But data is growing faster than voice – AT&T networks: 1 PB/day • Worldwide networks: tens of PB/day Eurospeech 2003 A lot of speech, but not PB per capita 59
Privacy Concerns: Private Data is Private (Exclusivity Value) • Data on private intranets cannot be distributed – And most telephone conversations cannot even be recorded • let alone distributed • But attitudes are changing – It used to be considered rude to have an answering machine – Now it is considered rude not to have one • Between answering machines and call centers, perhaps 10% of telephone traffic can be recorded (≈ 1 PB/day) – Customer expectation: call centers can retrieve recordings of previous calls based on content • New capabilities new public policy – Video recording: • Expected in banks (ATMs) • Prohibited in rest rooms (except children’s YMCA locker room) Eurospeech 2003 60
Privacy Concerns: Private Data is Private (Exclusivity Value) • Data on private intranets cannot be distributed – And most telephone conversations cannot even be recorded • let alone distributed • But attitudes are changing – It used to be considered rude to have an answering machine – Now it is considered rude not to have one • Between answering machines and call centers, perhaps 10% of telephone traffic can be recorded (≈ 1 PB/day) – Customer expectation: call centers can retrieve recordings of previous calls based on content • New capabilities new public policy – Video recording: • Expected in banks (ATMs) • Prohibited in rest rooms (except children’s YMCA locker room) Eurospeech 2003 61
Privacy Concerns: Private Data is Private (Exclusivity Value) • Data on private intranets cannot be distributed – And most telephone conversations cannot even be recorded • let alone distributed • But attitudes are changing – It used to be considered rude to have an answering machine – Now it is considered rude not to have one • Between answering machines and call centers, perhaps 10% of telephone traffic can be recorded (≈ 1 PB/day) – Customer expectation: call centers can retrieve recordings of previous calls based on content • New capabilities new public policy – Video recording: • Expected in banks (ATMs) • Prohibited in rest rooms (except children’s YMCA locker room) Eurospeech 2003 62
Privacy Concerns: Private Data is Private (Exclusivity Value) • Data on private intranets cannot be distributed – And most telephone conversations cannot even be recorded • let alone distributed • But attitudes are changing – It used to be considered rude to have an answering machine – Now it is considered rude not to have one • Between answering machines and call centers, perhaps 10% of telephone traffic can be recorded (≈ 1 PB/day) – Customer expectation: call centers can retrieve recordings of previous calls based on content • New capabilities new public policy – Video recording: • Expected in banks (ATMs) • Prohibited in rest rooms (except children’s YMCA locker room) Eurospeech 2003 63
In the past, recording all this data would have been prohibitively expensive • Thanks to Moore’s Law – Storage costs have been falling faster than transport – And will continue to do so for some time • Even at current prices, transport >> storage – Transport: Long-distance telephone calls: 5 cents per minute of speech – Storage: Disk space: ½ cent per minute of speech • If I am willing to pay for a call – I might as well keep the speech online forever • Similar comments hold for data (web pages) – If I am willing to pay to fetch a web page • I might as well cache it for a long time • Why flush a page if there is any chance that it might be requested again? – Web caches crawlers • Go find the pages that I might ask for and keep them forever • Storage is cheap (compared to transport) Eurospeech 2003 64
In the past, recording all this data would have been prohibitively expensive • Thanks to Moore’s Law – Storage costs have been falling faster than transport – And will continue to do so for some time • Even at current prices, transport >> storage – Transport: Long-distance telephone calls: 5 cents per minute of speech – Storage: Disk space: ½ cent per minute of speech • If I am willing to pay for a call – I might as well keep the speech online forever • Similar comments hold for data (web pages) – If I am willing to pay to fetch a web page • I might as well cache it for a long time • Why flush a page if there is any chance that it might be requested again? – Web caches crawlers • Go find the pages that I might ask for and keep them forever • Storage is cheap (compared to transport) Eurospeech 2003 65
In the past, recording all this data would have been prohibitively expensive • Thanks to Moore’s Law – Storage costs have been falling faster than transport – And will continue to do so for some time • Even at current prices, transport >> storage – Transport: Long-distance telephone calls: 5 cents per minute of speech – Storage: Disk space: ½ cent per minute of speech • If I am willing to pay for a call – I might as well keep the speech online forever • Similar comments hold for data (web pages) – If I am willing to pay to fetch a web page • I might as well cache it for a long time • Why flush a page if there is any chance that it might be requested again? – Web caches crawlers • Go find the pages that I might ask for and keep them forever • Storage is cheap (compared to transport) Eurospeech 2003 66
In the past, recording all this data would have been prohibitively expensive • Thanks to Moore’s Law – Storage costs have been falling faster than transport – And will continue to do so for some time • Even at current prices, transport >> storage – Transport: Long-distance telephone calls: 5 cents per minute of speech – Storage: Disk space: ½ cent per minute of speech • If I am willing to pay for a call – I might as well keep the speech online forever • Similar comments hold for data (web pages) – If I am willing to pay to fetch a web page • I might as well cache it for a long time • Why flush a page if there is any chance that it might be requested again? – Web caches crawlers • Go find the pages that I might ask for and keep them forever • Storage is cheap (compared to transport) Eurospeech 2003 67
In the past, recording all this data would have been prohibitively expensive • Thanks to Moore’s Law – Storage costs have been falling faster than transport – And will continue to do so for some time • Even at current prices, transport >> storage – Transport: Long-distance telephone calls: 5 cents per minute of speech – Storage: Disk space: ½ cent per minute of speech • If I am willing to pay for a call – I might as well keep the speech online forever • Similar comments hold for data (web pages) – If I am willing to pay to fetch a web page • I might as well cache it for a long time • Why flush a page if there is any chance that it might be requested again? – Web caches crawlers • Go find the pages that I might ask for and keep them forever • Storage is cheap (compared to transport) Eurospeech 2003 68
Bait: Use Web to Establish Excitement: More data is better data • Shocking at TMI-1992 (Bob Mercer) Larger market share More $$ for R&D Better Moore’s Law Time Constant – – • • but less so a decade later (Eric Brill) Many researchers are finding that performance improves with corpus size, over full range of sizes that are available. EMNLP-2002 Best paper (& CL): Using the Web to Overcome Data Sparseness, Keller et al – For many tasks: Google is displacing BNC just as PCs displaced Crays Larger corpora (100 B Google) >> Smaller corpora (100 M BNC) – – • – My spin Language modelling Predicting psycholinguistic judgements Collecting more data is better than tricks for not collecting data • • Smoothing, balance, etc. Tricks have limited power: – • – Still find papers on “tiny” corpora Collecting x data with tricks ≈ collecting 10 x data without tricks Wish list: more papers measuring power of various tricks Was balancing BNC (British National Corpus) worth the effort? • Should a corpus be balanced? (Oxford Debate, 1991) The rising tide of data will lift all boats! 1. 2. TREC Question Answering Collocations: Eurospeech 2003 69
Bait: Use Web to Establish Excitement: More data is better data • Shocking at TMI-1992 (Bob Mercer) Larger market share More $$ for R&D Better Moore’s Law Time Constant – – • • but less so a decade later (Eric Brill) Many researchers are finding that performance improves with corpus size, over full range of sizes that are available. EMNLP-2002 Best paper (& CL): Using the Web to Overcome Data Sparseness, Keller et al – For many tasks: Google is displacing BNC just as PCs displaced Crays Larger corpora (100 B Google) >> Smaller corpora (100 M BNC) – – • – My spin Language modelling Predicting psycholinguistic judgements Collecting more data is better than tricks for not collecting data • • Smoothing, balance, etc. Tricks have limited power: – • – Still find papers on “tiny” corpora Collecting x data with tricks ≈ collecting 10 x data without tricks Wish list: more papers measuring power of various tricks Was balancing BNC (British National Corpus) worth the effort? • Should a corpus be balanced? (Oxford Debate, 1991) The rising tide of data will lift all boats! 1. 2. TREC Question Answering Collocations: http: //labs 1. google. com/sets Eurospeech 2003 70
The rising tide of data will lift all boats! TREC Question Answering & Google: What is the highest point on Earth? Eurospeech 2003 71
The rising tide of data will lift all boats! Acquiring Lexical Resources from Data: Dictionaries, Ontologies, Word. Nets, Language Models, etc. http: //labs 1. google. com/sets Cat Dog Horse cat more England France Japan China Fish Bird Rabbit Cattle Rat Eurospeech 2003 72
The rising tide of data will lift all boats! Acquiring Lexical Resources from Data: Dictionaries, Ontologies, Word. Nets, Language Models, etc. http: //labs 1. google. com/sets Cat Dog Horse cat more ls Fish rm Bird mv Rabbit cd Cattle cp Rat England France mkdir Eurospeech 2003 Japan China 73
The rising tide of data will lift all boats! Acquiring Lexical Resources from Data: Dictionaries, Ontologies, Word. Nets, Language Models, etc. http: //labs 1. google. com/sets Cat Dog Horse cat more ls England France Germany Fish rm Italy Bird mv Ireland Rabbit cd Spain Cattle cp Scotland Rat mkdir Belgium Eurospeech 2003 Japan China 74
The rising tide of data will lift all boats! Acquiring Lexical Resources from Data: Dictionaries, Ontologies, Word. Nets, Language Models, etc. http: //labs 1. google. com/sets Cat Dog Horse Fish cat more ls rm England France Germany Italy Japan China India Indonesia Bird Rabbit Cattle Rat Livestock Mouse Human mv cd cp mkdir man tail pwd Ireland Spain Scotland Belgium Canada Austria Australia Malaysia Korea Taiwan Thailand Singapore Australia Bangladesh Eurospeech 2003 75
Rising Tide of Data Lifts all Boats Bait: use public web to create & socialize new ideas Bait • More data better results – TREC Question Answering • Remarkable performance: Google and not much else – Norvig (ACL-02) – Ask. MSR (SIGIR-02) – Lexical Acquisition • Google Sets – We tried similar things » but with tiny corpora » which we called large Switch: port these ideas to private repositories Switch Eurospeech 2003 76
Recommendations Bait and Switch Strategy • Strategy: Use the public Intranet to develop, test and socialize new ways to extract value from large linguistic repositories – Value to society: Port solutions to private repositories • Research papers: Bait – Keep up the good work! – There is already considerable interest in evaluation of new ideas on corpora (public repositories) – There will be more interest in Switch • How well methods port to new corpora • How well performance scales with size – Hopefully corpus size helps • But of course, all the data in the world – Will not solve all the world’s problems – Need to understand when more data will help • And when it is better to do something else – Revival of Rationalism (Linguistics) Eurospeech 2003 77
More Recommendations Bait and Switch Strategy • Infrastructure Bait – In addition to traditional public repositories (large) • Web data, data collection efforts such as LDC – We ought to think more about private repositories (even larger) • Most of us do not keep voice mail for long Switch – But I have been using Scanmail to copy my voice mail to email – And like many, I keep email online for a long time • Private repositories would be much larger if – It was more convenient to capture private data – and there was obvious value in doing so. • Currently, tools for public repositories (e. g. , Google) – are better than comparable tools for private data (e. g. , searching email) • Better search tools (email, speech & video) Larger private repositories • New priorities (consume space) new killer apps – Search (consumes space) >> Dictation (data entry) & Compression Eurospeech 2003 78
Summary: More realistic expectations Where have we been and where are we going? • 1970 s: Hot debate: knowledge v. data intensive methods – People think about what they can afford to think about – Data was expensive • Only the richest industrial labs could play • Beyond the reach of most universities • Victor Zue dreams of having an hour of speech online (with annotations) • 1990 s: Revival of Empiricism: More data is better data! Demonstrate consistent progress over time – Everyone can afford to play (but still expensive) – Linguistic Data Consortium (LDC) Web – Evaluation, evaluation demonstrates consistent progress over time, but not as convincingly as Moore’s Law – Data intensive: method of choice • Pendulum swings (too) far • Is this progress, or is the pendulum about to swing back the other way? Oscillations • 2010 s: Petabytes everywhere (be careful what you ask for) – Big problem: Supply >> Demand tech meltdown (? ? ) – No problem: Demand has always kept up new killer apps Discontinuities • Search (consumes space) >> dictation (data entry) & compression • Video >> Speech >> Text Eurospeech 2003 79 Don’t see how to consume PB per capita
Where have we been and where are we going? 1. Consistent progress over decades • • 2. Moore’s Law, Speech Coding, Error Rate Time constant limited by: physics and/or R&D investment History repeats itself: • Mark Twain; bad idea then and still a bad idea now • • 3. Empiricism: 1950 s Rationalism: 1970 s Empiricism: 1990 s Rationalism: 2010 s (? ) Discontinuities: • Fundamental changes that invalidate fundamental assumptions • • Petabytes: $2, 000 Can demand keep up with supply? If not Tech meltdown New priorities: data entry create demand for petabytes – New Killer Apps: Search (creates demand) >> Compression & Dictation Eurospeech 2003 80
Backup Eurospeech 2003
Speech Language Shannon’s: Noisy Channel Model Language Model Channel Model • I Noisy Channel O • I΄ ≈ ARGMAXI Pr(I|O) = ARGMAXI Pr(I) Pr(O|I) Language Model Word Rank More likely alternatives We 9 The This One Two A Three Please In need 7 are will the would also do to 1 Application Independent resolve 85 have know do… all 9 The This One Two A Three Please In of 2 The This One Two A Three Please In the Channel Model Application Input Output Speech Recognition writer rider OCR (Optical Character Recognition) all a 1 l Spelling Correction government goverment 1 important issues 657 14 document question first… thing point to Eurospeech 2003 82
Speech Language Using (Abusing) Shannon’s Noisy Channel Model: Part of Speech Tagging and Machine Translation • Speech – Words Noisy Channel Acoustics • OCR – Words Noisy Channel Optics • Spelling Correction – Words Noisy Channel Typos • Part of Speech Tagging (POS): – POS Noisy Channel Words • Machine Translation: “Made in America” – English Noisy Channel French Eurospeech 2003 83
I am going to try to avoid making predictions like these because… • Too falsifiable • Appearance of conflicts of interest – Sound like you are trying to raise money for your favorite stuff • Committees do what committees do – Union of all (represented) positions = no position – Advocate what the members are currently working on • Rarely establish new strategic direction • Boring (too obviously correct) Eurospeech 2003 84
Predictions: Where are we going? Change the subject (engage in meta discussion) • Set unrealistic expectations (plenty of examples) – Sound like you are trying to raise money for your favorite stuff • And that you have lost touch with reality • Come up short (fewer examples) • Sound like you’re over the hill (old fogies session at Coling) – Kids these days don’t get it – Everyone should still be working on • what we thought was important when we were kids – Dress up old-style thinking (empiricism/rationalism) Ø • with current fashion (web) Meta discussion: consistent progress, history repeating itself, discontinuities Ø Come up with a new angle: bounds • Lower bound: we will solve such and such (x) – Extrapolations based on Moore’s Law • Upper bound: we won’t solve x (soon/ever) – e. g. , pass Turing Test, compress speech down to text rates – And you can bank on it good apps based on assumption x can’t be done Eurospeech 2003 85
Breaking Through Automation Barriers Illustrative Complexity of User Interaction Borrowed Slide Natural Language Dialog Agents n io at m Advanced to u ASR f. A o Word nt Spotting te Ex Traditional IVR Complexity of Services Eurospeech 2003 86
Borrowed Slide Past, Present, Future…. Directory Assistance VRCP 1990+ • Constrained speech • Minimal data collection • Manual design Keyword spotting Handcrafted grammars No dialogue 1995+ • Constrained speech • Moderate data collection • Some automation Airline reservation Medium size ASR Banking Handcrafted Grammars System Initiative 2000+ 2005+ • Spontaneous speech • Extensive data collection • Semi-automation • Spontaneous speech/pen • Fully automated systems MATCH: Multimodal Access To Call centers, Large size ASR City Help E-commerce Limited NLU Mixed-initiative Unlimited ASR Deeper NLU Adaptive systems Eurospeech 2003 Multimodal, Multilingual Help Desks, E-commerce 87
Example of Upper Bound: Reverse Turing Test (Kochanski et al. , ICSLP-2002) • Assume: won’t pass Turing Test (any time soon) • Assumptions you can bank on – Liberace: cry all the way to the bank • Good apps for crummy (limited) technology – “Good Applications for Crummy Machine Translation” • Church & Hovy (1993) • Reverse Turing Test – Owner of web site wants to grant access to people but not to spiders – Task: distinguish friend from foe, man from beast – Solution: assume there a class of problems (AI-complete) that any person can do and no machine can. • Currently deployed Reverse Turing Applications – Assume OCR is AI-complete – User is given a degraded image and asked to enter text into a form – Easy for people but challenging for machines • Problem: OCR is not challenging enough for machines • Proposal: Speech recognition with noise is more challenging – We can bank on not solving the cocktail party effect any time soon Eurospeech 2003 88
Where have we been and where are we going? 1. Consistent progress over decades • 2. Moore’s Law, Speech Coding, Error Rate History repeats itself • • Ø Empiricism: 1950 s Rationalism: 1970 s Empiricism: 1990 s Rationalism: 2010 s (? ) Discontinuities: Fundamental changes that invalidate fundamental assumptions • • Petabytes: $2, 000 Can demand keep up with supply? If not Tech meltdown New priorities: Search >> Compression & Dictation Eurospeech 2003 89
Statistical MT: IBM Models 1 -5 • E Noisy Channel F • E΄ = ARGMAXE Pr(E) Pr(F|E) • Language Model, Pr(E): – Trigram model (borrowed from speech recog) • Channel Model, Pr(F|E): – Based on aligned parallel corpora – Models 1 -5: alignment • Mercer & Church (Computational Linguistics, 1993) – Statistical MT may fail for reasons advanced by Chomsky – Regardless of its ultimate success or failure, – There is a growing community of researchers in corpus-based linguistics who believe it will produce valuable lexical resources • Bilingual concordances • Translation tools • Training & testing material for word sense disambig (senseval) Eurospeech 2003 90
Word Sense Disambiguation • Knowledge Acquisition Bottleneck – – Bar-Hillel (1960) Expert systems don’t scale Sense-tagged text: expensive Parallel text! • Translation = sense-tagged text – Sentence (judicial sense) peine – Sentence (syntactic sense) phrase • Yarowsky: bilingual monolingual • One sense per discourse • Machine Learning: early example of co-training (EM alg) Eurospeech 2003 91
TMI-02 Keynote (similar subject) The organizers asked me… • What's changed since TMI-92 (if anything)? – TMI-92: great excitement over the use of aligned parallel corpora to help human translators (translation tools) – Also, much controversy over IBM Models 1 -5 • Have IBM Models 1 -5 failed to solve all the world’s problems? • So what's happened (if anything) since 1992? – Empiricism has come of age • Textbooks: Charniak, Jelinek, Manning & Schultze, Jurafsky & Martin • Textbooks courses in many universities around the world – What used to be considered radical is now accepted practice • Evaluation is practically required for publication – Mercer’s fighting words: More data is better data! • Aren’t as shocking when Brill makes the case a decade later – The new field of Machine Learning has absorbed many good (and formally controversial) ideas including • IBM Models 1 -5 • Yarowsky's Word Sense Disambiguation – Grew out of Machine Translation, – But is now widely cited in Machine Learning as an early example of co-training Eurospeech 2003 92
What has happened to the IBMApproach to Machine Translation? • Support for human translators – – – • Terminology: translators don’t need help with the easy vocabulary and the easy grammar Translation Memory: translators are often asked to translate the same material again and again (e. g. , revisions of manuals) Alignment Fully automatic – – • CLIR: cross-language information retrieval Translating web pages Academic fields – – Machine Learning: most important contribution Corpus-based Lexicography: spreading into lots of other fields Eurospeech 2003 93
Revival of Empiricism: A Personal Perspective • As a student at MIT, I was solidly opposed to empiricism – But that changed soon after moving to AT&T Bell Labs (1983) • Letter-to-Sound Rules (speech synthesis) – Names: Letter stats Etymology Pronunciation video – Net. Talk: Neural Nets video • • • Demo: great theater unrealistic expectations Self-organizing systems v. empiricism Machine Learning v. Corpus-based Linguistics I did it, but… Part of Speech Tagging (1988) Word Associations (Hanks) – Mutual info collocations & word associations • Collocations: Strong tea v. powerful computers • Word Associations: bread and butter, doctor/nurse • • • Good-Turing Smoothing (Gale) Aligning Parallel Corpora (inspired by MT) Word Sense Disambiguation – Bilingual Monolingual • Even if IBM’s approach fails for MT lasting benefit (tools, linguistic resources, academic contributions to machine learning) Eurospeech 2003 94
Ceiling Speech Coding (Telephony) • More complicated than Moore’s Law – Many Dimensions: Bit Rate, Quality, Complexity and Delay – Quality ceiling (imposed by telephone standards) • Easy to reach the ceiling at high bit rates (≥ 8 kb/s) • More room for progress at low bit rates (≤ 8 kb/s) • Moore’s Law Time Constant – Bit rates half every decade (≤ 8 kb/s) – Relatively slow by Moore’s Law standards (not hyper-inflation) • Performance doubles every decade • Like disk seek or money in the bank (normal inflation) – Limited more by physics than investment • Potential compression opportunity – At most 10 x: 8 kb/s 2 kb/s 1 kb/s (? ) • Speech (2 kb/s) >> text (2 bits/char): 100 -1000 times more bits – Speech coding will not close this gap foreseeable future Eurospeech 2003 95
Скачать презентацию Speech and Language Processing Where have we been
61aca38dc9f012c5d34af73af5439c18.ppt