Seminar Speech Recognition 2003 E. M. Bakker LIACS Media Lab Leiden University Speech Recognition 2003
Outline • • Introduction and State of the Art A Speech Recognition Architecture – Acoustic modeling – Language modeling – Practical issues • Applications NB Some of the slides are adapted from the presentation: “Can Advances in Speech Recognition make Spoken Language as Convenient and as Accessible as Online Text? ”, an excellent presentation by: Dr. Patti Price, Speech Technology Consulting Menlo Park, California 94025, and Dr. Joseph Picone Institute for Signal and Information Processing Dept. of Elect. and Comp. Eng. Mississippi State University LIACS Media Lab Leiden University Speech Recognition 2003
Research Areas • Speech Analysis (Production, Perception, Parameter Estimation) • Speech Coding/Compression • Speech Synthesis (TTS) • Speaker Identification/Recognition/Verification (Sprint, TI) • Language Identification (Transparent Dialogue) • Speech Recognition (Dragon, IBM, ATT) Speech recognition sub-categories: • Discrete/Connected/Continuous Speech/Word Spotting • Speaker Dependent/Independent • Small/Medium/Large/Unlimited Vocabulary • Speaker-Independent Large Vocabulary Continuous Speech Recognition (or LVCSR for short : ) LIACS Media Lab Leiden University Speech Recognition 2003
Introduction What is Speech Recognition? Goal: Automatically extract the string of words spoken from the speech signal Speech Signal • Speech Recognition Words “How are you? ” Other interesting area’s: – Who is talker (speaker recognition, identification) – Speech output (speech synthesis) – What the words mean (speech understanding, semantics) LIACS Media Lab Leiden University Speech Recognition 2003
Introduction Applications • Command control –Manufacturing –Consumer products http: //www. speech. philips. com • Database query – Resource management – Air travel information – Stock quote Nuance, American Airlines: 1 -800 -433 -7300, touch 1 • Dictation –http: //www. lhsl. com/contacts/ –http: //www-4. ibm. com/software/speech –http: //www. microsoft. com/speech/ LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art Speech-recognition software • IBM (Via Voice, Voice Server Applications, . . . ) – – • • Speaker independent, continuous command recognition Large vocabulary recognition Text-to-speech confirmation Barge in (The ability to interrupt an audio prompt as it is playing) Dragon Systems, Lernout & Hauspie (L&H Voice Xpress™ (: ( ) Philips – Dictation – Telephone – Voice Control (Speech. Wave, Vo. Con SDK, chip-sets) • Microsoft (Whisper, Dr Who) LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art Speech over the telephone. : • • • AT&T Bell Labs pioneered the use of speechrecognition systems for telephone transactions companies such as Nuance, Philips and Speech. Works are active in this field for some years now. IBM Applications over telephone: – request news, internet pages, e-mail – stock quotes, traveling info – weather information LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art Speech over the telephone (Philips): • • Speech. Pearl® large vocabulary natural language recognition (up to 200, 000 words) Speech. Mania® mixed initiative dialog gives the caller the impression of a truly natural dialogue: full replacement of the human operator. Speech. Wave™ relatively small vocabularies (up to hundreds of words) available in nearly 40 languages Voice Re. Quest The system recognizes the request and routes the call to the appropriate extension, all without the intervention of an operator. LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art Speech over the telephone. : • • Brokerage: E*Trade, . . . Banking Travel: United Airlines, Continental Airlines Bell. South HP Federal Express Foodline LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art Speech over the telephone. LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art The following is a transcript of a telephone conversation between Victor Zue and the Jupiter system for weather information. Jupiter's toll-free number is 1 -888 -573 -8255: • • • [Phone ringing] Jupiter: Hi. Welcome to Jupiter, the M. I. T. Lab for Computer Science Weather Information System. This call is being recorded for system development. You may hang up or ask for help at any time. How can I help you? V. Z. : What is the forecast for Boston today? Jupiter: In Boston this afternoon, partly sunny and windy. High around 50. Tonight, clear, lows around 35. What other information can I give you? V. Z. : Will it rain tomorrow? Jupiter: To my knowledge, the forecast calls for no rain in Boston tomorrow. Is there anything else? LIACS Media Lab Leiden University Speech Recognition 2003
Introduction: State of the Art • • • V. Z. : How about Seattle? Jupiter: To my knowledge, the forecast calls for no rain in Seattle tomorrow. Is there something else you would like to know? V. Z. : Is it sunny anywhere in the Caribbean? Jupiter: Sorry, I am not sure what you said. V. Z. : Is it sunny anywhere in the Caribbean? Jupiter: I am sorry, I may not know the city you are asking about. Try asking for the state or country. For example, what cities do you know about in Massachusetts? V. Z. : What is the temperature in Montego Bay? Jupiter: In Montego Bay today, high 86 and low 73. Is there something else? V. Z. : Good-bye. LIACS Media Lab Leiden University Speech Recognition 2003
Factors that Affect Performance of Speech Recognition Systems 2005 wherever speech occurs 2000 vehicle noise radio cell phones NOISE ENVIRONMENT all speakers of the language including foreign regional accents native speakers competent foreign speakers 1995 normal office various microphones telephone quiet room fixed high – quality mic speaker independent and adaptive USER speakerdep. POPULATION 1985 careful reading SPEECH STYLE planned speech natural humanmachine dialog (user can adapt) all styles including human -human (unaware)Media Lab Leiden University LIACS application– specific speech and expertto years language create app – specific language model COMPLEXITY some application – specific data and one engineer year application independent or adaptive Speech Recognition 2003
How Do You Measure the Performance? USC, October 15, 1999: “the world's first machine system that can recognize spoken words better than humans can. ” “ In benchmark testing using just a few spoken words, USC's Berger-Liaw … System not only bested all existing computer speech recognition systems but outperformed the keenest human ears. ” • What benchmarks? • What was training? • What was the test? • Were they independent? • How large was the vocabulary and the sample size? • Did they really test all existing systems? Is that different from chance? • Was the noise added or coincident with speech? • What kind of noise? Was it independent of the speech? LIACS Media Lab Leiden University Speech Recognition 2003
Evaluation Metrics Word Error Rate (WER) Conversational Speech 40% 30% Broadcast News 20% Read Speech 10% Continuous Digits • Spontaneous telephone speech is still a “grand challenge”. • Telephone-quality speech is still central to the problem. • Broadcast news is a very dynamic domain. Letters and Numbers Command Control 0% Level Of Difficulty LIACS Media Lab Leiden University Speech Recognition 2003
Evaluation Metrics Human Performance Word Error Rate 20% Wall Street Journal (Additive Noise) • Human performance exceeds machine performance by a factor ranging from 4 x to 10 x depending on the task. • On some tasks, such as credit card number recognition, machine performance exceeds humans due to human memory retrieval capacity. 15% Machines 10% • The nature of the noise is as important as the SNR (e. g. , cellular phones). 5% Human Listeners (Committee) 0% 10 d. B 16 d. B 22 d. B Quiet • A primary failure mode for humans is inattention. • A second major failure mode is the lack of familiarity with the domain (i. e. , business terms and corporation names). Speech-To-Noise Ratio LIACS Media Lab Leiden University Speech Recognition 2003
Evaluation Metrics Machine Performance 100% (Foreign) Read Speech Conversational Speech 20 k Broadcast Spontaneous vocabularies Varied Speech (Foreign) Speech 10 X Microphones 10% 5 k 1 k Noisy • A Word Error Rate (WER) below 10% is considered acceptable. • Performance in the field is typically 2 x to 4 x worse than performance on an evaluation. 1% 1988 1989 1990 1991 1992 1993 1994 1995 LIACS Media Lab Leiden University 1996 1997 1998 1999 2000 2001 2002 Speech Recognition 2003
What does a speech signal look like? LIACS Media Lab Leiden University Speech Recognition 2003
Spectrogram LIACS Media Lab Leiden University Speech Recognition 2003
Speech Recognition LIACS Media Lab Leiden University Speech Recognition 2003
Recognition Architectures Why Is Speech Recognition So Difficult? • Comparison of “aa” in “IOck” vs. “iy” in b. EAt for conversational speech (SWB) Feature No. 2 Ph_1 Ph_2 Ph_3 Feature No. 1 • Measurements of the signal are ambiguous. • Region of overlap represents classification errors. • Reduce overlap by introducing acoustic and linguistic context (e. g. , context-dependent phones). LIACS Media Lab Leiden University Speech Recognition 2003
Overlap in the ceptral space (alphadigits) Female “aa” Female “iy” Male “aa” Male “iy” LIACS Media Lab Leiden University Speech Recognition 2003
Overlap in the cepstral space (alphadigits) Male “aa” (green) vs. Female “aa” (black) Male “iy” (blue) vs. Female “iy” (red) • Combined Comparisons: • Male "aa" (green) • Female "aa" (black) • Male "iy" (blue) • Female "iy" (red) LIACS Media Lab Leiden University Speech Recognition 2003
OVERLAP IN THE CEPSTRAL SPACE (SWB-All) The following plots demonstrate overlap of recognition features in the cepstral space. These plots consist of all vowels excised from tokens in the SWITCHBOARD conversational speech corpus. All Male Vowels All Female Vowels LIACS Media Lab Leiden University All Vowels Speech Recognition 2003
Recognition Architectures A Communication Theoretic Approach Message Source Observable: Message Linguistic Channel Articulatory Channel Acoustic Channel Words Sounds Features Bayesian formulation for speech recognition: • P(W|A) = P(A|W) P(W) / P(A) Objective: minimize the word error rate Approach: maximize P(W|A) during training Components: • P(A|W) : acoustic model (hidden Markov models, mixtures) • P(W) : language model (statistical, finite state networks, etc. ) The language model typically predicts a small set of next words based on knowledge of a finite number of previous words (N-grams). LIACS Media Lab Leiden University Speech Recognition 2003
Recognition Architectures Incorporating Multiple Knowledge Sources • The signal is converted to a sequence of feature vectors based on spectral and temporal measurements. Input Speech Acoustic Front-end Acoustic Models P(A/W) Language Model P(W) Search Recognized Utterance LIACS Media Lab Leiden University • Acoustic models represent sub-word units, such as phonemes, as a finitestate machine in which states model spectral structure and transitions model temporal structure. • The language model predicts the next set of words, and controls which models are hypothesized. • Search is crucial to the system, since many combinations of words must be investigated to find the most probable word sequence. Speech Recognition 2003
Acoustic Modeling Feature Extraction Fourier Transform Input Speech • Typical: 512 samples (16 k. Hz sampling rate) => Cepstral Analysis • Incorporate knowledge of the nature of speech sounds in measurement of the features. • Utilize rudimentary models of human perception. • Use a ~30 msec window for frequency domain analysis. • Include absolute energy and 12 spectral measurements. Perceptual Weighting Time Derivative Energy + Mel-Spaced Cepstrum Delta Energy + Delta Cepstrum Delta-Delta Energy + Delta-Delta Cepstrum • Time derivatives to model spectral change. LIACS Media Lab Leiden University Speech Recognition 2003
Acoustic Modeling Hidden Markov Models • Acoustic models encode the temporal evolution of the features (spectrum). • Gaussian mixture distributions are used to account for variations in speaker, accent, and pronunciation. • Phonetic model topologies are simple left-to-right structures. • Skip states (time-warping) and multiple paths (alternate pronunciations) are also common features of models. • Sharing model parameters is a common strategy to reduce complexity. LIACS Media Lab Leiden University Speech Recognition 2003
Acoustic Modeling Parameter Estimation • Initialization • Single Gaussian Estimation • • • Word level transcription Supervises a closed-loop data-driven modeling Initial parameter estimation • The expectation/maximization (EM) algorithm is used to improve our parameter estimates. • Computationally efficient training algorithms (Forward-Backward) are crucial. • Batch mode parameter updates are typically preferred. • Decision trees and the use of additional linguistic knowledge are used to optimize parameter-sharing, and system complexity, . • 2 -Way Split • Mixture Distribution Reestimation • 4 -Way Split • Reestimation • • • LIACS Media Lab Leiden University Speech Recognition 2003
Language Modeling Is A Lot Like Wheel of Fortune LIACS Media Lab Leiden University Speech Recognition 2003
Language Modeling N-Grams: The Good, The Bad, and The Ugly Unigrams (SWB): • Most Common: “I”, “and”, “the”, “you”, “a” • Rank-100: “she”, “an”, “going” • Least Common: “Abraham”, “Alastair”, “Acura” Bigrams (SWB): • Most Common: “you know”, “yeah SENT!”, “!SENT um-hum”, “I think” • Rank-100: “do it”, “that we”, “don’t think” • Least Common: “raw fish”, “moisture content”, “Reagan Bush” Trigrams (SWB): • Most Common: “!SENT um-hum SENT!”, “a lot of”, “I don’t know” • Rank-100: “it was a”, “you know that” • Least Common: “you have parents”, “you seen Brooklyn” LIACS Media Lab Leiden University Speech Recognition 2003
Language Modeling Integration of Natural Language • Natural language constraints can be easily incorporated. • Lack of punctuation and search space size pose problems. • Speech recognition typically produces a word-level time-aligned annotation. • Time alignments for other levels of information also available. LIACS Media Lab Leiden University Speech Recognition 2003
Implementation Issues Dynamic Programming-Based Search • Dynamic programming is used to find the most probable path through the network. • Beam search is used to control resources. • Search is time synchronous and left-to-right. • Arbitrary amounts of silence must be permitted between each word. • Words are hypothesized many times with different start/stop times, which significantly increases search complexity. LIACS Media Lab Leiden University Speech Recognition 2003
Implementation Issues Cross-Word Decoding Is Expensive • Cross-word Decoding: since word boundaries don’t occur in spontaneous speech, we must allow for sequences of sounds that span word boundaries. • Cross-word decoding significantly increases memory requirements. LIACS Media Lab Leiden University Speech Recognition 2003
Implementation Issues Search Is Resource Intensive • Typical LVCSR systems have about 10 M free parameters, which makes training a challenge. • Large speech databases are required (several hundred hours of speech). • Tying, smoothing, and interpolation are required. LIACS Media Lab Leiden University Speech Recognition 2003
Applications Conversational Speech • Conversational speech collected over the telephone contains background noise, music, fluctuations in the speech rate, laughter, partial words, hesitations, mouth noises, etc. • WER (Word Error Rate) has decreased from 100% to 30% in six years. • Laughter • Singing • Unintelligible • Spoonerism • Background Speech • No pauses • Restarts • Vocalized Noise • Coinage LIACS Media Lab Leiden University Speech Recognition 2003
Applications Audio Indexing of Broadcast News Broadcast news offers some unique challenges: • Lexicon: important information in infrequently occurring words • Acoustic Modeling: variations in channel, particularly within the same segment (“ in the studio” vs. “on location”) • Language Model: must adapt (“ Bush, ” “Clinton, ” “Bush, ” “Mc. Cain, ” “? ? ? ”) • Language: multilingual systems? language-independent acoustic modeling? LIACS Media Lab Leiden University Speech Recognition 2003
Applications Automatic Phone Centers • Portals: Bevocal, Tell. Me, Hey. Aniat • Voice. XML 2. 0 • Automatic Information Desk • Reservation Desk • Automatic Help-Desk • With Speaker identification • bank account services • e-mail services • corporate services LIACS Media Lab Leiden University Speech Recognition 2003
Applications Real-Time Translation • From President Clinton’s State of the Union address (January 27, 2000): “These kinds of innovations are also propelling our remarkable prosperity. . . Soon researchers will bring us devices that can translate foreign languages as fast as you can talk. . . molecular computers the size of a tear drop with the power of today’s fastest supercomputers. ” • Imagine a world where: • You book a travel reservation from your cellular phone while driving in your car without ever talking to a human (database query) • You converse with someone in a foreign country and neither speaks a common language (universal translator) • You place a call to your bank to inquire about your bank account and never have to remember a password (transparent telephony) • You can ask questions by voice and your Internet browser returns answers to your questions (intelligent query) • Human Language Engineering: a sophisticated integration of many speech and language related technologies. . . a science for the next millennium. LIACS Media Lab Leiden University Speech Recognition 2003
Technology Future Directions Analog Filter Banks 1960 Dynamic Time-Warping Hidden Markov Models 1980 1970 Conclusions: Challenges: • supervised training is a good machine learning technique • • 1990 2000 • large databases are essential for the development of robust statistics discrimination vs. representation generalization vs. memorization pronunciation modeling human-centered language modeling The algorithmic issues for the next decade: • Better features by extracting articulatory information? • Bayesian statistics? Bayesian networks? • Decision Trees? Information-theoretic measures? • Nonlinear dynamics? Chaos? LIACS Media Lab Leiden University Speech Recognition 2003
Скачать презентацию Seminar Speech Recognition 2003 E M Bakker LIACS
a0471e73a415a909886e23f6e7d6c903.ppt